2025-09-07T06:40:48.9269768Z Current runner version: '2.328.0' 2025-09-07T06:40:48.9275157Z Runner name: 'i-085acfb4aecab35f4' 2025-09-07T06:40:48.9275800Z Runner group name: 'default' 2025-09-07T06:40:48.9276729Z Machine name: 'ip-10-0-10-208' 2025-09-07T06:40:48.9278942Z ##[group]GITHUB_TOKEN Permissions 2025-09-07T06:40:48.9281142Z Contents: read 2025-09-07T06:40:48.9281591Z Metadata: read 2025-09-07T06:40:48.9282027Z ##[endgroup] 2025-09-07T06:40:48.9283839Z Secret source: Actions 2025-09-07T06:40:48.9284454Z Prepare workflow directory 2025-09-07T06:40:48.9697253Z Prepare all required actions 2025-09-07T06:40:48.9732132Z Getting action download info 2025-09-07T06:40:49.2773253Z Download action repository 'pytorch/test-infra@main' (SHA:548a4bc624d43a01cdf165a63b041f0ae014ddbd) 2025-09-07T06:40:50.8087945Z Download action repository 'pytorch/pytorch@main' (SHA:93fb23d6fae7c4e82c4239a1033e522088742634) 2025-09-07T06:41:05.3026779Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-09-07T06:41:05.7070176Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-09-07T06:41:05.9605596Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-09-07T06:41:06.1384366Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-09-07T06:41:06.5193708Z Getting action download info 2025-09-07T06:41:06.6378434Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-09-07T06:41:06.9022290Z Getting action download info 2025-09-07T06:41:07.0025162Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-09-07T06:41:07.2449139Z Getting action download info 2025-09-07T06:41:07.3702859Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-09-07T06:41:07.5936873Z Getting action download info 2025-09-07T06:41:07.7224254Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (93fb23d6fae7c4e82c4239a1033e522088742634) 2025-09-07T06:41:07.7227698Z ##[group] Inputs 2025-09-07T06:41:07.7227995Z build-environment: linux-jammy-py3.9-gcc11-build 2025-09-07T06:41:07.7229793Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-09-07T06:41:07.7231937Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:41:07.7232534Z sync-tag: 2025-09-07T06:41:07.7233206Z timeout-minutes: 240 2025-09-07T06:41:07.7233391Z use-gha: 2025-09-07T06:41:07.7233565Z dashboard-tag: 2025-09-07T06:41:07.7233834Z s3-bucket: gha-artifacts 2025-09-07T06:41:07.7234090Z aws-role-to-assume: 2025-09-07T06:41:07.7235045Z disable-monitor: false 2025-09-07T06:41:07.7235384Z monitor-log-interval: 5 2025-09-07T06:41:07.7235959Z monitor-data-collect-interval: 1 2025-09-07T06:41:07.7236388Z ##[endgroup] 2025-09-07T06:41:07.7236823Z Complete job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:41:07.7753524Z A job started hook has been configured by the self-hosted runner administrator 2025-09-07T06:41:07.7836463Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-09-07T06:41:07.7843692Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:41:07.7844237Z ##[endgroup] 2025-09-07T06:41:08.7651306Z Runner Type: linux.8xlarge.amx 2025-09-07T06:41:08.7651816Z Instance Type: m7i-flex.8xlarge 2025-09-07T06:41:08.7652179Z AMI Name: unknown 2025-09-07T06:41:08.7685082Z AMI ID: ami-05ffe3c48a9991133 2025-09-07T06:41:13.2490515Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-09-07T06:41:13.2491034Z with: 2025-09-07T06:41:13.2491688Z github-secret: *** 2025-09-07T06:41:13.2492267Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-09-07T06:41:13.2492832Z activate-with-label: false 2025-09-07T06:41:13.2493076Z label: with-ssh 2025-09-07T06:41:13.2493350Z remove-existing-keys: true 2025-09-07T06:41:13.2493703Z fail-silently: true 2025-09-07T06:41:13.2493941Z env: 2025-09-07T06:41:13.2494155Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:41:13.2494435Z ##[endgroup] 2025-09-07T06:41:13.3760506Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-09-07T06:41:13.3761482Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-09-07T06:41:13.4014295Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-09-07T06:41:13.4014683Z with: 2025-09-07T06:41:13.4014897Z no-sudo: true 2025-09-07T06:41:13.4015114Z submodules: recursive 2025-09-07T06:41:13.4015348Z fetch-depth: 0 2025-09-07T06:41:13.4015559Z env: 2025-09-07T06:41:13.4015748Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:41:13.4015983Z ##[endgroup] 2025-09-07T06:41:13.4095355Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-09-07T06:41:13.4095987Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-09-07T06:41:13.4103744Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:41:13.4104013Z env: 2025-09-07T06:41:13.4104238Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:41:13.4104479Z ##[endgroup] 2025-09-07T06:41:13.4197273Z ##[group]Run # Use all available CPUs for fetching 2025-09-07T06:41:13.4197621Z # Use all available CPUs for fetching 2025-09-07T06:41:13.4197849Z cd "${GITHUB_WORKSPACE}" 2025-09-07T06:41:13.4198082Z git config --global fetch.parallel 0 2025-09-07T06:41:13.4198342Z git config --global submodule.fetchJobs 0 2025-09-07T06:41:13.4198573Z  2025-09-07T06:41:13.4198831Z # Clean workspace. The default checkout action should also do this, but 2025-09-07T06:41:13.4199131Z # do it here as well just in case 2025-09-07T06:41:13.4199351Z if [[ -d .git ]]; then 2025-09-07T06:41:13.4199562Z  if [ -z "${NO_SUDO}" ]; then 2025-09-07T06:41:13.4199776Z  sudo git clean -ffdx 2025-09-07T06:41:13.4199964Z  else 2025-09-07T06:41:13.4200138Z  git clean -ffdx 2025-09-07T06:41:13.4200346Z  fi 2025-09-07T06:41:13.4200496Z fi 2025-09-07T06:41:13.4205289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:41:13.4205569Z env: 2025-09-07T06:41:13.4205868Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:41:13.4206098Z NO_SUDO: true 2025-09-07T06:41:13.4206267Z ##[endgroup] 2025-09-07T06:41:13.4324287Z ##[group]Run actions/checkout@v4 2025-09-07T06:41:13.4324541Z with: 2025-09-07T06:41:13.4324742Z ref: 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:41:13.4325169Z fetch-depth: 0 2025-09-07T06:41:13.4325354Z submodules: recursive 2025-09-07T06:41:13.4325550Z show-progress: false 2025-09-07T06:41:13.4325749Z repository: pytorch/pytorch 2025-09-07T06:41:13.4326095Z token: *** 2025-09-07T06:41:13.4326271Z ssh-strict: true 2025-09-07T06:41:13.4326453Z ssh-user: git 2025-09-07T06:41:13.4326650Z persist-credentials: true 2025-09-07T06:41:13.4326852Z clean: true 2025-09-07T06:41:13.4327046Z sparse-checkout-cone-mode: true 2025-09-07T06:41:13.4327258Z fetch-tags: false 2025-09-07T06:41:13.4327431Z lfs: false 2025-09-07T06:41:13.4327593Z set-safe-directory: true 2025-09-07T06:41:13.4327797Z env: 2025-09-07T06:41:13.4327958Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:41:13.4328146Z ##[endgroup] 2025-09-07T06:41:13.5247917Z Syncing repository: pytorch/pytorch 2025-09-07T06:41:13.5248973Z ##[group]Getting Git version info 2025-09-07T06:41:13.5249287Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-09-07T06:41:13.5249756Z [command]/usr/bin/git version 2025-09-07T06:41:13.5455127Z git version 2.47.1 2025-09-07T06:41:13.5479544Z ##[endgroup] 2025-09-07T06:41:13.5488686Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/117d47e1-60ce-4e26-8a16-9ccd379d2dc1/.gitconfig' 2025-09-07T06:41:13.5511880Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/117d47e1-60ce-4e26-8a16-9ccd379d2dc1' before making global git config changes 2025-09-07T06:41:13.5512590Z Adding repository directory to the temporary git global config as a safe directory 2025-09-07T06:41:13.5526635Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-09-07T06:41:13.5572452Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-09-07T06:41:13.5576057Z ##[group]Initializing the repository 2025-09-07T06:41:13.5580201Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-09-07T06:41:13.5640295Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-09-07T06:41:13.5645255Z hint: is subject to change. To configure the initial branch name to use in all 2025-09-07T06:41:13.5647567Z hint: of your new repositories, which will suppress this warning, call: 2025-09-07T06:41:13.5648028Z hint: 2025-09-07T06:41:13.5654657Z hint: git config --global init.defaultBranch 2025-09-07T06:41:13.5660677Z hint: 2025-09-07T06:41:13.5661306Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-09-07T06:41:13.5661768Z hint: 'development'. The just-created branch can be renamed via this command: 2025-09-07T06:41:13.5662080Z hint: 2025-09-07T06:41:13.5662265Z hint: git branch -m 2025-09-07T06:41:13.5662652Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-09-07T06:41:13.5679772Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-09-07T06:41:13.5716352Z ##[endgroup] 2025-09-07T06:41:13.5716779Z ##[group]Disabling automatic garbage collection 2025-09-07T06:41:13.5717118Z [command]/usr/bin/git config --local gc.auto 0 2025-09-07T06:41:13.5753936Z ##[endgroup] 2025-09-07T06:41:13.5754298Z ##[group]Setting up auth 2025-09-07T06:41:13.5760742Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-09-07T06:41:13.5793018Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-09-07T06:41:13.6167493Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-09-07T06:41:13.6202534Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-09-07T06:41:13.6554028Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-09-07T06:41:13.6611356Z ##[endgroup] 2025-09-07T06:41:13.6611739Z ##[group]Fetching the repository 2025-09-07T06:41:13.6617816Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-09-07T06:42:00.6153315Z From https://github.com/pytorch/pytorch 2025-09-07T06:42:00.6156173Z * [new branch] 160583 -> origin/160583 2025-09-07T06:42:00.6156619Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-09-07T06:42:00.6157113Z * [new branch] 5addvllmbuild -> origin/5addvllmbuild 2025-09-07T06:42:00.6162694Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-09-07T06:42:00.6165952Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-09-07T06:42:00.6166466Z * [new branch] ISSUE-154849 -> origin/ISSUE-154849 2025-09-07T06:42:00.6169771Z * [new branch] JackCaoG/dynamo_make_fx_non_core_aten_ops -> origin/JackCaoG/dynamo_make_fx_non_core_aten_ops 2025-09-07T06:42:00.6170347Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-09-07T06:42:00.6173858Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-09-07T06:42:00.6174355Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-09-07T06:42:00.6175151Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-09-07T06:42:00.6179467Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-09-07T06:42:00.6179947Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-09-07T06:42:00.6180325Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-09-07T06:42:00.6184445Z * [new branch] VLA_exp -> origin/VLA_exp 2025-09-07T06:42:00.6184834Z * [new branch] actually-run-mps-aot-inductor -> origin/actually-run-mps-aot-inductor 2025-09-07T06:42:00.6185286Z * [new branch] add-missing-args-normalization -> origin/add-missing-args-normalization 2025-09-07T06:42:00.6185834Z * [new branch] add-user-guide-structure -> origin/add-user-guide-structure 2025-09-07T06:42:00.6186260Z * [new branch] add-vllm-nightly-build -> origin/add-vllm-nightly-build 2025-09-07T06:42:00.6186668Z * [new branch] add_compile_benchmarking -> origin/add_compile_benchmarking 2025-09-07T06:42:00.6187056Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-09-07T06:42:00.6187383Z * [new branch] addsimde -> origin/addsimde 2025-09-07T06:42:00.6187683Z * [new branch] addvllmtest -> origin/addvllmtest 2025-09-07T06:42:00.6188138Z * [new branch] adi/acl_upgrade -> origin/adi/acl_upgrade 2025-09-07T06:42:00.6188456Z * [new branch] adi/test -> origin/adi/test 2025-09-07T06:42:00.6188752Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-09-07T06:42:00.6189088Z * [new branch] adi/test_fusions -> origin/adi/test_fusions 2025-09-07T06:42:00.6189432Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-09-07T06:42:00.6189855Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-09-07T06:42:00.6190312Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-09-07T06:42:00.6190645Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-09-07T06:42:00.6191007Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-09-07T06:42:00.6191507Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-09-07T06:42:00.6191865Z * [new branch] alt-disable -> origin/alt-disable 2025-09-07T06:42:00.6192250Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-09-07T06:42:00.6192674Z * [new branch] angelayi/aoti_inductor_fx -> origin/angelayi/aoti_inductor_fx 2025-09-07T06:42:00.6193023Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-09-07T06:42:00.6193370Z * [new branch] angelayi/benchmark2 -> origin/angelayi/benchmark2 2025-09-07T06:42:00.6193771Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-09-07T06:42:00.6194175Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-09-07T06:42:00.6194552Z * [new branch] angelayi/custom_op_subgraph -> origin/angelayi/custom_op_subgraph 2025-09-07T06:42:00.6194925Z * [new branch] angelayi/customop -> origin/angelayi/customop 2025-09-07T06:42:00.6195298Z * [new branch] angelayi/fake_cache_empty -> origin/angelayi/fake_cache_empty 2025-09-07T06:42:00.6195700Z * [new branch] angelayi/is_symbolic_tracing -> origin/angelayi/is_symbolic_tracing 2025-09-07T06:42:00.6196069Z * [new branch] angelayi/item -> origin/angelayi/item 2025-09-07T06:42:00.6196475Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-09-07T06:42:00.6196843Z * [new branch] angelayi/opoverload -> origin/angelayi/opoverload 2025-09-07T06:42:00.6197210Z * [new branch] angelayi/pattern -> origin/angelayi/pattern 2025-09-07T06:42:00.6197581Z * [new branch] angelayi/pytree -> origin/angelayi/pytree 2025-09-07T06:42:00.6197940Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-09-07T06:42:00.6198312Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-09-07T06:42:00.6198660Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-09-07T06:42:00.6199012Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-09-07T06:42:00.6199383Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-09-07T06:42:00.6199734Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-09-07T06:42:00.6200072Z * [new branch] aoti_weight_sharing -> origin/aoti_weight_sharing 2025-09-07T06:42:00.6200470Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-09-07T06:42:00.6200907Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-09-07T06:42:00.6201316Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2025-09-07T06:42:00.6201662Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-09-07T06:42:00.6201994Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-09-07T06:42:00.6202335Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-09-07T06:42:00.6202870Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-09-07T06:42:00.6203288Z * [new branch] atalman_inductor_2.3.0 -> origin/atalman_inductor_2.3.0 2025-09-07T06:42:00.6203701Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-09-07T06:42:00.6204079Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-09-07T06:42:00.6204457Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-09-07T06:42:00.6204968Z * [new branch] autoupdate-transformers-pin-via-pr -> origin/autoupdate-transformers-pin-via-pr 2025-09-07T06:42:00.6205411Z * [new branch] bahuang/dtensor_demo -> origin/bahuang/dtensor_demo 2025-09-07T06:42:00.6205755Z * [new branch] bahuang/test -> origin/bahuang/test 2025-09-07T06:42:00.6206071Z * [new branch] base/1.5 -> origin/base/1.5 2025-09-07T06:42:00.6206460Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-09-07T06:42:00.6206857Z * [new branch] bc-lint-config -> origin/bc-lint-config 2025-09-07T06:42:00.6207238Z * [new branch] bc-lint-test-new-config -> origin/bc-lint-test-new-config 2025-09-07T06:42:00.6207625Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-09-07T06:42:00.6208049Z * [new branch] benchmarker_compat_with_do_bench -> origin/benchmarker_compat_with_do_bench 2025-09-07T06:42:00.6208463Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-09-07T06:42:00.6208840Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-09-07T06:42:00.6209198Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-09-07T06:42:00.6209576Z * [new branch] bf/cg-custom-wrapper -> origin/bf/cg-custom-wrapper 2025-09-07T06:42:00.6209980Z * [new branch] bf/cg-or-error -> origin/bf/cg-or-error 2025-09-07T06:42:00.6210332Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-09-07T06:42:00.6210682Z * [new branch] bf/cg-skip-1-kernel -> origin/bf/cg-skip-1-kernel 2025-09-07T06:42:00.6211025Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-09-07T06:42:00.6211467Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-09-07T06:42:00.6212094Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-09-07T06:42:00.6212648Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-09-07T06:42:00.6213341Z * [new branch] bf/default-recompile-reason -> origin/bf/default-recompile-reason 2025-09-07T06:42:00.6213789Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-09-07T06:42:00.6214158Z * [new branch] bf/exp -> origin/bf/exp 2025-09-07T06:42:00.6214513Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-09-07T06:42:00.6214910Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-09-07T06:42:00.6215299Z * [new branch] bf/partition-turn-on -> origin/bf/partition-turn-on 2025-09-07T06:42:00.6215734Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-09-07T06:42:00.6216084Z * [new branch] bf/rope -> origin/bf/rope 2025-09-07T06:42:00.6216487Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-09-07T06:42:00.6217243Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-09-07T06:42:00.6217932Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-09-07T06:42:00.6218613Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-09-07T06:42:00.6219139Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-09-07T06:42:00.6220082Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-09-07T06:42:00.6221070Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-09-07T06:42:00.6221867Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-09-07T06:42:00.6224310Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-09-07T06:42:00.6224750Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-09-07T06:42:00.6225181Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-09-07T06:42:00.6225570Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-09-07T06:42:00.6226249Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-09-07T06:42:00.6226699Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-09-07T06:42:00.6227113Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-09-07T06:42:00.6230231Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-09-07T06:42:00.6230734Z * [new branch] bowbao/bench_updates_stage -> origin/bowbao/bench_updates_stage 2025-09-07T06:42:00.6231152Z * [new branch] bowbao/dort_rewriter -> origin/bowbao/dort_rewriter 2025-09-07T06:42:00.6232553Z * [new branch] bowbao/wip_prs -> origin/bowbao/wip_prs 2025-09-07T06:42:00.6233026Z * [new branch] brister/break_tensorbox -> origin/brister/break_tensorbox 2025-09-07T06:42:00.6233444Z * [new branch] brister/custom_fx_backend -> origin/brister/custom_fx_backend 2025-09-07T06:42:00.6233855Z * [new branch] brister/fx_custom_triton -> origin/brister/fx_custom_triton 2025-09-07T06:42:00.6234276Z * [new branch] brister/tensor_box_output -> origin/brister/tensor_box_output 2025-09-07T06:42:00.6234740Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-09-07T06:42:00.6235391Z * [new branch] c57382a49 -> origin/c57382a49 2025-09-07T06:42:00.6235870Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-09-07T06:42:00.6236410Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-09-07T06:42:00.6239970Z * [new branch] camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 -> origin/camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 2025-09-07T06:42:00.6240906Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-09-07T06:42:00.6241435Z * [new branch] cherry-pick-149654-by-pytorch_bot_bot_ -> origin/cherry-pick-149654-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6241979Z * [new branch] cherry-pick-151939-by-pytorch_bot_bot_ -> origin/cherry-pick-151939-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6242518Z * [new branch] cherry-pick-154174-by-pytorch_bot_bot_ -> origin/cherry-pick-154174-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6243016Z * [new branch] cherry-pick-156260-by-pytorch_bot_bot_ -> origin/cherry-pick-156260-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6243542Z * [new branch] cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6244056Z * [new branch] cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6244737Z * [new branch] cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6245360Z * [new branch] cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6246143Z * [new branch] cherry-pick-158537-by-pytorch_bot_bot_ -> origin/cherry-pick-158537-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6247297Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6247861Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-09-07T06:42:00.6248798Z * [new branch] chilli/flex_vllm -> origin/chilli/flex_vllm 2025-09-07T06:42:00.6249344Z * [new branch] cleanup-inductor-benchmark-images -> origin/cleanup-inductor-benchmark-images 2025-09-07T06:42:00.6249968Z * [new branch] codex-testing -> origin/codex-testing 2025-09-07T06:42:00.6251241Z * [new branch] codex/add-helper-function-to-sizevars.py -> origin/codex/add-helper-function-to-sizevars.py 2025-09-07T06:42:00.6252008Z * [new branch] codex/add-helper-function-to-sizevars.py_2025-09-05 -> origin/codex/add-helper-function-to-sizevars.py_2025-09-05 2025-09-07T06:42:00.6252764Z * [new branch] codex/add-metadata-field-for-file-path -> origin/codex/add-metadata-field-for-file-path 2025-09-07T06:42:00.6258050Z * [new branch] codex/add-test-for-inductor-local-cache-behavior -> origin/codex/add-test-for-inductor-local-cache-behavior 2025-09-07T06:42:00.6260350Z * [new branch] codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph 2025-09-07T06:42:00.6261109Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-09-07T06:42:00.6261601Z * [new branch] codex/fix-issue-160415-in-pytorch -> origin/codex/fix-issue-160415-in-pytorch 2025-09-07T06:42:00.6262192Z * [new branch] codex/fix-noqengine-quantized-engine-support -> origin/codex/fix-noqengine-quantized-engine-support 2025-09-07T06:42:00.6262806Z * [new branch] codex/fix-pin_memory-error-handling -> origin/codex/fix-pin_memory-error-handling 2025-09-07T06:42:00.6263333Z * [new branch] codex/propose-fix-for-issue-160332 -> origin/codex/propose-fix-for-issue-160332 2025-09-07T06:42:00.6263948Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-09-07T06:42:00.6264768Z * [new branch] codex/remove-allow-untyped-defs-and-fix-type-errors -> origin/codex/remove-allow-untyped-defs-and-fix-type-errors 2025-09-07T06:42:00.6265528Z * [new branch] compile_fsdp2_disable_stream_and_event -> origin/compile_fsdp2_disable_stream_and_event 2025-09-07T06:42:00.6266296Z * [new branch] context_test -> origin/context_test 2025-09-07T06:42:00.6266673Z * [new branch] copilot/fix-157446 -> origin/copilot/fix-157446 2025-09-07T06:42:00.6267025Z * [new branch] copy_graph -> origin/copy_graph 2025-09-07T06:42:00.6267383Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-09-07T06:42:00.6267762Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-09-07T06:42:00.6268163Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-09-07T06:42:00.6268560Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-09-07T06:42:00.6273345Z * [new branch] csl/disable_flaky_cpp_test -> origin/csl/disable_flaky_cpp_test 2025-09-07T06:42:00.6273847Z * [new branch] csl/disable_periodic_test -> origin/csl/disable_periodic_test 2025-09-07T06:42:00.6274275Z * [new branch] csl/exclude_rocm_viable_strict -> origin/csl/exclude_rocm_viable_strict 2025-09-07T06:42:00.6274886Z * [new branch] csl/katex -> origin/csl/katex 2025-09-07T06:42:00.6275244Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-09-07T06:42:00.6275639Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-09-07T06:42:00.6276017Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-09-07T06:42:00.6276385Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-09-07T06:42:00.6276780Z * [new branch] csl/name_link_check_job -> origin/csl/name_link_check_job 2025-09-07T06:42:00.6277164Z * [new branch] csl/no_keep_goin_rocm -> origin/csl/no_keep_goin_rocm 2025-09-07T06:42:00.6277520Z * [new branch] csl/not_600_timeout -> origin/csl/not_600_timeout 2025-09-07T06:42:00.6277882Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-09-07T06:42:00.6278234Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-09-07T06:42:00.6278640Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-09-07T06:42:00.6279043Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-09-07T06:42:00.6279403Z * [new branch] cublasltrelax2 -> origin/cublasltrelax2 2025-09-07T06:42:00.6279760Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-09-07T06:42:00.6280227Z * [new branch] cudnnsdparefactor -> origin/cudnnsdparefactor 2025-09-07T06:42:00.6280782Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-09-07T06:42:00.6288095Z * [new branch] czhuge_muon_dev -> origin/czhuge_muon_dev 2025-09-07T06:42:00.6288673Z * [new branch] d4l3k/delete_hook -> origin/d4l3k/delete_hook 2025-09-07T06:42:00.6289238Z * [new branch] dcp_zoc -> origin/dcp_zoc 2025-09-07T06:42:00.6289738Z * [new branch] debug-guard -> origin/debug-guard 2025-09-07T06:42:00.6290669Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-09-07T06:42:00.6291367Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.2 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.2 2025-09-07T06:42:00.6292250Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.3 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.3 2025-09-07T06:42:00.6293068Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.4 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.4 2025-09-07T06:42:00.6293869Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.56.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.56.0 2025-09-07T06:42:00.6294576Z * [new branch] dependabot/pip/dot-ci/docker/protobuf-5.29.5 -> origin/dependabot/pip/dot-ci/docker/protobuf-5.29.5 2025-09-07T06:42:00.6295247Z * [new branch] dependabot/pip/dot-github/requirements/protobuf-5.29.5 -> origin/dependabot/pip/dot-github/requirements/protobuf-5.29.5 2025-09-07T06:42:00.6295827Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-09-07T06:42:00.6296283Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-09-07T06:42:00.6296729Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-09-07T06:42:00.6297116Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-09-07T06:42:00.6297523Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-09-07T06:42:00.6298057Z * [new branch] dev/joona/cat_remove_graph -> origin/dev/joona/cat_remove_graph 2025-09-07T06:42:00.6298456Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-09-07T06:42:00.6298912Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-09-07T06:42:00.6300529Z * [new branch] dev/joona/maxpool2dwithindices_errmsg -> origin/dev/joona/maxpool2dwithindices_errmsg 2025-09-07T06:42:00.6301037Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-09-07T06:42:00.6302520Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-09-07T06:42:00.6303285Z * [new branch] dev/joona/topk_newapi -> origin/dev/joona/topk_newapi 2025-09-07T06:42:00.6304084Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-09-07T06:42:00.6304925Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-09-07T06:42:00.6306002Z * [new branch] disable -> origin/disable 2025-09-07T06:42:00.6306644Z * [new branch] e2e-baseline -> origin/e2e-baseline 2025-09-07T06:42:00.6307465Z * [new branch] eigen_for_sparse_addmm_v2 -> origin/eigen_for_sparse_addmm_v2 2025-09-07T06:42:00.6308593Z * [new branch] embg/test_inductor_ci_128B -> origin/embg/test_inductor_ci_128B 2025-09-07T06:42:00.6309084Z * [new branch] embg/test_inductor_ci_base -> origin/embg/test_inductor_ci_base 2025-09-07T06:42:00.6310008Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-09-07T06:42:00.6310463Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-09-07T06:42:00.6311655Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-09-07T06:42:00.6312322Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-09-07T06:42:00.6313028Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-09-07T06:42:00.6313697Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-09-07T06:42:00.6314534Z * [new branch] eqy-patch-4 -> origin/eqy-patch-4 2025-09-07T06:42:00.6315300Z * [new branch] example-convert-torch.nn -> origin/example-convert-torch.nn 2025-09-07T06:42:00.6316680Z * [new branch] exclamaforte/add-contiguous-threshold -> origin/exclamaforte/add-contiguous-threshold 2025-09-07T06:42:00.6317151Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-09-07T06:42:00.6317662Z * [new branch] exclamaforte/bump-transformer-version -> origin/exclamaforte/bump-transformer-version 2025-09-07T06:42:00.6318439Z * [new branch] exclamaforte/clear-feedback-savers -> origin/exclamaforte/clear-feedback-savers 2025-09-07T06:42:00.6318977Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-09-07T06:42:00.6320312Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-09-07T06:42:00.6320918Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-09-07T06:42:00.6321799Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-09-07T06:42:00.6322590Z * [new branch] exclamaforte/fix-exhuastive-autotuning-reland -> origin/exclamaforte/fix-exhuastive-autotuning-reland 2025-09-07T06:42:00.6323142Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-09-07T06:42:00.6323894Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-09-07T06:42:00.6324516Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-09-07T06:42:00.6325286Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-09-07T06:42:00.6325863Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-09-07T06:42:00.6326455Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-09-07T06:42:00.6327292Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-09-07T06:42:00.6327774Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-09-07T06:42:00.6328565Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-09-07T06:42:00.6329357Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-09-07T06:42:00.6329840Z * [new branch] exclamaforte/max-autotune-ieee -> origin/exclamaforte/max-autotune-ieee 2025-09-07T06:42:00.6330357Z * [new branch] exclamaforte/memory-counter -> origin/exclamaforte/memory-counter 2025-09-07T06:42:00.6331098Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-09-07T06:42:00.6331742Z * [new branch] exclamaforte/profiler-combo -> origin/exclamaforte/profiler-combo 2025-09-07T06:42:00.6332545Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-09-07T06:42:00.6333039Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-09-07T06:42:00.6333806Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-09-07T06:42:00.6335054Z * [new branch] exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final 2025-09-07T06:42:00.6335411Z * [new branch] exec -> origin/exec 2025-09-07T06:42:00.6336322Z * [new branch] executorch-module-shim -> origin/executorch-module-shim 2025-09-07T06:42:00.6336934Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-09-07T06:42:00.6338272Z * [new branch] export-D58091437 -> origin/export-D58091437 2025-09-07T06:42:00.6338963Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-09-07T06:42:00.6339521Z * [new branch] export-D70112642 -> origin/export-D70112642 2025-09-07T06:42:00.6340254Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-09-07T06:42:00.6341410Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-09-07T06:42:00.6341756Z * [new branch] export-D75183591 -> origin/export-D75183591 2025-09-07T06:42:00.6342806Z * [new branch] export-D75617432 -> origin/export-D75617432 2025-09-07T06:42:00.6343269Z * [new branch] export-D75659965 -> origin/export-D75659965 2025-09-07T06:42:00.6343946Z * [new branch] export-D76080931 -> origin/export-D76080931 2025-09-07T06:42:00.6344637Z * [new branch] export-D76797250 -> origin/export-D76797250 2025-09-07T06:42:00.6345340Z * [new branch] export-D76885271 -> origin/export-D76885271 2025-09-07T06:42:00.6346153Z * [new branch] export-D76885620 -> origin/export-D76885620 2025-09-07T06:42:00.6347072Z * [new branch] export-D76936623 -> origin/export-D76936623 2025-09-07T06:42:00.6347705Z * [new branch] export-D76958268 -> origin/export-D76958268 2025-09-07T06:42:00.6348450Z * [new branch] export-D78375400 -> origin/export-D78375400 2025-09-07T06:42:00.6349099Z * [new branch] export-D78431305 -> origin/export-D78431305 2025-09-07T06:42:00.6349837Z * [new branch] export-D78580107 -> origin/export-D78580107 2025-09-07T06:42:00.6350503Z * [new branch] export-D78822171 -> origin/export-D78822171 2025-09-07T06:42:00.6351186Z * [new branch] export-D78822351 -> origin/export-D78822351 2025-09-07T06:42:00.6352105Z * [new branch] export-D78822507 -> origin/export-D78822507 2025-09-07T06:42:00.6352695Z * [new branch] export-D78826994 -> origin/export-D78826994 2025-09-07T06:42:00.6353365Z * [new branch] export-D78894324 -> origin/export-D78894324 2025-09-07T06:42:00.6353983Z * [new branch] export-D78929245 -> origin/export-D78929245 2025-09-07T06:42:00.6354745Z * [new branch] export-D78934925 -> origin/export-D78934925 2025-09-07T06:42:00.6356239Z * [new branch] export-D78953203 -> origin/export-D78953203 2025-09-07T06:42:00.6357177Z * [new branch] export-D78953229 -> origin/export-D78953229 2025-09-07T06:42:00.6357800Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-09-07T06:42:00.6358309Z * [new branch] export-D78957389 -> origin/export-D78957389 2025-09-07T06:42:00.6359451Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-09-07T06:42:00.6359910Z * [new branch] export-D79026433 -> origin/export-D79026433 2025-09-07T06:42:00.6360264Z * [new branch] export-D79230339 -> origin/export-D79230339 2025-09-07T06:42:00.6360991Z * [new branch] export-D79319835 -> origin/export-D79319835 2025-09-07T06:42:00.6361402Z * [new branch] export-D79328456 -> origin/export-D79328456 2025-09-07T06:42:00.6362429Z * [new branch] export-D79534608 -> origin/export-D79534608 2025-09-07T06:42:00.6363143Z * [new branch] export-D79785974 -> origin/export-D79785974 2025-09-07T06:42:00.6363927Z * [new branch] export-D80025417 -> origin/export-D80025417 2025-09-07T06:42:00.6364592Z * [new branch] export-D80120333 -> origin/export-D80120333 2025-09-07T06:42:00.6365566Z * [new branch] export-D80214882 -> origin/export-D80214882 2025-09-07T06:42:00.6365942Z * [new branch] export-D80319069 -> origin/export-D80319069 2025-09-07T06:42:00.6366714Z * [new branch] export-D80321215 -> origin/export-D80321215 2025-09-07T06:42:00.6367289Z * [new branch] export-D80503451 -> origin/export-D80503451 2025-09-07T06:42:00.6367969Z * [new branch] export-D80771648 -> origin/export-D80771648 2025-09-07T06:42:00.6368654Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-09-07T06:42:00.6369360Z * [new branch] export-D80948073 -> origin/export-D80948073 2025-09-07T06:42:00.6370054Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-09-07T06:42:00.6370725Z * [new branch] export-D80970483 -> origin/export-D80970483 2025-09-07T06:42:00.6371282Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-09-07T06:42:00.6372104Z * [new branch] export-D81060182 -> origin/export-D81060182 2025-09-07T06:42:00.6372849Z * [new branch] export-D81078973 -> origin/export-D81078973 2025-09-07T06:42:00.6373512Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-09-07T06:42:00.6374215Z * [new branch] export-D81284190 -> origin/export-D81284190 2025-09-07T06:42:00.6374917Z * [new branch] export-D81299840 -> origin/export-D81299840 2025-09-07T06:42:00.6375717Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-09-07T06:42:00.6376390Z * [new branch] export-D81698719 -> origin/export-D81698719 2025-09-07T06:42:00.6378423Z * [new branch] export-D81747409 -> origin/export-D81747409 2025-09-07T06:42:00.6378931Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-09-07T06:42:00.6379669Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-09-07T06:42:00.6380085Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-09-07T06:42:00.6380787Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-09-07T06:42:00.6382358Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-09-07T06:42:00.6382869Z * [new branch] fca -> origin/fca 2025-09-07T06:42:00.6383740Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-09-07T06:42:00.6384805Z * [new branch] fca5 -> origin/fca5 2025-09-07T06:42:00.6386158Z * [new branch] feature/function-numa-binding -> origin/feature/function-numa-binding 2025-09-07T06:42:00.6386835Z * [new branch] feature/function-numa-binding-take2 -> origin/feature/function-numa-binding-take2 2025-09-07T06:42:00.6387343Z * [new branch] feature/numa-nproc-fix -> origin/feature/numa-nproc-fix 2025-09-07T06:42:00.6387990Z * [new branch] feature/numa-signpost-serialize -> origin/feature/numa-signpost-serialize 2025-09-07T06:42:00.6388712Z * [new branch] feature/parallel-numa-binding -> origin/feature/parallel-numa-binding 2025-09-07T06:42:00.6389942Z * [new branch] fengyuan/external-proj -> origin/fengyuan/external-proj 2025-09-07T06:42:00.6390783Z * [new branch] fengyuan/out-of-tree-xpu-ops-improve-test -> origin/fengyuan/out-of-tree-xpu-ops-improve-test 2025-09-07T06:42:00.6391405Z * [new branch] fengyuan/out-of-tree-xpu-ops-remove-dtype -> origin/fengyuan/out-of-tree-xpu-ops-remove-dtype 2025-09-07T06:42:00.6391884Z * [new branch] fengyuan/test-xpu -> origin/fengyuan/test-xpu 2025-09-07T06:42:00.6393225Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-09-07T06:42:00.6396833Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-09-07T06:42:00.6397197Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-09-07T06:42:00.6397559Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-09-07T06:42:00.6397927Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-09-07T06:42:00.6398301Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-09-07T06:42:00.6398664Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-09-07T06:42:00.6399045Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-09-07T06:42:00.6399426Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-09-07T06:42:00.6399798Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-09-07T06:42:00.6400139Z * [new branch] fix -> origin/fix 2025-09-07T06:42:00.6401337Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-09-07T06:42:00.6401730Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-09-07T06:42:00.6402444Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-09-07T06:42:00.6403189Z * [new branch] fix-inductor-periodic-0528 -> origin/fix-inductor-periodic-0528 2025-09-07T06:42:00.6403789Z * [new branch] fix-mps-benchmark -> origin/fix-mps-benchmark 2025-09-07T06:42:00.6404584Z * [new branch] fix-rlease-feature-template -> origin/fix-rlease-feature-template 2025-09-07T06:42:00.6405356Z * [new branch] fix-run-condition-upload-results -> origin/fix-run-condition-upload-results 2025-09-07T06:42:00.6405931Z * [new branch] fix-torchbench -> origin/fix-torchbench 2025-09-07T06:42:00.6406636Z * [new branch] fix_153389 -> origin/fix_153389 2025-09-07T06:42:00.6407335Z * [new branch] fix_fsdp_rs_bucket2 -> origin/fix_fsdp_rs_bucket2 2025-09-07T06:42:00.6408462Z * [new branch] fix_inductor_peridic_tests -> origin/fix_inductor_peridic_tests 2025-09-07T06:42:00.6408827Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-09-07T06:42:00.6409527Z * [new branch] fixes-triage -> origin/fixes-triage 2025-09-07T06:42:00.6410217Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-09-07T06:42:00.6410934Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-09-07T06:42:00.6411648Z * [new branch] flex-flash -> origin/flex-flash 2025-09-07T06:42:00.6412346Z * [new branch] flex-lowering -> origin/flex-lowering 2025-09-07T06:42:00.6413058Z * [new branch] flex-warning -> origin/flex-warning 2025-09-07T06:42:00.6413751Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-09-07T06:42:00.6414989Z * [new branch] flex_flash -> origin/flex_flash 2025-09-07T06:42:00.6415606Z * [new branch] flexdecode-gqa-groups -> origin/flexdecode-gqa-groups 2025-09-07T06:42:00.6416861Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-09-07T06:42:00.6417278Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-09-07T06:42:00.6418351Z * [new branch] fsdpv2_3d -> origin/fsdpv2_3d 2025-09-07T06:42:00.6418930Z * [new branch] fsdpv2_3d_m1 -> origin/fsdpv2_3d_m1 2025-09-07T06:42:00.6420415Z * [new branch] fx_cpp -> origin/fx_cpp 2025-09-07T06:42:00.6421675Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-09-07T06:42:00.6423961Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-09-07T06:42:00.6424319Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-09-07T06:42:00.6426507Z * [new branch] gh/CaoE/2/base -> origin/gh/CaoE/2/base 2025-09-07T06:42:00.6426924Z * [new branch] gh/CaoE/2/head -> origin/gh/CaoE/2/head 2025-09-07T06:42:00.6427261Z * [new branch] gh/CaoE/2/orig -> origin/gh/CaoE/2/orig 2025-09-07T06:42:00.6431696Z * [new branch] gh/ColinPeppler/79/base -> origin/gh/ColinPeppler/79/base 2025-09-07T06:42:00.6432215Z * [new branch] gh/ColinPeppler/79/head -> origin/gh/ColinPeppler/79/head 2025-09-07T06:42:00.6432620Z * [new branch] gh/ColinPeppler/79/orig -> origin/gh/ColinPeppler/79/orig 2025-09-07T06:42:00.6433029Z * [new branch] gh/ColinPeppler/80/base -> origin/gh/ColinPeppler/80/base 2025-09-07T06:42:00.6433564Z * [new branch] gh/ColinPeppler/80/head -> origin/gh/ColinPeppler/80/head 2025-09-07T06:42:00.6434381Z * [new branch] gh/ColinPeppler/80/orig -> origin/gh/ColinPeppler/80/orig 2025-09-07T06:42:00.6434866Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-09-07T06:42:00.6435259Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-09-07T06:42:00.6435631Z * [new branch] gh/EikanWang/80/base -> origin/gh/EikanWang/80/base 2025-09-07T06:42:00.6436469Z * [new branch] gh/EikanWang/80/head -> origin/gh/EikanWang/80/head 2025-09-07T06:42:00.6437116Z * [new branch] gh/EikanWang/80/orig -> origin/gh/EikanWang/80/orig 2025-09-07T06:42:00.6444655Z * [new branch] gh/EikanWang/81/base -> origin/gh/EikanWang/81/base 2025-09-07T06:42:00.6450791Z * [new branch] gh/EikanWang/81/head -> origin/gh/EikanWang/81/head 2025-09-07T06:42:00.6459601Z * [new branch] gh/EikanWang/81/orig -> origin/gh/EikanWang/81/orig 2025-09-07T06:42:00.6460319Z * [new branch] gh/EikanWang/82/base -> origin/gh/EikanWang/82/base 2025-09-07T06:42:00.6460749Z * [new branch] gh/EikanWang/82/head -> origin/gh/EikanWang/82/head 2025-09-07T06:42:00.6461165Z * [new branch] gh/EikanWang/82/orig -> origin/gh/EikanWang/82/orig 2025-09-07T06:42:00.6461596Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-09-07T06:42:00.6462003Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-09-07T06:42:00.6462642Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-09-07T06:42:00.6463007Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-09-07T06:42:00.6463353Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-09-07T06:42:00.6463724Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-09-07T06:42:00.6464094Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-09-07T06:42:00.6464454Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-09-07T06:42:00.6464817Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-09-07T06:42:00.6465174Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-09-07T06:42:00.6465537Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-09-07T06:42:00.6466219Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-09-07T06:42:00.6466590Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-09-07T06:42:00.6466952Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-09-07T06:42:00.6467305Z * [new branch] gh/H-Huang/187/base -> origin/gh/H-Huang/187/base 2025-09-07T06:42:00.6467665Z * [new branch] gh/H-Huang/187/head -> origin/gh/H-Huang/187/head 2025-09-07T06:42:00.6468004Z * [new branch] gh/H-Huang/187/orig -> origin/gh/H-Huang/187/orig 2025-09-07T06:42:00.6468340Z * [new branch] gh/H-Huang/202/base -> origin/gh/H-Huang/202/base 2025-09-07T06:42:00.6468681Z * [new branch] gh/H-Huang/202/head -> origin/gh/H-Huang/202/head 2025-09-07T06:42:00.6469010Z * [new branch] gh/H-Huang/202/orig -> origin/gh/H-Huang/202/orig 2025-09-07T06:42:00.6469352Z * [new branch] gh/H-Huang/203/base -> origin/gh/H-Huang/203/base 2025-09-07T06:42:00.6469691Z * [new branch] gh/H-Huang/203/head -> origin/gh/H-Huang/203/head 2025-09-07T06:42:00.6470029Z * [new branch] gh/H-Huang/203/orig -> origin/gh/H-Huang/203/orig 2025-09-07T06:42:00.6470384Z * [new branch] gh/H-Huang/204/base -> origin/gh/H-Huang/204/base 2025-09-07T06:42:00.6470783Z * [new branch] gh/H-Huang/204/head -> origin/gh/H-Huang/204/head 2025-09-07T06:42:00.6471124Z * [new branch] gh/H-Huang/204/orig -> origin/gh/H-Huang/204/orig 2025-09-07T06:42:00.6471459Z * [new branch] gh/H-Huang/205/base -> origin/gh/H-Huang/205/base 2025-09-07T06:42:00.6471793Z * [new branch] gh/H-Huang/205/head -> origin/gh/H-Huang/205/head 2025-09-07T06:42:00.6472137Z * [new branch] gh/H-Huang/205/orig -> origin/gh/H-Huang/205/orig 2025-09-07T06:42:00.6472468Z * [new branch] gh/H-Huang/206/base -> origin/gh/H-Huang/206/base 2025-09-07T06:42:00.6472808Z * [new branch] gh/H-Huang/206/head -> origin/gh/H-Huang/206/head 2025-09-07T06:42:00.6473147Z * [new branch] gh/H-Huang/206/orig -> origin/gh/H-Huang/206/orig 2025-09-07T06:42:00.6473489Z * [new branch] gh/H-Huang/207/base -> origin/gh/H-Huang/207/base 2025-09-07T06:42:00.6473826Z * [new branch] gh/H-Huang/207/head -> origin/gh/H-Huang/207/head 2025-09-07T06:42:00.6474160Z * [new branch] gh/H-Huang/207/orig -> origin/gh/H-Huang/207/orig 2025-09-07T06:42:00.6474508Z * [new branch] gh/H-Huang/208/base -> origin/gh/H-Huang/208/base 2025-09-07T06:42:00.6474843Z * [new branch] gh/H-Huang/208/head -> origin/gh/H-Huang/208/head 2025-09-07T06:42:00.6475227Z * [new branch] gh/H-Huang/208/orig -> origin/gh/H-Huang/208/orig 2025-09-07T06:42:00.6475567Z * [new branch] gh/H-Huang/209/base -> origin/gh/H-Huang/209/base 2025-09-07T06:42:00.6475904Z * [new branch] gh/H-Huang/209/head -> origin/gh/H-Huang/209/head 2025-09-07T06:42:00.6476241Z * [new branch] gh/H-Huang/209/orig -> origin/gh/H-Huang/209/orig 2025-09-07T06:42:00.6476918Z * [new branch] gh/H-Huang/210/base -> origin/gh/H-Huang/210/base 2025-09-07T06:42:00.6477553Z * [new branch] gh/H-Huang/210/head -> origin/gh/H-Huang/210/head 2025-09-07T06:42:00.6478222Z * [new branch] gh/H-Huang/210/orig -> origin/gh/H-Huang/210/orig 2025-09-07T06:42:00.6480020Z * [new branch] gh/H-Huang/211/base -> origin/gh/H-Huang/211/base 2025-09-07T06:42:00.6480365Z * [new branch] gh/H-Huang/211/head -> origin/gh/H-Huang/211/head 2025-09-07T06:42:00.6480717Z * [new branch] gh/H-Huang/211/orig -> origin/gh/H-Huang/211/orig 2025-09-07T06:42:00.6482570Z * [new branch] gh/H-Huang/212/base -> origin/gh/H-Huang/212/base 2025-09-07T06:42:00.6482909Z * [new branch] gh/H-Huang/212/head -> origin/gh/H-Huang/212/head 2025-09-07T06:42:00.6483237Z * [new branch] gh/H-Huang/212/orig -> origin/gh/H-Huang/212/orig 2025-09-07T06:42:00.6486661Z * [new branch] gh/H-Huang/213/base -> origin/gh/H-Huang/213/base 2025-09-07T06:42:00.6486992Z * [new branch] gh/H-Huang/213/head -> origin/gh/H-Huang/213/head 2025-09-07T06:42:00.6487320Z * [new branch] gh/H-Huang/213/orig -> origin/gh/H-Huang/213/orig 2025-09-07T06:42:00.6487661Z * [new branch] gh/H-Huang/214/base -> origin/gh/H-Huang/214/base 2025-09-07T06:42:00.6488001Z * [new branch] gh/H-Huang/214/head -> origin/gh/H-Huang/214/head 2025-09-07T06:42:00.6488344Z * [new branch] gh/H-Huang/214/orig -> origin/gh/H-Huang/214/orig 2025-09-07T06:42:00.6492090Z * [new branch] gh/IvanKobzarev/112/base -> origin/gh/IvanKobzarev/112/base 2025-09-07T06:42:00.6492491Z * [new branch] gh/IvanKobzarev/112/head -> origin/gh/IvanKobzarev/112/head 2025-09-07T06:42:00.6492885Z * [new branch] gh/IvanKobzarev/112/orig -> origin/gh/IvanKobzarev/112/orig 2025-09-07T06:42:00.6493359Z * [new branch] gh/IvanKobzarev/115/base -> origin/gh/IvanKobzarev/115/base 2025-09-07T06:42:00.6493722Z * [new branch] gh/IvanKobzarev/115/head -> origin/gh/IvanKobzarev/115/head 2025-09-07T06:42:00.6494072Z * [new branch] gh/IvanKobzarev/115/orig -> origin/gh/IvanKobzarev/115/orig 2025-09-07T06:42:00.6494436Z * [new branch] gh/IvanKobzarev/116/base -> origin/gh/IvanKobzarev/116/base 2025-09-07T06:42:00.6494816Z * [new branch] gh/IvanKobzarev/116/head -> origin/gh/IvanKobzarev/116/head 2025-09-07T06:42:00.6495199Z * [new branch] gh/IvanKobzarev/116/orig -> origin/gh/IvanKobzarev/116/orig 2025-09-07T06:42:00.6498969Z * [new branch] gh/IvanKobzarev/118/base -> origin/gh/IvanKobzarev/118/base 2025-09-07T06:42:00.6499344Z * [new branch] gh/IvanKobzarev/118/head -> origin/gh/IvanKobzarev/118/head 2025-09-07T06:42:00.6499736Z * [new branch] gh/IvanKobzarev/118/orig -> origin/gh/IvanKobzarev/118/orig 2025-09-07T06:42:00.6500114Z * [new branch] gh/IvanKobzarev/126/base -> origin/gh/IvanKobzarev/126/base 2025-09-07T06:42:00.6500494Z * [new branch] gh/IvanKobzarev/126/head -> origin/gh/IvanKobzarev/126/head 2025-09-07T06:42:00.6500870Z * [new branch] gh/IvanKobzarev/126/orig -> origin/gh/IvanKobzarev/126/orig 2025-09-07T06:42:00.6501347Z * [new branch] gh/IvanKobzarev/127/base -> origin/gh/IvanKobzarev/127/base 2025-09-07T06:42:00.6501788Z * [new branch] gh/IvanKobzarev/127/head -> origin/gh/IvanKobzarev/127/head 2025-09-07T06:42:00.6502169Z * [new branch] gh/IvanKobzarev/127/orig -> origin/gh/IvanKobzarev/127/orig 2025-09-07T06:42:00.6502551Z * [new branch] gh/IvanKobzarev/128/base -> origin/gh/IvanKobzarev/128/base 2025-09-07T06:42:00.6502956Z * [new branch] gh/IvanKobzarev/128/head -> origin/gh/IvanKobzarev/128/head 2025-09-07T06:42:00.6503699Z * [new branch] gh/IvanKobzarev/128/orig -> origin/gh/IvanKobzarev/128/orig 2025-09-07T06:42:00.6504968Z * [new branch] gh/IvanKobzarev/132/base -> origin/gh/IvanKobzarev/132/base 2025-09-07T06:42:00.6505494Z * [new branch] gh/IvanKobzarev/132/head -> origin/gh/IvanKobzarev/132/head 2025-09-07T06:42:00.6506285Z * [new branch] gh/IvanKobzarev/132/orig -> origin/gh/IvanKobzarev/132/orig 2025-09-07T06:42:00.6508087Z * [new branch] gh/IvanKobzarev/133/base -> origin/gh/IvanKobzarev/133/base 2025-09-07T06:42:00.6508743Z * [new branch] gh/IvanKobzarev/133/head -> origin/gh/IvanKobzarev/133/head 2025-09-07T06:42:00.6509499Z * [new branch] gh/IvanKobzarev/133/orig -> origin/gh/IvanKobzarev/133/orig 2025-09-07T06:42:00.6510616Z * [new branch] gh/IvanKobzarev/134/base -> origin/gh/IvanKobzarev/134/base 2025-09-07T06:42:00.6511018Z * [new branch] gh/IvanKobzarev/134/head -> origin/gh/IvanKobzarev/134/head 2025-09-07T06:42:00.6511900Z * [new branch] gh/IvanKobzarev/134/orig -> origin/gh/IvanKobzarev/134/orig 2025-09-07T06:42:00.6513328Z * [new branch] gh/IvanKobzarev/135/base -> origin/gh/IvanKobzarev/135/base 2025-09-07T06:42:00.6513718Z * [new branch] gh/IvanKobzarev/135/head -> origin/gh/IvanKobzarev/135/head 2025-09-07T06:42:00.6514509Z * [new branch] gh/IvanKobzarev/135/orig -> origin/gh/IvanKobzarev/135/orig 2025-09-07T06:42:00.6515860Z * [new branch] gh/IvanKobzarev/136/base -> origin/gh/IvanKobzarev/136/base 2025-09-07T06:42:00.6516249Z * [new branch] gh/IvanKobzarev/136/head -> origin/gh/IvanKobzarev/136/head 2025-09-07T06:42:00.6517323Z * [new branch] gh/IvanKobzarev/136/orig -> origin/gh/IvanKobzarev/136/orig 2025-09-07T06:42:00.6517956Z * [new branch] gh/IvanKobzarev/137/base -> origin/gh/IvanKobzarev/137/base 2025-09-07T06:42:00.6518645Z * [new branch] gh/IvanKobzarev/137/head -> origin/gh/IvanKobzarev/137/head 2025-09-07T06:42:00.6519923Z * [new branch] gh/IvanKobzarev/137/orig -> origin/gh/IvanKobzarev/137/orig 2025-09-07T06:42:00.6520452Z * [new branch] gh/IvanKobzarev/138/base -> origin/gh/IvanKobzarev/138/base 2025-09-07T06:42:00.6521764Z * [new branch] gh/IvanKobzarev/138/head -> origin/gh/IvanKobzarev/138/head 2025-09-07T06:42:00.6522385Z * [new branch] gh/IvanKobzarev/138/orig -> origin/gh/IvanKobzarev/138/orig 2025-09-07T06:42:00.6523675Z * [new branch] gh/IvanKobzarev/139/base -> origin/gh/IvanKobzarev/139/base 2025-09-07T06:42:00.6524075Z * [new branch] gh/IvanKobzarev/139/head -> origin/gh/IvanKobzarev/139/head 2025-09-07T06:42:00.6524758Z * [new branch] gh/IvanKobzarev/139/orig -> origin/gh/IvanKobzarev/139/orig 2025-09-07T06:42:00.6526354Z * [new branch] gh/IvanKobzarev/140/base -> origin/gh/IvanKobzarev/140/base 2025-09-07T06:42:00.6526715Z * [new branch] gh/IvanKobzarev/140/head -> origin/gh/IvanKobzarev/140/head 2025-09-07T06:42:00.6527086Z * [new branch] gh/IvanKobzarev/140/orig -> origin/gh/IvanKobzarev/140/orig 2025-09-07T06:42:00.6529160Z * [new branch] gh/IvanKobzarev/141/base -> origin/gh/IvanKobzarev/141/base 2025-09-07T06:42:00.6529667Z * [new branch] gh/IvanKobzarev/141/head -> origin/gh/IvanKobzarev/141/head 2025-09-07T06:42:00.6530278Z * [new branch] gh/IvanKobzarev/141/orig -> origin/gh/IvanKobzarev/141/orig 2025-09-07T06:42:00.6531486Z * [new branch] gh/IvanKobzarev/142/base -> origin/gh/IvanKobzarev/142/base 2025-09-07T06:42:00.6531949Z * [new branch] gh/IvanKobzarev/142/head -> origin/gh/IvanKobzarev/142/head 2025-09-07T06:42:00.6532616Z * [new branch] gh/IvanKobzarev/142/orig -> origin/gh/IvanKobzarev/142/orig 2025-09-07T06:42:00.6535069Z * [new branch] gh/IvanKobzarev/143/base -> origin/gh/IvanKobzarev/143/base 2025-09-07T06:42:00.6535584Z * [new branch] gh/IvanKobzarev/143/head -> origin/gh/IvanKobzarev/143/head 2025-09-07T06:42:00.6536004Z * [new branch] gh/IvanKobzarev/143/orig -> origin/gh/IvanKobzarev/143/orig 2025-09-07T06:42:00.6536669Z * [new branch] gh/IvanKobzarev/144/base -> origin/gh/IvanKobzarev/144/base 2025-09-07T06:42:00.6537130Z * [new branch] gh/IvanKobzarev/144/head -> origin/gh/IvanKobzarev/144/head 2025-09-07T06:42:00.6537846Z * [new branch] gh/IvanKobzarev/144/orig -> origin/gh/IvanKobzarev/144/orig 2025-09-07T06:42:00.6540167Z * [new branch] gh/IvanKobzarev/145/base -> origin/gh/IvanKobzarev/145/base 2025-09-07T06:42:00.6540783Z * [new branch] gh/IvanKobzarev/145/head -> origin/gh/IvanKobzarev/145/head 2025-09-07T06:42:00.6541213Z * [new branch] gh/IvanKobzarev/145/orig -> origin/gh/IvanKobzarev/145/orig 2025-09-07T06:42:00.6541615Z * [new branch] gh/IvanKobzarev/146/base -> origin/gh/IvanKobzarev/146/base 2025-09-07T06:42:00.6542039Z * [new branch] gh/IvanKobzarev/146/head -> origin/gh/IvanKobzarev/146/head 2025-09-07T06:42:00.6542815Z * [new branch] gh/IvanKobzarev/146/orig -> origin/gh/IvanKobzarev/146/orig 2025-09-07T06:42:00.6544238Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-09-07T06:42:00.6544902Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-09-07T06:42:00.6546222Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-09-07T06:42:00.6546629Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-09-07T06:42:00.6548039Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-09-07T06:42:00.6548791Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-09-07T06:42:00.6552684Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2025-09-07T06:42:00.6553106Z * [new branch] gh/PaliC/1/head -> origin/gh/PaliC/1/head 2025-09-07T06:42:00.6553502Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2025-09-07T06:42:00.6553908Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-09-07T06:42:00.6554332Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-09-07T06:42:00.6554696Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-09-07T06:42:00.6555059Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-09-07T06:42:00.6555411Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-09-07T06:42:00.6555849Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-09-07T06:42:00.6559322Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2025-09-07T06:42:00.6559923Z * [new branch] gh/PaliC/2/head -> origin/gh/PaliC/2/head 2025-09-07T06:42:00.6560436Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2025-09-07T06:42:00.6561364Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-09-07T06:42:00.6561975Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-09-07T06:42:00.6562367Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-09-07T06:42:00.6562732Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-09-07T06:42:00.6563135Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-09-07T06:42:00.6563689Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-09-07T06:42:00.6564493Z * [new branch] gh/PaliC/22/base -> origin/gh/PaliC/22/base 2025-09-07T06:42:00.6564924Z * [new branch] gh/PaliC/22/head -> origin/gh/PaliC/22/head 2025-09-07T06:42:00.6565477Z * [new branch] gh/PaliC/22/orig -> origin/gh/PaliC/22/orig 2025-09-07T06:42:00.6565850Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-09-07T06:42:00.6566386Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-09-07T06:42:00.6566763Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-09-07T06:42:00.6567466Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-09-07T06:42:00.6568078Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-09-07T06:42:00.6568728Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-09-07T06:42:00.6573522Z * [new branch] gh/PaulZhang12/17/base -> origin/gh/PaulZhang12/17/base 2025-09-07T06:42:00.6573951Z * [new branch] gh/PaulZhang12/17/head -> origin/gh/PaulZhang12/17/head 2025-09-07T06:42:00.6574349Z * [new branch] gh/PaulZhang12/20/base -> origin/gh/PaulZhang12/20/base 2025-09-07T06:42:00.6574769Z * [new branch] gh/PaulZhang12/20/head -> origin/gh/PaulZhang12/20/head 2025-09-07T06:42:00.6575160Z * [new branch] gh/PaulZhang12/20/orig -> origin/gh/PaulZhang12/20/orig 2025-09-07T06:42:00.6575550Z * [new branch] gh/PaulZhang12/21/base -> origin/gh/PaulZhang12/21/base 2025-09-07T06:42:00.6575944Z * [new branch] gh/PaulZhang12/21/head -> origin/gh/PaulZhang12/21/head 2025-09-07T06:42:00.6576502Z * [new branch] gh/PaulZhang12/21/orig -> origin/gh/PaulZhang12/21/orig 2025-09-07T06:42:00.6576898Z * [new branch] gh/PaulZhang12/22/base -> origin/gh/PaulZhang12/22/base 2025-09-07T06:42:00.6577284Z * [new branch] gh/PaulZhang12/22/head -> origin/gh/PaulZhang12/22/head 2025-09-07T06:42:00.6577671Z * [new branch] gh/PaulZhang12/22/orig -> origin/gh/PaulZhang12/22/orig 2025-09-07T06:42:00.6578395Z * [new branch] gh/PaulZhang12/23/base -> origin/gh/PaulZhang12/23/base 2025-09-07T06:42:00.6579069Z * [new branch] gh/PaulZhang12/23/head -> origin/gh/PaulZhang12/23/head 2025-09-07T06:42:00.6581004Z * [new branch] gh/PaulZhang12/23/orig -> origin/gh/PaulZhang12/23/orig 2025-09-07T06:42:00.6581472Z * [new branch] gh/PaulZhang12/24/base -> origin/gh/PaulZhang12/24/base 2025-09-07T06:42:00.6581866Z * [new branch] gh/PaulZhang12/24/head -> origin/gh/PaulZhang12/24/head 2025-09-07T06:42:00.6582299Z * [new branch] gh/PaulZhang12/24/orig -> origin/gh/PaulZhang12/24/orig 2025-09-07T06:42:00.6583061Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-09-07T06:42:00.6583724Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-09-07T06:42:00.6584973Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-09-07T06:42:00.6586247Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-09-07T06:42:00.6586878Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-09-07T06:42:00.6588788Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-09-07T06:42:00.6589515Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-09-07T06:42:00.6590715Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-09-07T06:42:00.6591431Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-09-07T06:42:00.6592833Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-09-07T06:42:00.6593231Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-09-07T06:42:00.6594540Z * [new branch] gh/StrongerXi/133/base -> origin/gh/StrongerXi/133/base 2025-09-07T06:42:00.6594995Z * [new branch] gh/StrongerXi/133/head -> origin/gh/StrongerXi/133/head 2025-09-07T06:42:00.6595715Z * [new branch] gh/StrongerXi/133/orig -> origin/gh/StrongerXi/133/orig 2025-09-07T06:42:00.6596947Z * [new branch] gh/StrongerXi/134/base -> origin/gh/StrongerXi/134/base 2025-09-07T06:42:00.6597295Z * [new branch] gh/StrongerXi/134/head -> origin/gh/StrongerXi/134/head 2025-09-07T06:42:00.6597928Z * [new branch] gh/StrongerXi/134/orig -> origin/gh/StrongerXi/134/orig 2025-09-07T06:42:00.6598999Z * [new branch] gh/StrongerXi/136/base -> origin/gh/StrongerXi/136/base 2025-09-07T06:42:00.6599353Z * [new branch] gh/StrongerXi/136/head -> origin/gh/StrongerXi/136/head 2025-09-07T06:42:00.6600048Z * [new branch] gh/StrongerXi/136/orig -> origin/gh/StrongerXi/136/orig 2025-09-07T06:42:00.6601126Z * [new branch] gh/StrongerXi/137/base -> origin/gh/StrongerXi/137/base 2025-09-07T06:42:00.6601496Z * [new branch] gh/StrongerXi/137/head -> origin/gh/StrongerXi/137/head 2025-09-07T06:42:00.6605670Z * [new branch] gh/StrongerXi/137/orig -> origin/gh/StrongerXi/137/orig 2025-09-07T06:42:00.6606000Z * [new branch] gh/StrongerXi/138/base -> origin/gh/StrongerXi/138/base 2025-09-07T06:42:00.6606344Z * [new branch] gh/StrongerXi/138/head -> origin/gh/StrongerXi/138/head 2025-09-07T06:42:00.6606756Z * [new branch] gh/StrongerXi/138/orig -> origin/gh/StrongerXi/138/orig 2025-09-07T06:42:00.6607095Z * [new branch] gh/StrongerXi/139/base -> origin/gh/StrongerXi/139/base 2025-09-07T06:42:00.6607431Z * [new branch] gh/StrongerXi/139/head -> origin/gh/StrongerXi/139/head 2025-09-07T06:42:00.6610992Z * [new branch] gh/StrongerXi/139/orig -> origin/gh/StrongerXi/139/orig 2025-09-07T06:42:00.6611360Z * [new branch] gh/StrongerXi/140/base -> origin/gh/StrongerXi/140/base 2025-09-07T06:42:00.6611703Z * [new branch] gh/StrongerXi/140/head -> origin/gh/StrongerXi/140/head 2025-09-07T06:42:00.6612046Z * [new branch] gh/StrongerXi/140/orig -> origin/gh/StrongerXi/140/orig 2025-09-07T06:42:00.6612400Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-09-07T06:42:00.6612761Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-09-07T06:42:00.6613125Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-09-07T06:42:00.6613474Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-09-07T06:42:00.6615341Z * [new branch] gh/XilunWu/133/base -> origin/gh/XilunWu/133/base 2025-09-07T06:42:00.6615682Z * [new branch] gh/XilunWu/133/head -> origin/gh/XilunWu/133/head 2025-09-07T06:42:00.6616061Z * [new branch] gh/XilunWu/133/orig -> origin/gh/XilunWu/133/orig 2025-09-07T06:42:00.6616391Z * [new branch] gh/XilunWu/139/base -> origin/gh/XilunWu/139/base 2025-09-07T06:42:00.6616719Z * [new branch] gh/XilunWu/139/head -> origin/gh/XilunWu/139/head 2025-09-07T06:42:00.6617046Z * [new branch] gh/XilunWu/139/orig -> origin/gh/XilunWu/139/orig 2025-09-07T06:42:00.6617378Z * [new branch] gh/XilunWu/143/base -> origin/gh/XilunWu/143/base 2025-09-07T06:42:00.6620465Z * [new branch] gh/XilunWu/143/head -> origin/gh/XilunWu/143/head 2025-09-07T06:42:00.6620796Z * [new branch] gh/XilunWu/143/orig -> origin/gh/XilunWu/143/orig 2025-09-07T06:42:00.6621124Z * [new branch] gh/XilunWu/144/base -> origin/gh/XilunWu/144/base 2025-09-07T06:42:00.6621456Z * [new branch] gh/XilunWu/144/head -> origin/gh/XilunWu/144/head 2025-09-07T06:42:00.6621823Z * [new branch] gh/XilunWu/144/orig -> origin/gh/XilunWu/144/orig 2025-09-07T06:42:00.6622211Z * [new branch] gh/XilunWu/145/base -> origin/gh/XilunWu/145/base 2025-09-07T06:42:00.6623146Z * [new branch] gh/XilunWu/145/head -> origin/gh/XilunWu/145/head 2025-09-07T06:42:00.6623617Z * [new branch] gh/XilunWu/145/orig -> origin/gh/XilunWu/145/orig 2025-09-07T06:42:00.6624002Z * [new branch] gh/XilunWu/146/base -> origin/gh/XilunWu/146/base 2025-09-07T06:42:00.6629789Z * [new branch] gh/XilunWu/146/head -> origin/gh/XilunWu/146/head 2025-09-07T06:42:00.6635252Z * [new branch] gh/XilunWu/146/orig -> origin/gh/XilunWu/146/orig 2025-09-07T06:42:00.6641097Z * [new branch] gh/XilunWu/147/base -> origin/gh/XilunWu/147/base 2025-09-07T06:42:00.6641646Z * [new branch] gh/XilunWu/147/head -> origin/gh/XilunWu/147/head 2025-09-07T06:42:00.6642102Z * [new branch] gh/XilunWu/147/orig -> origin/gh/XilunWu/147/orig 2025-09-07T06:42:00.6642953Z * [new branch] gh/XilunWu/148/base -> origin/gh/XilunWu/148/base 2025-09-07T06:42:00.6643382Z * [new branch] gh/XilunWu/148/head -> origin/gh/XilunWu/148/head 2025-09-07T06:42:00.6643722Z * [new branch] gh/XilunWu/148/orig -> origin/gh/XilunWu/148/orig 2025-09-07T06:42:00.6644289Z * [new branch] gh/XilunWu/149/base -> origin/gh/XilunWu/149/base 2025-09-07T06:42:00.6644626Z * [new branch] gh/XilunWu/149/head -> origin/gh/XilunWu/149/head 2025-09-07T06:42:00.6644949Z * [new branch] gh/XilunWu/149/orig -> origin/gh/XilunWu/149/orig 2025-09-07T06:42:00.6645277Z * [new branch] gh/XilunWu/150/base -> origin/gh/XilunWu/150/base 2025-09-07T06:42:00.6645602Z * [new branch] gh/XilunWu/150/head -> origin/gh/XilunWu/150/head 2025-09-07T06:42:00.6645940Z * [new branch] gh/XilunWu/150/orig -> origin/gh/XilunWu/150/orig 2025-09-07T06:42:00.6646260Z * [new branch] gh/XilunWu/151/base -> origin/gh/XilunWu/151/base 2025-09-07T06:42:00.6646587Z * [new branch] gh/XilunWu/151/head -> origin/gh/XilunWu/151/head 2025-09-07T06:42:00.6646915Z * [new branch] gh/XilunWu/151/orig -> origin/gh/XilunWu/151/orig 2025-09-07T06:42:00.6647256Z * [new branch] gh/XilunWu/152/base -> origin/gh/XilunWu/152/base 2025-09-07T06:42:00.6647579Z * [new branch] gh/XilunWu/152/head -> origin/gh/XilunWu/152/head 2025-09-07T06:42:00.6647932Z * [new branch] gh/XilunWu/152/orig -> origin/gh/XilunWu/152/orig 2025-09-07T06:42:00.6648255Z * [new branch] gh/XilunWu/153/base -> origin/gh/XilunWu/153/base 2025-09-07T06:42:00.6648646Z * [new branch] gh/XilunWu/153/head -> origin/gh/XilunWu/153/head 2025-09-07T06:42:00.6648969Z * [new branch] gh/XilunWu/153/orig -> origin/gh/XilunWu/153/orig 2025-09-07T06:42:00.6649287Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-09-07T06:42:00.6649600Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-09-07T06:42:00.6649922Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-09-07T06:42:00.6650267Z * [new branch] gh/XilunWu/161/base -> origin/gh/XilunWu/161/base 2025-09-07T06:42:00.6650594Z * [new branch] gh/XilunWu/161/head -> origin/gh/XilunWu/161/head 2025-09-07T06:42:00.6650928Z * [new branch] gh/XilunWu/161/orig -> origin/gh/XilunWu/161/orig 2025-09-07T06:42:00.6651247Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-09-07T06:42:00.6651570Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-09-07T06:42:00.6651889Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-09-07T06:42:00.6652214Z * [new branch] gh/XilunWu/164/base -> origin/gh/XilunWu/164/base 2025-09-07T06:42:00.6652538Z * [new branch] gh/XilunWu/164/head -> origin/gh/XilunWu/164/head 2025-09-07T06:42:00.6652865Z * [new branch] gh/XilunWu/164/orig -> origin/gh/XilunWu/164/orig 2025-09-07T06:42:00.6653184Z * [new branch] gh/XilunWu/165/base -> origin/gh/XilunWu/165/base 2025-09-07T06:42:00.6653502Z * [new branch] gh/XilunWu/165/head -> origin/gh/XilunWu/165/head 2025-09-07T06:42:00.6653815Z * [new branch] gh/XilunWu/165/orig -> origin/gh/XilunWu/165/orig 2025-09-07T06:42:00.6654134Z * [new branch] gh/XilunWu/166/base -> origin/gh/XilunWu/166/base 2025-09-07T06:42:00.6654447Z * [new branch] gh/XilunWu/166/head -> origin/gh/XilunWu/166/head 2025-09-07T06:42:00.6654768Z * [new branch] gh/XilunWu/166/orig -> origin/gh/XilunWu/166/orig 2025-09-07T06:42:00.6655273Z * [new branch] gh/XilunWu/167/base -> origin/gh/XilunWu/167/base 2025-09-07T06:42:00.6655717Z * [new branch] gh/XilunWu/167/head -> origin/gh/XilunWu/167/head 2025-09-07T06:42:00.6656212Z * [new branch] gh/XilunWu/167/orig -> origin/gh/XilunWu/167/orig 2025-09-07T06:42:00.6657054Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-09-07T06:42:00.6657621Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-09-07T06:42:00.6658086Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-09-07T06:42:00.6658994Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-09-07T06:42:00.6661651Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-09-07T06:42:00.6662275Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-09-07T06:42:00.6662672Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-09-07T06:42:00.6663039Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-09-07T06:42:00.6663429Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-09-07T06:42:00.6663816Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-09-07T06:42:00.6664210Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-09-07T06:42:00.6664592Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-09-07T06:42:00.6664967Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-09-07T06:42:00.6665498Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-09-07T06:42:00.6666112Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-09-07T06:42:00.6666482Z * [new branch] gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-09-07T06:42:00.6666851Z * [new branch] gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-09-07T06:42:00.6667221Z * [new branch] gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig 2025-09-07T06:42:00.6667562Z * [new branch] gh/XuehaiPan/232/base -> origin/gh/XuehaiPan/232/base 2025-09-07T06:42:00.6668099Z * [new branch] gh/XuehaiPan/232/head -> origin/gh/XuehaiPan/232/head 2025-09-07T06:42:00.6668604Z * [new branch] gh/XuehaiPan/232/orig -> origin/gh/XuehaiPan/232/orig 2025-09-07T06:42:00.6670021Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-09-07T06:42:00.6670445Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-09-07T06:42:00.6670839Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-09-07T06:42:00.6672199Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-09-07T06:42:00.6672602Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-09-07T06:42:00.6673225Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-09-07T06:42:00.6674099Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-09-07T06:42:00.6674726Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-09-07T06:42:00.6680638Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-09-07T06:42:00.6686269Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-09-07T06:42:00.6686717Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-09-07T06:42:00.6687094Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-09-07T06:42:00.6687463Z * [new branch] gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-09-07T06:42:00.6687991Z * [new branch] gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-09-07T06:42:00.6688367Z * [new branch] gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig 2025-09-07T06:42:00.6688738Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-09-07T06:42:00.6689106Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-09-07T06:42:00.6689482Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-09-07T06:42:00.6689909Z * [new branch] gh/XuehaiPan/290/base -> origin/gh/XuehaiPan/290/base 2025-09-07T06:42:00.6690287Z * [new branch] gh/XuehaiPan/290/head -> origin/gh/XuehaiPan/290/head 2025-09-07T06:42:00.6690655Z * [new branch] gh/XuehaiPan/290/orig -> origin/gh/XuehaiPan/290/orig 2025-09-07T06:42:00.6691030Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-09-07T06:42:00.6691401Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-09-07T06:42:00.6691765Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-09-07T06:42:00.6692139Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-09-07T06:42:00.6692503Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-09-07T06:42:00.6692904Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-09-07T06:42:00.6693348Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-09-07T06:42:00.6693973Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-09-07T06:42:00.6694841Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-09-07T06:42:00.6695311Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-09-07T06:42:00.6695696Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-09-07T06:42:00.6696105Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-09-07T06:42:00.6696481Z * [new branch] gh/XuehaiPan/356/base -> origin/gh/XuehaiPan/356/base 2025-09-07T06:42:00.6696849Z * [new branch] gh/XuehaiPan/356/head -> origin/gh/XuehaiPan/356/head 2025-09-07T06:42:00.6697250Z * [new branch] gh/XuehaiPan/356/orig -> origin/gh/XuehaiPan/356/orig 2025-09-07T06:42:00.6697686Z * [new branch] gh/XuehaiPan/357/base -> origin/gh/XuehaiPan/357/base 2025-09-07T06:42:00.6698069Z * [new branch] gh/XuehaiPan/357/head -> origin/gh/XuehaiPan/357/head 2025-09-07T06:42:00.6698457Z * [new branch] gh/XuehaiPan/357/orig -> origin/gh/XuehaiPan/357/orig 2025-09-07T06:42:00.6698834Z * [new branch] gh/XuehaiPan/358/base -> origin/gh/XuehaiPan/358/base 2025-09-07T06:42:00.6699214Z * [new branch] gh/XuehaiPan/358/head -> origin/gh/XuehaiPan/358/head 2025-09-07T06:42:00.6699590Z * [new branch] gh/XuehaiPan/358/orig -> origin/gh/XuehaiPan/358/orig 2025-09-07T06:42:00.6699969Z * [new branch] gh/XuehaiPan/359/base -> origin/gh/XuehaiPan/359/base 2025-09-07T06:42:00.6700337Z * [new branch] gh/XuehaiPan/359/head -> origin/gh/XuehaiPan/359/head 2025-09-07T06:42:00.6701021Z * [new branch] gh/XuehaiPan/359/orig -> origin/gh/XuehaiPan/359/orig 2025-09-07T06:42:00.6701639Z * [new branch] gh/XuehaiPan/360/base -> origin/gh/XuehaiPan/360/base 2025-09-07T06:42:00.6702304Z * [new branch] gh/XuehaiPan/360/head -> origin/gh/XuehaiPan/360/head 2025-09-07T06:42:00.6702976Z * [new branch] gh/XuehaiPan/360/orig -> origin/gh/XuehaiPan/360/orig 2025-09-07T06:42:00.6704175Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-09-07T06:42:00.6704564Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-09-07T06:42:00.6705312Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-09-07T06:42:00.6706679Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-09-07T06:42:00.6707174Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-09-07T06:42:00.6708421Z * [new branch] gh/XuehaiPan/369/base -> origin/gh/XuehaiPan/369/base 2025-09-07T06:42:00.6708816Z * [new branch] gh/XuehaiPan/369/head -> origin/gh/XuehaiPan/369/head 2025-09-07T06:42:00.6709577Z * [new branch] gh/XuehaiPan/369/orig -> origin/gh/XuehaiPan/369/orig 2025-09-07T06:42:00.6710779Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-09-07T06:42:00.6711176Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-09-07T06:42:00.6711979Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-09-07T06:42:00.6713155Z * [new branch] gh/XuehaiPan/380/base -> origin/gh/XuehaiPan/380/base 2025-09-07T06:42:00.6713554Z * [new branch] gh/XuehaiPan/380/head -> origin/gh/XuehaiPan/380/head 2025-09-07T06:42:00.6714472Z * [new branch] gh/XuehaiPan/380/orig -> origin/gh/XuehaiPan/380/orig 2025-09-07T06:42:00.6715284Z * [new branch] gh/XuehaiPan/381/base -> origin/gh/XuehaiPan/381/base 2025-09-07T06:42:00.6715912Z * [new branch] gh/XuehaiPan/381/head -> origin/gh/XuehaiPan/381/head 2025-09-07T06:42:00.6717127Z * [new branch] gh/XuehaiPan/382/base -> origin/gh/XuehaiPan/382/base 2025-09-07T06:42:00.6717711Z * [new branch] gh/XuehaiPan/382/head -> origin/gh/XuehaiPan/382/head 2025-09-07T06:42:00.6718139Z * [new branch] gh/XuehaiPan/382/orig -> origin/gh/XuehaiPan/382/orig 2025-09-07T06:42:00.6720415Z * [new branch] gh/XuehaiPan/383/base -> origin/gh/XuehaiPan/383/base 2025-09-07T06:42:00.6720909Z * [new branch] gh/XuehaiPan/383/head -> origin/gh/XuehaiPan/383/head 2025-09-07T06:42:00.6721279Z * [new branch] gh/XuehaiPan/383/orig -> origin/gh/XuehaiPan/383/orig 2025-09-07T06:42:00.6721746Z * [new branch] gh/XuehaiPan/384/base -> origin/gh/XuehaiPan/384/base 2025-09-07T06:42:00.6722402Z * [new branch] gh/XuehaiPan/384/head -> origin/gh/XuehaiPan/384/head 2025-09-07T06:42:00.6723016Z * [new branch] gh/XuehaiPan/384/orig -> origin/gh/XuehaiPan/384/orig 2025-09-07T06:42:00.6724352Z * [new branch] gh/XuehaiPan/385/base -> origin/gh/XuehaiPan/385/base 2025-09-07T06:42:00.6725266Z * [new branch] gh/XuehaiPan/385/head -> origin/gh/XuehaiPan/385/head 2025-09-07T06:42:00.6725863Z * [new branch] gh/XuehaiPan/385/orig -> origin/gh/XuehaiPan/385/orig 2025-09-07T06:42:00.6726277Z * [new branch] gh/XuehaiPan/386/base -> origin/gh/XuehaiPan/386/base 2025-09-07T06:42:00.6726891Z * [new branch] gh/XuehaiPan/386/head -> origin/gh/XuehaiPan/386/head 2025-09-07T06:42:00.6727523Z * [new branch] gh/XuehaiPan/386/orig -> origin/gh/XuehaiPan/386/orig 2025-09-07T06:42:00.6729226Z * [new branch] gh/XuehaiPan/387/base -> origin/gh/XuehaiPan/387/base 2025-09-07T06:42:00.6729668Z * [new branch] gh/XuehaiPan/387/head -> origin/gh/XuehaiPan/387/head 2025-09-07T06:42:00.6730035Z * [new branch] gh/XuehaiPan/387/orig -> origin/gh/XuehaiPan/387/orig 2025-09-07T06:42:00.6735412Z * [new branch] gh/ZainRizvi/1/base -> origin/gh/ZainRizvi/1/base 2025-09-07T06:42:00.6737627Z * [new branch] gh/ZainRizvi/1/head -> origin/gh/ZainRizvi/1/head 2025-09-07T06:42:00.6741470Z * [new branch] gh/ZainRizvi/2/base -> origin/gh/ZainRizvi/2/base 2025-09-07T06:42:00.6741913Z * [new branch] gh/ZainRizvi/2/head -> origin/gh/ZainRizvi/2/head 2025-09-07T06:42:00.6742289Z * [new branch] gh/ZainRizvi/3/base -> origin/gh/ZainRizvi/3/base 2025-09-07T06:42:00.6742673Z * [new branch] gh/ZainRizvi/3/head -> origin/gh/ZainRizvi/3/head 2025-09-07T06:42:00.6743052Z * [new branch] gh/ZainRizvi/4/base -> origin/gh/ZainRizvi/4/base 2025-09-07T06:42:00.6743465Z * [new branch] gh/ZainRizvi/4/head -> origin/gh/ZainRizvi/4/head 2025-09-07T06:42:00.6743831Z * [new branch] gh/ZainRizvi/5/base -> origin/gh/ZainRizvi/5/base 2025-09-07T06:42:00.6744199Z * [new branch] gh/ZainRizvi/5/head -> origin/gh/ZainRizvi/5/head 2025-09-07T06:42:00.6744559Z * [new branch] gh/ZainRizvi/6/base -> origin/gh/ZainRizvi/6/base 2025-09-07T06:42:00.6744925Z * [new branch] gh/ZainRizvi/6/head -> origin/gh/ZainRizvi/6/head 2025-09-07T06:42:00.6745285Z * [new branch] gh/ZainRizvi/6/orig -> origin/gh/ZainRizvi/6/orig 2025-09-07T06:42:00.6745837Z * [new branch] gh/ZainRizvi/7/base -> origin/gh/ZainRizvi/7/base 2025-09-07T06:42:00.6746470Z * [new branch] gh/ZainRizvi/7/head -> origin/gh/ZainRizvi/7/head 2025-09-07T06:42:00.6746856Z * [new branch] gh/ZainRizvi/7/orig -> origin/gh/ZainRizvi/7/orig 2025-09-07T06:42:00.6747224Z * [new branch] gh/ZainRizvi/8/base -> origin/gh/ZainRizvi/8/base 2025-09-07T06:42:00.6747548Z * [new branch] gh/ZainRizvi/8/head -> origin/gh/ZainRizvi/8/head 2025-09-07T06:42:00.6747879Z * [new branch] gh/ZainRizvi/9/base -> origin/gh/ZainRizvi/9/base 2025-09-07T06:42:00.6748211Z * [new branch] gh/ZainRizvi/9/head -> origin/gh/ZainRizvi/9/head 2025-09-07T06:42:00.6748545Z * [new branch] gh/ZainRizvi/9/orig -> origin/gh/ZainRizvi/9/orig 2025-09-07T06:42:00.6749242Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-09-07T06:42:00.6749602Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-09-07T06:42:00.6749965Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-09-07T06:42:00.6750318Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-09-07T06:42:00.6755862Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-09-07T06:42:00.6756459Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-09-07T06:42:00.6756973Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-09-07T06:42:00.6757328Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-09-07T06:42:00.6757696Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-09-07T06:42:00.6758044Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-09-07T06:42:00.6758417Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-09-07T06:42:00.6758785Z * [new branch] gh/ZhiweiYan-96/64/base -> origin/gh/ZhiweiYan-96/64/base 2025-09-07T06:42:00.6759144Z * [new branch] gh/ZhiweiYan-96/64/head -> origin/gh/ZhiweiYan-96/64/head 2025-09-07T06:42:00.6760335Z * [new branch] gh/ZhiweiYan-96/64/orig -> origin/gh/ZhiweiYan-96/64/orig 2025-09-07T06:42:00.6761023Z * [new branch] gh/ZhiweiYan-96/65/base -> origin/gh/ZhiweiYan-96/65/base 2025-09-07T06:42:00.6761539Z * [new branch] gh/ZhiweiYan-96/65/head -> origin/gh/ZhiweiYan-96/65/head 2025-09-07T06:42:00.6762065Z * [new branch] gh/ZhiweiYan-96/65/orig -> origin/gh/ZhiweiYan-96/65/orig 2025-09-07T06:42:00.6762912Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-09-07T06:42:00.6763364Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-09-07T06:42:00.6767671Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-09-07T06:42:00.6768274Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-09-07T06:42:00.6768798Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-09-07T06:42:00.6769680Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-09-07T06:42:00.6770141Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-09-07T06:42:00.6770544Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-09-07T06:42:00.6770910Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-09-07T06:42:00.6771263Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-09-07T06:42:00.6771611Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-09-07T06:42:00.6772111Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-09-07T06:42:00.6772491Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-09-07T06:42:00.6772848Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-09-07T06:42:00.6773550Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-09-07T06:42:00.6774084Z * [new branch] gh/alexsamardzic/9/base -> origin/gh/alexsamardzic/9/base 2025-09-07T06:42:00.6774481Z * [new branch] gh/alexsamardzic/9/head -> origin/gh/alexsamardzic/9/head 2025-09-07T06:42:00.6774866Z * [new branch] gh/alexsamardzic/9/orig -> origin/gh/alexsamardzic/9/orig 2025-09-07T06:42:00.6775238Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-09-07T06:42:00.6775620Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-09-07T06:42:00.6776268Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-09-07T06:42:00.6777558Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-09-07T06:42:00.6777998Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-09-07T06:42:00.6778577Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-09-07T06:42:00.6780729Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-09-07T06:42:00.6781139Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-09-07T06:42:00.6781483Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-09-07T06:42:00.6782183Z * [new branch] gh/andrewor14/51/base -> origin/gh/andrewor14/51/base 2025-09-07T06:42:00.6782903Z * [new branch] gh/andrewor14/51/orig -> origin/gh/andrewor14/51/orig 2025-09-07T06:42:00.6784546Z * [new branch] gh/andyanwang/1/base -> origin/gh/andyanwang/1/base 2025-09-07T06:42:00.6784942Z * [new branch] gh/andyanwang/1/head -> origin/gh/andyanwang/1/head 2025-09-07T06:42:00.6785631Z * [new branch] gh/andyanwang/1/orig -> origin/gh/andyanwang/1/orig 2025-09-07T06:42:00.6787206Z * [new branch] gh/andyanwang/13/base -> origin/gh/andyanwang/13/base 2025-09-07T06:42:00.6787599Z * [new branch] gh/andyanwang/13/head -> origin/gh/andyanwang/13/head 2025-09-07T06:42:00.6791950Z * [new branch] gh/andyanwang/13/orig -> origin/gh/andyanwang/13/orig 2025-09-07T06:42:00.6792303Z * [new branch] gh/andyanwang/2/base -> origin/gh/andyanwang/2/base 2025-09-07T06:42:00.6792647Z * [new branch] gh/andyanwang/2/head -> origin/gh/andyanwang/2/head 2025-09-07T06:42:00.6792986Z * [new branch] gh/andyanwang/2/orig -> origin/gh/andyanwang/2/orig 2025-09-07T06:42:00.6793319Z * [new branch] gh/andyanwang/28/base -> origin/gh/andyanwang/28/base 2025-09-07T06:42:00.6793673Z * [new branch] gh/andyanwang/28/head -> origin/gh/andyanwang/28/head 2025-09-07T06:42:00.6794024Z * [new branch] gh/andyanwang/28/orig -> origin/gh/andyanwang/28/orig 2025-09-07T06:42:00.6799100Z * [new branch] gh/andyanwang/3/base -> origin/gh/andyanwang/3/base 2025-09-07T06:42:00.6799514Z * [new branch] gh/andyanwang/3/head -> origin/gh/andyanwang/3/head 2025-09-07T06:42:00.6799857Z * [new branch] gh/andyanwang/3/orig -> origin/gh/andyanwang/3/orig 2025-09-07T06:42:00.6800217Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-09-07T06:42:00.6800929Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-09-07T06:42:00.6812631Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-09-07T06:42:00.6813066Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-09-07T06:42:00.6813443Z * [new branch] gh/andyanwang/32/base -> origin/gh/andyanwang/32/base 2025-09-07T06:42:00.6813798Z * [new branch] gh/andyanwang/32/head -> origin/gh/andyanwang/32/head 2025-09-07T06:42:00.6814139Z * [new branch] gh/andyanwang/32/orig -> origin/gh/andyanwang/32/orig 2025-09-07T06:42:00.6814480Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-09-07T06:42:00.6814823Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-09-07T06:42:00.6815165Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-09-07T06:42:00.6815523Z * [new branch] gh/andyanwang/4/base -> origin/gh/andyanwang/4/base 2025-09-07T06:42:00.6815877Z * [new branch] gh/andyanwang/4/head -> origin/gh/andyanwang/4/head 2025-09-07T06:42:00.6816222Z * [new branch] gh/andyanwang/4/orig -> origin/gh/andyanwang/4/orig 2025-09-07T06:42:00.6816558Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-09-07T06:42:00.6816891Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-09-07T06:42:00.6817220Z * [new branch] gh/angelayi/111/base -> origin/gh/angelayi/111/base 2025-09-07T06:42:00.6817551Z * [new branch] gh/angelayi/111/head -> origin/gh/angelayi/111/head 2025-09-07T06:42:00.6817881Z * [new branch] gh/angelayi/111/orig -> origin/gh/angelayi/111/orig 2025-09-07T06:42:00.6818218Z * [new branch] gh/angelayi/112/base -> origin/gh/angelayi/112/base 2025-09-07T06:42:00.6818554Z * [new branch] gh/angelayi/112/head -> origin/gh/angelayi/112/head 2025-09-07T06:42:00.6818913Z * [new branch] gh/angelayi/112/orig -> origin/gh/angelayi/112/orig 2025-09-07T06:42:00.6819263Z * [new branch] gh/angelayi/113/base -> origin/gh/angelayi/113/base 2025-09-07T06:42:00.6819968Z * [new branch] gh/angelayi/113/head -> origin/gh/angelayi/113/head 2025-09-07T06:42:00.6820334Z * [new branch] gh/angelayi/113/orig -> origin/gh/angelayi/113/orig 2025-09-07T06:42:00.6820696Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-09-07T06:42:00.6821054Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-09-07T06:42:00.6821410Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-09-07T06:42:00.6821772Z * [new branch] gh/angelayi/115/base -> origin/gh/angelayi/115/base 2025-09-07T06:42:00.6822122Z * [new branch] gh/angelayi/115/head -> origin/gh/angelayi/115/head 2025-09-07T06:42:00.6822483Z * [new branch] gh/angelayi/115/orig -> origin/gh/angelayi/115/orig 2025-09-07T06:42:00.6822867Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-09-07T06:42:00.6823253Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-09-07T06:42:00.6823636Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-09-07T06:42:00.6824023Z * [new branch] gh/anijain2305/766/base -> origin/gh/anijain2305/766/base 2025-09-07T06:42:00.6824394Z * [new branch] gh/anijain2305/766/head -> origin/gh/anijain2305/766/head 2025-09-07T06:42:00.6824821Z * [new branch] gh/anijain2305/766/orig -> origin/gh/anijain2305/766/orig 2025-09-07T06:42:00.6825303Z * [new branch] gh/anijain2305/790/base -> origin/gh/anijain2305/790/base 2025-09-07T06:42:00.6825860Z * [new branch] gh/anijain2305/790/head -> origin/gh/anijain2305/790/head 2025-09-07T06:42:00.6826255Z * [new branch] gh/anijain2305/790/orig -> origin/gh/anijain2305/790/orig 2025-09-07T06:42:00.6826790Z * [new branch] gh/anijain2305/792/base -> origin/gh/anijain2305/792/base 2025-09-07T06:42:00.6827184Z * [new branch] gh/anijain2305/792/head -> origin/gh/anijain2305/792/head 2025-09-07T06:42:00.6827836Z * [new branch] gh/anijain2305/792/orig -> origin/gh/anijain2305/792/orig 2025-09-07T06:42:00.6829971Z * [new branch] gh/anijain2305/803/base -> origin/gh/anijain2305/803/base 2025-09-07T06:42:00.6830472Z * [new branch] gh/anijain2305/803/head -> origin/gh/anijain2305/803/head 2025-09-07T06:42:00.6830899Z * [new branch] gh/anijain2305/803/orig -> origin/gh/anijain2305/803/orig 2025-09-07T06:42:00.6831633Z * [new branch] gh/anijain2305/804/base -> origin/gh/anijain2305/804/base 2025-09-07T06:42:00.6832221Z * [new branch] gh/anijain2305/804/head -> origin/gh/anijain2305/804/head 2025-09-07T06:42:00.6832892Z * [new branch] gh/anijain2305/804/orig -> origin/gh/anijain2305/804/orig 2025-09-07T06:42:00.6833880Z * [new branch] gh/anijain2305/805/base -> origin/gh/anijain2305/805/base 2025-09-07T06:42:00.6834469Z * [new branch] gh/anijain2305/805/head -> origin/gh/anijain2305/805/head 2025-09-07T06:42:00.6835230Z * [new branch] gh/anijain2305/805/orig -> origin/gh/anijain2305/805/orig 2025-09-07T06:42:00.6836440Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-09-07T06:42:00.6836847Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-09-07T06:42:00.6837469Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-09-07T06:42:00.6838723Z * [new branch] gh/anijain2305/812/base -> origin/gh/anijain2305/812/base 2025-09-07T06:42:00.6839129Z * [new branch] gh/anijain2305/812/head -> origin/gh/anijain2305/812/head 2025-09-07T06:42:00.6839814Z * [new branch] gh/anijain2305/812/orig -> origin/gh/anijain2305/812/orig 2025-09-07T06:42:00.6840686Z * [new branch] gh/anijain2305/838/base -> origin/gh/anijain2305/838/base 2025-09-07T06:42:00.6841369Z * [new branch] gh/anijain2305/838/head -> origin/gh/anijain2305/838/head 2025-09-07T06:42:00.6842015Z * [new branch] gh/anijain2305/838/orig -> origin/gh/anijain2305/838/orig 2025-09-07T06:42:00.6843087Z * [new branch] gh/anijain2305/839/base -> origin/gh/anijain2305/839/base 2025-09-07T06:42:00.6843506Z * [new branch] gh/anijain2305/839/head -> origin/gh/anijain2305/839/head 2025-09-07T06:42:00.6844200Z * [new branch] gh/anijain2305/839/orig -> origin/gh/anijain2305/839/orig 2025-09-07T06:42:00.6845290Z * [new branch] gh/anijain2305/843/base -> origin/gh/anijain2305/843/base 2025-09-07T06:42:00.6845778Z * [new branch] gh/anijain2305/843/head -> origin/gh/anijain2305/843/head 2025-09-07T06:42:00.6846432Z * [new branch] gh/anijain2305/843/orig -> origin/gh/anijain2305/843/orig 2025-09-07T06:42:00.6847624Z * [new branch] gh/anijain2305/844/base -> origin/gh/anijain2305/844/base 2025-09-07T06:42:00.6848107Z * [new branch] gh/anijain2305/844/head -> origin/gh/anijain2305/844/head 2025-09-07T06:42:00.6848732Z * [new branch] gh/anijain2305/844/orig -> origin/gh/anijain2305/844/orig 2025-09-07T06:42:00.6849883Z * [new branch] gh/anijain2305/846/base -> origin/gh/anijain2305/846/base 2025-09-07T06:42:00.6851019Z * [new branch] gh/anijain2305/846/head -> origin/gh/anijain2305/846/head 2025-09-07T06:42:00.6851503Z * [new branch] gh/anijain2305/846/orig -> origin/gh/anijain2305/846/orig 2025-09-07T06:42:00.6851969Z * [new branch] gh/anijain2305/848/base -> origin/gh/anijain2305/848/base 2025-09-07T06:42:00.6856499Z * [new branch] gh/anijain2305/848/head -> origin/gh/anijain2305/848/head 2025-09-07T06:42:00.6856974Z * [new branch] gh/anijain2305/848/orig -> origin/gh/anijain2305/848/orig 2025-09-07T06:42:00.6857361Z * [new branch] gh/anijain2305/849/base -> origin/gh/anijain2305/849/base 2025-09-07T06:42:00.6857731Z * [new branch] gh/anijain2305/849/head -> origin/gh/anijain2305/849/head 2025-09-07T06:42:00.6858105Z * [new branch] gh/anijain2305/849/orig -> origin/gh/anijain2305/849/orig 2025-09-07T06:42:00.6858492Z * [new branch] gh/anijain2305/850/base -> origin/gh/anijain2305/850/base 2025-09-07T06:42:00.6858870Z * [new branch] gh/anijain2305/850/head -> origin/gh/anijain2305/850/head 2025-09-07T06:42:00.6859278Z * [new branch] gh/anijain2305/850/orig -> origin/gh/anijain2305/850/orig 2025-09-07T06:42:00.6859683Z * [new branch] gh/anijain2305/851/base -> origin/gh/anijain2305/851/base 2025-09-07T06:42:00.6860074Z * [new branch] gh/anijain2305/851/head -> origin/gh/anijain2305/851/head 2025-09-07T06:42:00.6860931Z * [new branch] gh/anijain2305/851/orig -> origin/gh/anijain2305/851/orig 2025-09-07T06:42:00.6862179Z * [new branch] gh/anijain2305/852/base -> origin/gh/anijain2305/852/base 2025-09-07T06:42:00.6862590Z * [new branch] gh/anijain2305/852/head -> origin/gh/anijain2305/852/head 2025-09-07T06:42:00.6863296Z * [new branch] gh/anijain2305/852/orig -> origin/gh/anijain2305/852/orig 2025-09-07T06:42:00.6864376Z * [new branch] gh/anijain2305/853/base -> origin/gh/anijain2305/853/base 2025-09-07T06:42:00.6864752Z * [new branch] gh/anijain2305/853/head -> origin/gh/anijain2305/853/head 2025-09-07T06:42:00.6865394Z * [new branch] gh/anijain2305/853/orig -> origin/gh/anijain2305/853/orig 2025-09-07T06:42:00.6866933Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-09-07T06:42:00.6867705Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-09-07T06:42:00.6868084Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-09-07T06:42:00.6870329Z * [new branch] gh/anijain2305/855/base -> origin/gh/anijain2305/855/base 2025-09-07T06:42:00.6870784Z * [new branch] gh/anijain2305/855/head -> origin/gh/anijain2305/855/head 2025-09-07T06:42:00.6871185Z * [new branch] gh/anijain2305/855/orig -> origin/gh/anijain2305/855/orig 2025-09-07T06:42:00.6871775Z * [new branch] gh/anijain2305/856/base -> origin/gh/anijain2305/856/base 2025-09-07T06:42:00.6872179Z * [new branch] gh/anijain2305/856/head -> origin/gh/anijain2305/856/head 2025-09-07T06:42:00.6872867Z * [new branch] gh/anijain2305/856/orig -> origin/gh/anijain2305/856/orig 2025-09-07T06:42:00.6876267Z * [new branch] gh/anijain2305/857/base -> origin/gh/anijain2305/857/base 2025-09-07T06:42:00.6876684Z * [new branch] gh/anijain2305/857/head -> origin/gh/anijain2305/857/head 2025-09-07T06:42:00.6877046Z * [new branch] gh/anijain2305/857/orig -> origin/gh/anijain2305/857/orig 2025-09-07T06:42:00.6877398Z * [new branch] gh/anijain2305/858/base -> origin/gh/anijain2305/858/base 2025-09-07T06:42:00.6877755Z * [new branch] gh/anijain2305/858/head -> origin/gh/anijain2305/858/head 2025-09-07T06:42:00.6878272Z * [new branch] gh/anijain2305/858/orig -> origin/gh/anijain2305/858/orig 2025-09-07T06:42:00.6878680Z * [new branch] gh/anijain2305/859/base -> origin/gh/anijain2305/859/base 2025-09-07T06:42:00.6879430Z * [new branch] gh/anijain2305/859/head -> origin/gh/anijain2305/859/head 2025-09-07T06:42:00.6879998Z * [new branch] gh/anijain2305/859/orig -> origin/gh/anijain2305/859/orig 2025-09-07T06:42:00.6885340Z * [new branch] gh/anijain2305/860/base -> origin/gh/anijain2305/860/base 2025-09-07T06:42:00.6885746Z * [new branch] gh/anijain2305/860/head -> origin/gh/anijain2305/860/head 2025-09-07T06:42:00.6886116Z * [new branch] gh/anijain2305/860/orig -> origin/gh/anijain2305/860/orig 2025-09-07T06:42:00.6886524Z * [new branch] gh/anijain2305/861/base -> origin/gh/anijain2305/861/base 2025-09-07T06:42:00.6886913Z * [new branch] gh/anijain2305/861/head -> origin/gh/anijain2305/861/head 2025-09-07T06:42:00.6887311Z * [new branch] gh/anijain2305/861/orig -> origin/gh/anijain2305/861/orig 2025-09-07T06:42:00.6887690Z * [new branch] gh/anijain2305/862/base -> origin/gh/anijain2305/862/base 2025-09-07T06:42:00.6888051Z * [new branch] gh/anijain2305/862/head -> origin/gh/anijain2305/862/head 2025-09-07T06:42:00.6888407Z * [new branch] gh/anijain2305/862/orig -> origin/gh/anijain2305/862/orig 2025-09-07T06:42:00.6888750Z * [new branch] gh/anijain2305/863/base -> origin/gh/anijain2305/863/base 2025-09-07T06:42:00.6889103Z * [new branch] gh/anijain2305/863/head -> origin/gh/anijain2305/863/head 2025-09-07T06:42:00.6889659Z * [new branch] gh/anijain2305/863/orig -> origin/gh/anijain2305/863/orig 2025-09-07T06:42:00.6890069Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-09-07T06:42:00.6890694Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-09-07T06:42:00.6891320Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-09-07T06:42:00.6893144Z * [new branch] gh/anijain2305/865/base -> origin/gh/anijain2305/865/base 2025-09-07T06:42:00.6893523Z * [new branch] gh/anijain2305/865/head -> origin/gh/anijain2305/865/head 2025-09-07T06:42:00.6894059Z * [new branch] gh/anijain2305/865/orig -> origin/gh/anijain2305/865/orig 2025-09-07T06:42:00.6894714Z * [new branch] gh/anijain2305/866/base -> origin/gh/anijain2305/866/base 2025-09-07T06:42:00.6895306Z * [new branch] gh/anijain2305/866/head -> origin/gh/anijain2305/866/head 2025-09-07T06:42:00.6896019Z * [new branch] gh/anijain2305/866/orig -> origin/gh/anijain2305/866/orig 2025-09-07T06:42:00.6897567Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-09-07T06:42:00.6898096Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-09-07T06:42:00.6898626Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-09-07T06:42:00.6900058Z * [new branch] gh/ankitageorge/13/base -> origin/gh/ankitageorge/13/base 2025-09-07T06:42:00.6900459Z * [new branch] gh/ankitageorge/13/head -> origin/gh/ankitageorge/13/head 2025-09-07T06:42:00.6901255Z * [new branch] gh/ankitageorge/13/orig -> origin/gh/ankitageorge/13/orig 2025-09-07T06:42:00.6902595Z * [new branch] gh/ankitageorge/14/base -> origin/gh/ankitageorge/14/base 2025-09-07T06:42:00.6902996Z * [new branch] gh/ankitageorge/14/head -> origin/gh/ankitageorge/14/head 2025-09-07T06:42:00.6903822Z * [new branch] gh/ankitageorge/14/orig -> origin/gh/ankitageorge/14/orig 2025-09-07T06:42:00.6905186Z * [new branch] gh/ankitageorge/15/base -> origin/gh/ankitageorge/15/base 2025-09-07T06:42:00.6905596Z * [new branch] gh/ankitageorge/15/head -> origin/gh/ankitageorge/15/head 2025-09-07T06:42:00.6906195Z * [new branch] gh/ankitageorge/15/orig -> origin/gh/ankitageorge/15/orig 2025-09-07T06:42:00.6907846Z * [new branch] gh/ankitageorge/16/base -> origin/gh/ankitageorge/16/base 2025-09-07T06:42:00.6908225Z * [new branch] gh/ankitageorge/16/head -> origin/gh/ankitageorge/16/head 2025-09-07T06:42:00.6908809Z * [new branch] gh/ankitageorge/16/orig -> origin/gh/ankitageorge/16/orig 2025-09-07T06:42:00.6910848Z * [new branch] gh/ankitageorge/17/base -> origin/gh/ankitageorge/17/base 2025-09-07T06:42:00.6911288Z * [new branch] gh/ankitageorge/17/head -> origin/gh/ankitageorge/17/head 2025-09-07T06:42:00.6911664Z * [new branch] gh/ankitageorge/17/orig -> origin/gh/ankitageorge/17/orig 2025-09-07T06:42:00.6912407Z * [new branch] gh/ankitageorge/21/base -> origin/gh/ankitageorge/21/base 2025-09-07T06:42:00.6913003Z * [new branch] gh/ankitageorge/21/head -> origin/gh/ankitageorge/21/head 2025-09-07T06:42:00.6913688Z * [new branch] gh/ankitageorge/21/orig -> origin/gh/ankitageorge/21/orig 2025-09-07T06:42:00.6915295Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-09-07T06:42:00.6917835Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-09-07T06:42:00.6918230Z * [new branch] gh/anshul-si/15/base -> origin/gh/anshul-si/15/base 2025-09-07T06:42:00.6918572Z * [new branch] gh/anshul-si/15/head -> origin/gh/anshul-si/15/head 2025-09-07T06:42:00.6918912Z * [new branch] gh/anshul-si/15/orig -> origin/gh/anshul-si/15/orig 2025-09-07T06:42:00.6919238Z * [new branch] gh/anshul-si/16/base -> origin/gh/anshul-si/16/base 2025-09-07T06:42:00.6919813Z * [new branch] gh/anshul-si/16/head -> origin/gh/anshul-si/16/head 2025-09-07T06:42:00.6925893Z * [new branch] gh/anshul-si/16/orig -> origin/gh/anshul-si/16/orig 2025-09-07T06:42:00.6926312Z * [new branch] gh/anshul-si/17/base -> origin/gh/anshul-si/17/base 2025-09-07T06:42:00.6926877Z * [new branch] gh/anshul-si/17/head -> origin/gh/anshul-si/17/head 2025-09-07T06:42:00.6927208Z * [new branch] gh/anshul-si/17/orig -> origin/gh/anshul-si/17/orig 2025-09-07T06:42:00.6927544Z * [new branch] gh/anshul-si/18/base -> origin/gh/anshul-si/18/base 2025-09-07T06:42:00.6927874Z * [new branch] gh/anshul-si/18/head -> origin/gh/anshul-si/18/head 2025-09-07T06:42:00.6928204Z * [new branch] gh/anshul-si/18/orig -> origin/gh/anshul-si/18/orig 2025-09-07T06:42:00.6932032Z * [new branch] gh/anshul-si/19/base -> origin/gh/anshul-si/19/base 2025-09-07T06:42:00.6932615Z * [new branch] gh/anshul-si/19/head -> origin/gh/anshul-si/19/head 2025-09-07T06:42:00.6933125Z * [new branch] gh/anshul-si/19/orig -> origin/gh/anshul-si/19/orig 2025-09-07T06:42:00.6934004Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-09-07T06:42:00.6934507Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-09-07T06:42:00.6934972Z * [new branch] gh/anshul-si/20/base -> origin/gh/anshul-si/20/base 2025-09-07T06:42:00.6940783Z * [new branch] gh/anshul-si/20/head -> origin/gh/anshul-si/20/head 2025-09-07T06:42:00.6941357Z * [new branch] gh/anshul-si/20/orig -> origin/gh/anshul-si/20/orig 2025-09-07T06:42:00.6941849Z * [new branch] gh/anshul-si/21/base -> origin/gh/anshul-si/21/base 2025-09-07T06:42:00.6942548Z * [new branch] gh/anshul-si/21/head -> origin/gh/anshul-si/21/head 2025-09-07T06:42:00.6942926Z * [new branch] gh/anshul-si/21/orig -> origin/gh/anshul-si/21/orig 2025-09-07T06:42:00.6943290Z * [new branch] gh/anshul-si/22/base -> origin/gh/anshul-si/22/base 2025-09-07T06:42:00.6943638Z * [new branch] gh/anshul-si/22/head -> origin/gh/anshul-si/22/head 2025-09-07T06:42:00.6943998Z * [new branch] gh/anshul-si/22/orig -> origin/gh/anshul-si/22/orig 2025-09-07T06:42:00.6944346Z * [new branch] gh/anshul-si/23/base -> origin/gh/anshul-si/23/base 2025-09-07T06:42:00.6944691Z * [new branch] gh/anshul-si/23/head -> origin/gh/anshul-si/23/head 2025-09-07T06:42:00.6945041Z * [new branch] gh/anshul-si/23/orig -> origin/gh/anshul-si/23/orig 2025-09-07T06:42:00.6945393Z * [new branch] gh/anshul-si/24/base -> origin/gh/anshul-si/24/base 2025-09-07T06:42:00.6946048Z * [new branch] gh/anshul-si/24/head -> origin/gh/anshul-si/24/head 2025-09-07T06:42:00.6946424Z * [new branch] gh/anshul-si/24/orig -> origin/gh/anshul-si/24/orig 2025-09-07T06:42:00.6946779Z * [new branch] gh/anshul-si/25/base -> origin/gh/anshul-si/25/base 2025-09-07T06:42:00.6947146Z * [new branch] gh/anshul-si/25/head -> origin/gh/anshul-si/25/head 2025-09-07T06:42:00.6947486Z * [new branch] gh/anshul-si/25/orig -> origin/gh/anshul-si/25/orig 2025-09-07T06:42:00.6947833Z * [new branch] gh/anshul-si/26/base -> origin/gh/anshul-si/26/base 2025-09-07T06:42:00.6948181Z * [new branch] gh/anshul-si/26/head -> origin/gh/anshul-si/26/head 2025-09-07T06:42:00.6948523Z * [new branch] gh/anshul-si/26/orig -> origin/gh/anshul-si/26/orig 2025-09-07T06:42:00.6948875Z * [new branch] gh/anshul-si/27/base -> origin/gh/anshul-si/27/base 2025-09-07T06:42:00.6949218Z * [new branch] gh/anshul-si/27/head -> origin/gh/anshul-si/27/head 2025-09-07T06:42:00.6949566Z * [new branch] gh/anshul-si/27/orig -> origin/gh/anshul-si/27/orig 2025-09-07T06:42:00.6949909Z * [new branch] gh/anshul-si/28/base -> origin/gh/anshul-si/28/base 2025-09-07T06:42:00.6950162Z * [new branch] gh/anshul-si/28/head -> origin/gh/anshul-si/28/head 2025-09-07T06:42:00.6950309Z * [new branch] gh/anshul-si/28/orig -> origin/gh/anshul-si/28/orig 2025-09-07T06:42:00.6950447Z * [new branch] gh/anshul-si/29/base -> origin/gh/anshul-si/29/base 2025-09-07T06:42:00.6950585Z * [new branch] gh/anshul-si/29/head -> origin/gh/anshul-si/29/head 2025-09-07T06:42:00.6950735Z * [new branch] gh/anshul-si/29/orig -> origin/gh/anshul-si/29/orig 2025-09-07T06:42:00.6951052Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-09-07T06:42:00.6951226Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-09-07T06:42:00.6952670Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-09-07T06:42:00.6952922Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-09-07T06:42:00.6956341Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-09-07T06:42:00.6956604Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-09-07T06:42:00.6956869Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-09-07T06:42:00.6959323Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-09-07T06:42:00.6959653Z * [new branch] gh/bdhirsh/650/base -> origin/gh/bdhirsh/650/base 2025-09-07T06:42:00.6959972Z * [new branch] gh/bdhirsh/650/head -> origin/gh/bdhirsh/650/head 2025-09-07T06:42:00.6960117Z * [new branch] gh/bdhirsh/650/orig -> origin/gh/bdhirsh/650/orig 2025-09-07T06:42:00.6964153Z * [new branch] gh/bdhirsh/663/base -> origin/gh/bdhirsh/663/base 2025-09-07T06:42:00.6964491Z * [new branch] gh/bdhirsh/663/head -> origin/gh/bdhirsh/663/head 2025-09-07T06:42:00.6964709Z * [new branch] gh/bdhirsh/663/orig -> origin/gh/bdhirsh/663/orig 2025-09-07T06:42:00.6964863Z * [new branch] gh/bdhirsh/665/base -> origin/gh/bdhirsh/665/base 2025-09-07T06:42:00.6965024Z * [new branch] gh/bdhirsh/665/head -> origin/gh/bdhirsh/665/head 2025-09-07T06:42:00.6965572Z * [new branch] gh/bdhirsh/665/orig -> origin/gh/bdhirsh/665/orig 2025-09-07T06:42:00.6966020Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-09-07T06:42:00.6970173Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-09-07T06:42:00.6970356Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-09-07T06:42:00.6970500Z * [new branch] gh/bdhirsh/667/base -> origin/gh/bdhirsh/667/base 2025-09-07T06:42:00.6970633Z * [new branch] gh/bdhirsh/667/head -> origin/gh/bdhirsh/667/head 2025-09-07T06:42:00.6970798Z * [new branch] gh/bdhirsh/667/orig -> origin/gh/bdhirsh/667/orig 2025-09-07T06:42:00.6970933Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-09-07T06:42:00.6974114Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-09-07T06:42:00.6974261Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-09-07T06:42:00.6974394Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-09-07T06:42:00.6974543Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-09-07T06:42:00.6974672Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-09-07T06:42:00.6974810Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-09-07T06:42:00.6978443Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-09-07T06:42:00.6978724Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-09-07T06:42:00.6978910Z * [new branch] gh/benjaminglass1/100/base -> origin/gh/benjaminglass1/100/base 2025-09-07T06:42:00.6979067Z * [new branch] gh/benjaminglass1/100/head -> origin/gh/benjaminglass1/100/head 2025-09-07T06:42:00.6979232Z * [new branch] gh/benjaminglass1/100/orig -> origin/gh/benjaminglass1/100/orig 2025-09-07T06:42:00.6979393Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-09-07T06:42:00.6979555Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-09-07T06:42:00.6979715Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-09-07T06:42:00.6979897Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-09-07T06:42:00.6980779Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-09-07T06:42:00.6981011Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-09-07T06:42:00.6982357Z * [new branch] gh/benjaminglass1/103/base -> origin/gh/benjaminglass1/103/base 2025-09-07T06:42:00.6982774Z * [new branch] gh/benjaminglass1/103/head -> origin/gh/benjaminglass1/103/head 2025-09-07T06:42:00.6986044Z * [new branch] gh/benjaminglass1/103/orig -> origin/gh/benjaminglass1/103/orig 2025-09-07T06:42:00.6986305Z * [new branch] gh/benjaminglass1/104/base -> origin/gh/benjaminglass1/104/base 2025-09-07T06:42:00.6986481Z * [new branch] gh/benjaminglass1/104/head -> origin/gh/benjaminglass1/104/head 2025-09-07T06:42:00.6986649Z * [new branch] gh/benjaminglass1/104/orig -> origin/gh/benjaminglass1/104/orig 2025-09-07T06:42:00.6990474Z * [new branch] gh/benjaminglass1/105/base -> origin/gh/benjaminglass1/105/base 2025-09-07T06:42:00.6990695Z * [new branch] gh/benjaminglass1/105/head -> origin/gh/benjaminglass1/105/head 2025-09-07T06:42:00.6990869Z * [new branch] gh/benjaminglass1/105/orig -> origin/gh/benjaminglass1/105/orig 2025-09-07T06:42:00.6991057Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-09-07T06:42:00.6991243Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-09-07T06:42:00.6991463Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-09-07T06:42:00.6996362Z * [new branch] gh/benjaminglass1/79/base -> origin/gh/benjaminglass1/79/base 2025-09-07T06:42:00.6999498Z * [new branch] gh/benjaminglass1/79/head -> origin/gh/benjaminglass1/79/head 2025-09-07T06:42:00.7000057Z * [new branch] gh/benjaminglass1/79/orig -> origin/gh/benjaminglass1/79/orig 2025-09-07T06:42:00.7000306Z * [new branch] gh/benjaminglass1/86/base -> origin/gh/benjaminglass1/86/base 2025-09-07T06:42:00.7000491Z * [new branch] gh/benjaminglass1/86/head -> origin/gh/benjaminglass1/86/head 2025-09-07T06:42:00.7000667Z * [new branch] gh/benjaminglass1/86/orig -> origin/gh/benjaminglass1/86/orig 2025-09-07T06:42:00.7000837Z * [new branch] gh/benjaminglass1/89/base -> origin/gh/benjaminglass1/89/base 2025-09-07T06:42:00.7001023Z * [new branch] gh/benjaminglass1/89/head -> origin/gh/benjaminglass1/89/head 2025-09-07T06:42:00.7001194Z * [new branch] gh/benjaminglass1/89/orig -> origin/gh/benjaminglass1/89/orig 2025-09-07T06:42:00.7001362Z * [new branch] gh/benjaminglass1/91/base -> origin/gh/benjaminglass1/91/base 2025-09-07T06:42:00.7001537Z * [new branch] gh/benjaminglass1/91/head -> origin/gh/benjaminglass1/91/head 2025-09-07T06:42:00.7001847Z * [new branch] gh/benjaminglass1/91/orig -> origin/gh/benjaminglass1/91/orig 2025-09-07T06:42:00.7004202Z * [new branch] gh/benjaminglass1/93/base -> origin/gh/benjaminglass1/93/base 2025-09-07T06:42:00.7004380Z * [new branch] gh/benjaminglass1/93/head -> origin/gh/benjaminglass1/93/head 2025-09-07T06:42:00.7004564Z * [new branch] gh/benjaminglass1/93/orig -> origin/gh/benjaminglass1/93/orig 2025-09-07T06:42:00.7004726Z * [new branch] gh/benjaminglass1/95/base -> origin/gh/benjaminglass1/95/base 2025-09-07T06:42:00.7004901Z * [new branch] gh/benjaminglass1/95/head -> origin/gh/benjaminglass1/95/head 2025-09-07T06:42:00.7005079Z * [new branch] gh/benjaminglass1/95/orig -> origin/gh/benjaminglass1/95/orig 2025-09-07T06:42:00.7007964Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-09-07T06:42:00.7008168Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-09-07T06:42:00.7012489Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-09-07T06:42:00.7012666Z * [new branch] gh/benjaminglass1/99/base -> origin/gh/benjaminglass1/99/base 2025-09-07T06:42:00.7013090Z * [new branch] gh/benjaminglass1/99/head -> origin/gh/benjaminglass1/99/head 2025-09-07T06:42:00.7013275Z * [new branch] gh/benjaminglass1/99/orig -> origin/gh/benjaminglass1/99/orig 2025-09-07T06:42:00.7013623Z * [new branch] gh/bobrenjc93/514/base -> origin/gh/bobrenjc93/514/base 2025-09-07T06:42:00.7013780Z * [new branch] gh/bobrenjc93/514/head -> origin/gh/bobrenjc93/514/head 2025-09-07T06:42:00.7013939Z * [new branch] gh/bobrenjc93/514/orig -> origin/gh/bobrenjc93/514/orig 2025-09-07T06:42:00.7014087Z * [new branch] gh/bobrenjc93/521/base -> origin/gh/bobrenjc93/521/base 2025-09-07T06:42:00.7014268Z * [new branch] gh/bobrenjc93/521/head -> origin/gh/bobrenjc93/521/head 2025-09-07T06:42:00.7014421Z * [new branch] gh/bobrenjc93/521/orig -> origin/gh/bobrenjc93/521/orig 2025-09-07T06:42:00.7019875Z * [new branch] gh/bobrenjc93/522/base -> origin/gh/bobrenjc93/522/base 2025-09-07T06:42:00.7020238Z * [new branch] gh/bobrenjc93/522/head -> origin/gh/bobrenjc93/522/head 2025-09-07T06:42:00.7020401Z * [new branch] gh/bobrenjc93/522/orig -> origin/gh/bobrenjc93/522/orig 2025-09-07T06:42:00.7020608Z * [new branch] gh/bobrenjc93/525/base -> origin/gh/bobrenjc93/525/base 2025-09-07T06:42:00.7020771Z * [new branch] gh/bobrenjc93/525/head -> origin/gh/bobrenjc93/525/head 2025-09-07T06:42:00.7020925Z * [new branch] gh/bobrenjc93/525/orig -> origin/gh/bobrenjc93/525/orig 2025-09-07T06:42:00.7021092Z * [new branch] gh/bobrenjc93/526/base -> origin/gh/bobrenjc93/526/base 2025-09-07T06:42:00.7021276Z * [new branch] gh/bobrenjc93/526/head -> origin/gh/bobrenjc93/526/head 2025-09-07T06:42:00.7026860Z * [new branch] gh/bobrenjc93/526/orig -> origin/gh/bobrenjc93/526/orig 2025-09-07T06:42:00.7031801Z * [new branch] gh/bobrenjc93/527/base -> origin/gh/bobrenjc93/527/base 2025-09-07T06:42:00.7037250Z * [new branch] gh/bobrenjc93/527/head -> origin/gh/bobrenjc93/527/head 2025-09-07T06:42:00.7042762Z * [new branch] gh/bobrenjc93/527/orig -> origin/gh/bobrenjc93/527/orig 2025-09-07T06:42:00.7043163Z * [new branch] gh/bobrenjc93/528/base -> origin/gh/bobrenjc93/528/base 2025-09-07T06:42:00.7043309Z * [new branch] gh/bobrenjc93/528/head -> origin/gh/bobrenjc93/528/head 2025-09-07T06:42:00.7043472Z * [new branch] gh/bobrenjc93/528/orig -> origin/gh/bobrenjc93/528/orig 2025-09-07T06:42:00.7043829Z * [new branch] gh/bobrenjc93/529/base -> origin/gh/bobrenjc93/529/base 2025-09-07T06:42:00.7043997Z * [new branch] gh/bobrenjc93/529/head -> origin/gh/bobrenjc93/529/head 2025-09-07T06:42:00.7044136Z * [new branch] gh/bobrenjc93/529/orig -> origin/gh/bobrenjc93/529/orig 2025-09-07T06:42:00.7044282Z * [new branch] gh/bobrenjc93/535/base -> origin/gh/bobrenjc93/535/base 2025-09-07T06:42:00.7044421Z * [new branch] gh/bobrenjc93/535/head -> origin/gh/bobrenjc93/535/head 2025-09-07T06:42:00.7044562Z * [new branch] gh/bobrenjc93/535/orig -> origin/gh/bobrenjc93/535/orig 2025-09-07T06:42:00.7044707Z * [new branch] gh/bobrenjc93/537/base -> origin/gh/bobrenjc93/537/base 2025-09-07T06:42:00.7044841Z * [new branch] gh/bobrenjc93/537/head -> origin/gh/bobrenjc93/537/head 2025-09-07T06:42:00.7044985Z * [new branch] gh/bobrenjc93/537/orig -> origin/gh/bobrenjc93/537/orig 2025-09-07T06:42:00.7045127Z * [new branch] gh/bobrenjc93/539/base -> origin/gh/bobrenjc93/539/base 2025-09-07T06:42:00.7045273Z * [new branch] gh/bobrenjc93/539/head -> origin/gh/bobrenjc93/539/head 2025-09-07T06:42:00.7045414Z * [new branch] gh/bobrenjc93/539/orig -> origin/gh/bobrenjc93/539/orig 2025-09-07T06:42:00.7045546Z * [new branch] gh/bobrenjc93/540/base -> origin/gh/bobrenjc93/540/base 2025-09-07T06:42:00.7045686Z * [new branch] gh/bobrenjc93/540/head -> origin/gh/bobrenjc93/540/head 2025-09-07T06:42:00.7045882Z * [new branch] gh/bobrenjc93/540/orig -> origin/gh/bobrenjc93/540/orig 2025-09-07T06:42:00.7046028Z * [new branch] gh/bobrenjc93/541/base -> origin/gh/bobrenjc93/541/base 2025-09-07T06:42:00.7046162Z * [new branch] gh/bobrenjc93/541/head -> origin/gh/bobrenjc93/541/head 2025-09-07T06:42:00.7046300Z * [new branch] gh/bobrenjc93/541/orig -> origin/gh/bobrenjc93/541/orig 2025-09-07T06:42:00.7046442Z * [new branch] gh/bobrenjc93/542/base -> origin/gh/bobrenjc93/542/base 2025-09-07T06:42:00.7046582Z * [new branch] gh/bobrenjc93/542/head -> origin/gh/bobrenjc93/542/head 2025-09-07T06:42:00.7046729Z * [new branch] gh/bobrenjc93/542/orig -> origin/gh/bobrenjc93/542/orig 2025-09-07T06:42:00.7046868Z * [new branch] gh/bobrenjc93/543/base -> origin/gh/bobrenjc93/543/base 2025-09-07T06:42:00.7047013Z * [new branch] gh/bobrenjc93/543/head -> origin/gh/bobrenjc93/543/head 2025-09-07T06:42:00.7051896Z * [new branch] gh/bobrenjc93/543/orig -> origin/gh/bobrenjc93/543/orig 2025-09-07T06:42:00.7057517Z * [new branch] gh/bobrenjc93/544/base -> origin/gh/bobrenjc93/544/base 2025-09-07T06:42:00.7059943Z * [new branch] gh/bobrenjc93/544/head -> origin/gh/bobrenjc93/544/head 2025-09-07T06:42:00.7060152Z * [new branch] gh/bobrenjc93/544/orig -> origin/gh/bobrenjc93/544/orig 2025-09-07T06:42:00.7060306Z * [new branch] gh/bobrenjc93/545/base -> origin/gh/bobrenjc93/545/base 2025-09-07T06:42:00.7060451Z * [new branch] gh/bobrenjc93/545/head -> origin/gh/bobrenjc93/545/head 2025-09-07T06:42:00.7060590Z * [new branch] gh/bobrenjc93/545/orig -> origin/gh/bobrenjc93/545/orig 2025-09-07T06:42:00.7060739Z * [new branch] gh/bobrenjc93/546/base -> origin/gh/bobrenjc93/546/base 2025-09-07T06:42:00.7060883Z * [new branch] gh/bobrenjc93/546/head -> origin/gh/bobrenjc93/546/head 2025-09-07T06:42:00.7061035Z * [new branch] gh/bobrenjc93/546/orig -> origin/gh/bobrenjc93/546/orig 2025-09-07T06:42:00.7061184Z * [new branch] gh/bobrenjc93/547/base -> origin/gh/bobrenjc93/547/base 2025-09-07T06:42:00.7061336Z * [new branch] gh/bobrenjc93/547/head -> origin/gh/bobrenjc93/547/head 2025-09-07T06:42:00.7061626Z * [new branch] gh/bobrenjc93/547/orig -> origin/gh/bobrenjc93/547/orig 2025-09-07T06:42:00.7061783Z * [new branch] gh/bobrenjc93/548/base -> origin/gh/bobrenjc93/548/base 2025-09-07T06:42:00.7061937Z * [new branch] gh/bobrenjc93/548/head -> origin/gh/bobrenjc93/548/head 2025-09-07T06:42:00.7062084Z * [new branch] gh/bobrenjc93/548/orig -> origin/gh/bobrenjc93/548/orig 2025-09-07T06:42:00.7062242Z * [new branch] gh/bobrenjc93/549/base -> origin/gh/bobrenjc93/549/base 2025-09-07T06:42:00.7062393Z * [new branch] gh/bobrenjc93/549/head -> origin/gh/bobrenjc93/549/head 2025-09-07T06:42:00.7062548Z * [new branch] gh/bobrenjc93/549/orig -> origin/gh/bobrenjc93/549/orig 2025-09-07T06:42:00.7062703Z * [new branch] gh/bobrenjc93/550/base -> origin/gh/bobrenjc93/550/base 2025-09-07T06:42:00.7062864Z * [new branch] gh/bobrenjc93/550/head -> origin/gh/bobrenjc93/550/head 2025-09-07T06:42:00.7063342Z * [new branch] gh/bobrenjc93/550/orig -> origin/gh/bobrenjc93/550/orig 2025-09-07T06:42:00.7067493Z * [new branch] gh/bobrenjc93/551/base -> origin/gh/bobrenjc93/551/base 2025-09-07T06:42:00.7071089Z * [new branch] gh/bobrenjc93/551/head -> origin/gh/bobrenjc93/551/head 2025-09-07T06:42:00.7071272Z * [new branch] gh/bobrenjc93/551/orig -> origin/gh/bobrenjc93/551/orig 2025-09-07T06:42:00.7071557Z * [new branch] gh/bobrenjc93/552/base -> origin/gh/bobrenjc93/552/base 2025-09-07T06:42:00.7071714Z * [new branch] gh/bobrenjc93/552/head -> origin/gh/bobrenjc93/552/head 2025-09-07T06:42:00.7071857Z * [new branch] gh/bobrenjc93/552/orig -> origin/gh/bobrenjc93/552/orig 2025-09-07T06:42:00.7072004Z * [new branch] gh/bobrenjc93/553/base -> origin/gh/bobrenjc93/553/base 2025-09-07T06:42:00.7072152Z * [new branch] gh/bobrenjc93/553/head -> origin/gh/bobrenjc93/553/head 2025-09-07T06:42:00.7072305Z * [new branch] gh/bobrenjc93/553/orig -> origin/gh/bobrenjc93/553/orig 2025-09-07T06:42:00.7072461Z * [new branch] gh/bobrenjc93/554/base -> origin/gh/bobrenjc93/554/base 2025-09-07T06:42:00.7076829Z * [new branch] gh/bobrenjc93/554/head -> origin/gh/bobrenjc93/554/head 2025-09-07T06:42:00.7082179Z * [new branch] gh/bobrenjc93/554/orig -> origin/gh/bobrenjc93/554/orig 2025-09-07T06:42:00.7087955Z * [new branch] gh/bobrenjc93/555/base -> origin/gh/bobrenjc93/555/base 2025-09-07T06:42:00.7092742Z * [new branch] gh/bobrenjc93/555/head -> origin/gh/bobrenjc93/555/head 2025-09-07T06:42:00.7097548Z * [new branch] gh/bobrenjc93/555/orig -> origin/gh/bobrenjc93/555/orig 2025-09-07T06:42:00.7099491Z * [new branch] gh/bobrenjc93/556/base -> origin/gh/bobrenjc93/556/base 2025-09-07T06:42:00.7099696Z * [new branch] gh/bobrenjc93/556/head -> origin/gh/bobrenjc93/556/head 2025-09-07T06:42:00.7099840Z * [new branch] gh/bobrenjc93/556/orig -> origin/gh/bobrenjc93/556/orig 2025-09-07T06:42:00.7100014Z * [new branch] gh/briancoutinho/2/base -> origin/gh/briancoutinho/2/base 2025-09-07T06:42:00.7100165Z * [new branch] gh/briancoutinho/2/head -> origin/gh/briancoutinho/2/head 2025-09-07T06:42:00.7100307Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-09-07T06:42:00.7100438Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-09-07T06:42:00.7100558Z * [new branch] gh/c00w/48/base -> origin/gh/c00w/48/base 2025-09-07T06:42:00.7100684Z * [new branch] gh/c00w/48/head -> origin/gh/c00w/48/head 2025-09-07T06:42:00.7100946Z * [new branch] gh/c00w/48/orig -> origin/gh/c00w/48/orig 2025-09-07T06:42:00.7101078Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-09-07T06:42:00.7101217Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-09-07T06:42:00.7101348Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-09-07T06:42:00.7101486Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-09-07T06:42:00.7101622Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-09-07T06:42:00.7101762Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-09-07T06:42:00.7101897Z * [new branch] gh/c00w/55/base -> origin/gh/c00w/55/base 2025-09-07T06:42:00.7102029Z * [new branch] gh/c00w/55/head -> origin/gh/c00w/55/head 2025-09-07T06:42:00.7102181Z * [new branch] gh/c00w/55/orig -> origin/gh/c00w/55/orig 2025-09-07T06:42:00.7102313Z * [new branch] gh/c00w/56/base -> origin/gh/c00w/56/base 2025-09-07T06:42:00.7102452Z * [new branch] gh/c00w/56/head -> origin/gh/c00w/56/head 2025-09-07T06:42:00.7102584Z * [new branch] gh/c00w/56/orig -> origin/gh/c00w/56/orig 2025-09-07T06:42:00.7102733Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-09-07T06:42:00.7102939Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-09-07T06:42:00.7103086Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-09-07T06:42:00.7103263Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-09-07T06:42:00.7103438Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-09-07T06:42:00.7103619Z * [new branch] gh/coconutruben/11/base -> origin/gh/coconutruben/11/base 2025-09-07T06:42:00.7103786Z * [new branch] gh/coconutruben/11/head -> origin/gh/coconutruben/11/head 2025-09-07T06:42:00.7103947Z * [new branch] gh/coconutruben/11/orig -> origin/gh/coconutruben/11/orig 2025-09-07T06:42:00.7104117Z * [new branch] gh/coconutruben/12/base -> origin/gh/coconutruben/12/base 2025-09-07T06:42:00.7104272Z * [new branch] gh/coconutruben/12/head -> origin/gh/coconutruben/12/head 2025-09-07T06:42:00.7104439Z * [new branch] gh/coconutruben/12/orig -> origin/gh/coconutruben/12/orig 2025-09-07T06:42:00.7104603Z * [new branch] gh/coconutruben/13/base -> origin/gh/coconutruben/13/base 2025-09-07T06:42:00.7104805Z * [new branch] gh/coconutruben/13/head -> origin/gh/coconutruben/13/head 2025-09-07T06:42:00.7104971Z * [new branch] gh/coconutruben/13/orig -> origin/gh/coconutruben/13/orig 2025-09-07T06:42:00.7106080Z * [new branch] gh/coconutruben/14/base -> origin/gh/coconutruben/14/base 2025-09-07T06:42:00.7106569Z * [new branch] gh/coconutruben/14/head -> origin/gh/coconutruben/14/head 2025-09-07T06:42:00.7107031Z * [new branch] gh/coconutruben/14/orig -> origin/gh/coconutruben/14/orig 2025-09-07T06:42:00.7110187Z * [new branch] gh/coconutruben/15/base -> origin/gh/coconutruben/15/base 2025-09-07T06:42:00.7110540Z * [new branch] gh/coconutruben/15/head -> origin/gh/coconutruben/15/head 2025-09-07T06:42:00.7110727Z * [new branch] gh/coconutruben/15/orig -> origin/gh/coconutruben/15/orig 2025-09-07T06:42:00.7111030Z * [new branch] gh/coconutruben/16/base -> origin/gh/coconutruben/16/base 2025-09-07T06:42:00.7111591Z * [new branch] gh/coconutruben/16/head -> origin/gh/coconutruben/16/head 2025-09-07T06:42:00.7112517Z * [new branch] gh/coconutruben/16/orig -> origin/gh/coconutruben/16/orig 2025-09-07T06:42:00.7116987Z * [new branch] gh/coconutruben/17/base -> origin/gh/coconutruben/17/base 2025-09-07T06:42:00.7117366Z * [new branch] gh/coconutruben/17/head -> origin/gh/coconutruben/17/head 2025-09-07T06:42:00.7117634Z * [new branch] gh/coconutruben/17/orig -> origin/gh/coconutruben/17/orig 2025-09-07T06:42:00.7117844Z * [new branch] gh/coconutruben/18/base -> origin/gh/coconutruben/18/base 2025-09-07T06:42:00.7118108Z * [new branch] gh/coconutruben/18/head -> origin/gh/coconutruben/18/head 2025-09-07T06:42:00.7118833Z * [new branch] gh/coconutruben/18/orig -> origin/gh/coconutruben/18/orig 2025-09-07T06:42:00.7119196Z * [new branch] gh/coconutruben/19/base -> origin/gh/coconutruben/19/base 2025-09-07T06:42:00.7119435Z * [new branch] gh/coconutruben/19/head -> origin/gh/coconutruben/19/head 2025-09-07T06:42:00.7120965Z * [new branch] gh/coconutruben/19/orig -> origin/gh/coconutruben/19/orig 2025-09-07T06:42:00.7121360Z * [new branch] gh/coconutruben/20/base -> origin/gh/coconutruben/20/base 2025-09-07T06:42:00.7124319Z * [new branch] gh/coconutruben/20/head -> origin/gh/coconutruben/20/head 2025-09-07T06:42:00.7124688Z * [new branch] gh/coconutruben/20/orig -> origin/gh/coconutruben/20/orig 2025-09-07T06:42:00.7124934Z * [new branch] gh/coconutruben/21/base -> origin/gh/coconutruben/21/base 2025-09-07T06:42:00.7125349Z * [new branch] gh/coconutruben/21/head -> origin/gh/coconutruben/21/head 2025-09-07T06:42:00.7125664Z * [new branch] gh/coconutruben/21/orig -> origin/gh/coconutruben/21/orig 2025-09-07T06:42:00.7126331Z * [new branch] gh/coconutruben/22/base -> origin/gh/coconutruben/22/base 2025-09-07T06:42:00.7126557Z * [new branch] gh/coconutruben/22/head -> origin/gh/coconutruben/22/head 2025-09-07T06:42:00.7130989Z * [new branch] gh/coconutruben/22/orig -> origin/gh/coconutruben/22/orig 2025-09-07T06:42:00.7131330Z * [new branch] gh/coconutruben/24/base -> origin/gh/coconutruben/24/base 2025-09-07T06:42:00.7131586Z * [new branch] gh/coconutruben/24/head -> origin/gh/coconutruben/24/head 2025-09-07T06:42:00.7131844Z * [new branch] gh/coconutruben/24/orig -> origin/gh/coconutruben/24/orig 2025-09-07T06:42:00.7132062Z * [new branch] gh/coconutruben/25/base -> origin/gh/coconutruben/25/base 2025-09-07T06:42:00.7132213Z * [new branch] gh/coconutruben/25/head -> origin/gh/coconutruben/25/head 2025-09-07T06:42:00.7136517Z * [new branch] gh/coconutruben/25/orig -> origin/gh/coconutruben/25/orig 2025-09-07T06:42:00.7136866Z * [new branch] gh/coconutruben/28/base -> origin/gh/coconutruben/28/base 2025-09-07T06:42:00.7137056Z * [new branch] gh/coconutruben/28/head -> origin/gh/coconutruben/28/head 2025-09-07T06:42:00.7137211Z * [new branch] gh/coconutruben/28/orig -> origin/gh/coconutruben/28/orig 2025-09-07T06:42:00.7137355Z * [new branch] gh/coconutruben/29/base -> origin/gh/coconutruben/29/base 2025-09-07T06:42:00.7137649Z * [new branch] gh/coconutruben/29/head -> origin/gh/coconutruben/29/head 2025-09-07T06:42:00.7137826Z * [new branch] gh/coconutruben/29/orig -> origin/gh/coconutruben/29/orig 2025-09-07T06:42:00.7142235Z * [new branch] gh/coconutruben/30/base -> origin/gh/coconutruben/30/base 2025-09-07T06:42:00.7142432Z * [new branch] gh/coconutruben/30/head -> origin/gh/coconutruben/30/head 2025-09-07T06:42:00.7142660Z * [new branch] gh/coconutruben/30/orig -> origin/gh/coconutruben/30/orig 2025-09-07T06:42:00.7142836Z * [new branch] gh/coconutruben/31/base -> origin/gh/coconutruben/31/base 2025-09-07T06:42:00.7143223Z * [new branch] gh/coconutruben/31/head -> origin/gh/coconutruben/31/head 2025-09-07T06:42:00.7143612Z * [new branch] gh/coconutruben/31/orig -> origin/gh/coconutruben/31/orig 2025-09-07T06:42:00.7147925Z * [new branch] gh/coconutruben/32/base -> origin/gh/coconutruben/32/base 2025-09-07T06:42:00.7153660Z * [new branch] gh/coconutruben/32/head -> origin/gh/coconutruben/32/head 2025-09-07T06:42:00.7155978Z * [new branch] gh/coconutruben/32/orig -> origin/gh/coconutruben/32/orig 2025-09-07T06:42:00.7163753Z * [new branch] gh/coconutruben/33/base -> origin/gh/coconutruben/33/base 2025-09-07T06:42:00.7172771Z * [new branch] gh/coconutruben/33/head -> origin/gh/coconutruben/33/head 2025-09-07T06:42:00.7172971Z * [new branch] gh/coconutruben/33/orig -> origin/gh/coconutruben/33/orig 2025-09-07T06:42:00.7173502Z * [new branch] gh/coconutruben/34/base -> origin/gh/coconutruben/34/base 2025-09-07T06:42:00.7173686Z * [new branch] gh/coconutruben/34/head -> origin/gh/coconutruben/34/head 2025-09-07T06:42:00.7173854Z * [new branch] gh/coconutruben/34/orig -> origin/gh/coconutruben/34/orig 2025-09-07T06:42:00.7174028Z * [new branch] gh/coconutruben/35/base -> origin/gh/coconutruben/35/base 2025-09-07T06:42:00.7174193Z * [new branch] gh/coconutruben/35/head -> origin/gh/coconutruben/35/head 2025-09-07T06:42:00.7174485Z * [new branch] gh/coconutruben/35/orig -> origin/gh/coconutruben/35/orig 2025-09-07T06:42:00.7174687Z * [new branch] gh/coconutruben/36/base -> origin/gh/coconutruben/36/base 2025-09-07T06:42:00.7174852Z * [new branch] gh/coconutruben/36/head -> origin/gh/coconutruben/36/head 2025-09-07T06:42:00.7175021Z * [new branch] gh/coconutruben/36/orig -> origin/gh/coconutruben/36/orig 2025-09-07T06:42:00.7175189Z * [new branch] gh/coconutruben/37/base -> origin/gh/coconutruben/37/base 2025-09-07T06:42:00.7175357Z * [new branch] gh/coconutruben/37/head -> origin/gh/coconutruben/37/head 2025-09-07T06:42:00.7175528Z * [new branch] gh/coconutruben/37/orig -> origin/gh/coconutruben/37/orig 2025-09-07T06:42:00.7175673Z * [new branch] gh/coconutruben/38/base -> origin/gh/coconutruben/38/base 2025-09-07T06:42:00.7175826Z * [new branch] gh/coconutruben/38/head -> origin/gh/coconutruben/38/head 2025-09-07T06:42:00.7175970Z * [new branch] gh/coconutruben/38/orig -> origin/gh/coconutruben/38/orig 2025-09-07T06:42:00.7176154Z * [new branch] gh/coconutruben/39/base -> origin/gh/coconutruben/39/base 2025-09-07T06:42:00.7176319Z * [new branch] gh/coconutruben/39/head -> origin/gh/coconutruben/39/head 2025-09-07T06:42:00.7176484Z * [new branch] gh/coconutruben/39/orig -> origin/gh/coconutruben/39/orig 2025-09-07T06:42:00.7176653Z * [new branch] gh/coconutruben/40/base -> origin/gh/coconutruben/40/base 2025-09-07T06:42:00.7176819Z * [new branch] gh/coconutruben/40/head -> origin/gh/coconutruben/40/head 2025-09-07T06:42:00.7176963Z * [new branch] gh/coconutruben/40/orig -> origin/gh/coconutruben/40/orig 2025-09-07T06:42:00.7177103Z * [new branch] gh/coconutruben/41/base -> origin/gh/coconutruben/41/base 2025-09-07T06:42:00.7177249Z * [new branch] gh/coconutruben/41/head -> origin/gh/coconutruben/41/head 2025-09-07T06:42:00.7177390Z * [new branch] gh/coconutruben/41/orig -> origin/gh/coconutruben/41/orig 2025-09-07T06:42:00.7177532Z * [new branch] gh/coconutruben/42/base -> origin/gh/coconutruben/42/base 2025-09-07T06:42:00.7177683Z * [new branch] gh/coconutruben/42/head -> origin/gh/coconutruben/42/head 2025-09-07T06:42:00.7177911Z * [new branch] gh/coconutruben/42/orig -> origin/gh/coconutruben/42/orig 2025-09-07T06:42:00.7178077Z * [new branch] gh/coconutruben/43/base -> origin/gh/coconutruben/43/base 2025-09-07T06:42:00.7178221Z * [new branch] gh/coconutruben/43/head -> origin/gh/coconutruben/43/head 2025-09-07T06:42:00.7178373Z * [new branch] gh/coconutruben/43/orig -> origin/gh/coconutruben/43/orig 2025-09-07T06:42:00.7178521Z * [new branch] gh/coconutruben/44/base -> origin/gh/coconutruben/44/base 2025-09-07T06:42:00.7178666Z * [new branch] gh/coconutruben/44/head -> origin/gh/coconutruben/44/head 2025-09-07T06:42:00.7178818Z * [new branch] gh/coconutruben/44/orig -> origin/gh/coconutruben/44/orig 2025-09-07T06:42:00.7185116Z * [new branch] gh/coconutruben/45/base -> origin/gh/coconutruben/45/base 2025-09-07T06:42:00.7185363Z * [new branch] gh/coconutruben/45/head -> origin/gh/coconutruben/45/head 2025-09-07T06:42:00.7185546Z * [new branch] gh/coconutruben/45/orig -> origin/gh/coconutruben/45/orig 2025-09-07T06:42:00.7185995Z * [new branch] gh/coconutruben/46/base -> origin/gh/coconutruben/46/base 2025-09-07T06:42:00.7186166Z * [new branch] gh/coconutruben/46/head -> origin/gh/coconutruben/46/head 2025-09-07T06:42:00.7186331Z * [new branch] gh/coconutruben/46/orig -> origin/gh/coconutruben/46/orig 2025-09-07T06:42:00.7186647Z * [new branch] gh/coconutruben/47/base -> origin/gh/coconutruben/47/base 2025-09-07T06:42:00.7186804Z * [new branch] gh/coconutruben/47/head -> origin/gh/coconutruben/47/head 2025-09-07T06:42:00.7186972Z * [new branch] gh/coconutruben/47/orig -> origin/gh/coconutruben/47/orig 2025-09-07T06:42:00.7187140Z * [new branch] gh/coconutruben/48/base -> origin/gh/coconutruben/48/base 2025-09-07T06:42:00.7187337Z * [new branch] gh/coconutruben/48/head -> origin/gh/coconutruben/48/head 2025-09-07T06:42:00.7187565Z * [new branch] gh/coconutruben/48/orig -> origin/gh/coconutruben/48/orig 2025-09-07T06:42:00.7190667Z * [new branch] gh/coconutruben/49/base -> origin/gh/coconutruben/49/base 2025-09-07T06:42:00.7191020Z * [new branch] gh/coconutruben/49/head -> origin/gh/coconutruben/49/head 2025-09-07T06:42:00.7191231Z * [new branch] gh/coconutruben/49/orig -> origin/gh/coconutruben/49/orig 2025-09-07T06:42:00.7191472Z * [new branch] gh/coconutruben/50/base -> origin/gh/coconutruben/50/base 2025-09-07T06:42:00.7196279Z * [new branch] gh/coconutruben/50/head -> origin/gh/coconutruben/50/head 2025-09-07T06:42:00.7200581Z * [new branch] gh/coconutruben/50/orig -> origin/gh/coconutruben/50/orig 2025-09-07T06:42:00.7200948Z * [new branch] gh/coconutruben/51/base -> origin/gh/coconutruben/51/base 2025-09-07T06:42:00.7201132Z * [new branch] gh/coconutruben/51/head -> origin/gh/coconutruben/51/head 2025-09-07T06:42:00.7201280Z * [new branch] gh/coconutruben/51/orig -> origin/gh/coconutruben/51/orig 2025-09-07T06:42:00.7201431Z * [new branch] gh/coconutruben/52/base -> origin/gh/coconutruben/52/base 2025-09-07T06:42:00.7201574Z * [new branch] gh/coconutruben/52/head -> origin/gh/coconutruben/52/head 2025-09-07T06:42:00.7201730Z * [new branch] gh/coconutruben/52/orig -> origin/gh/coconutruben/52/orig 2025-09-07T06:42:00.7201873Z * [new branch] gh/coconutruben/53/base -> origin/gh/coconutruben/53/base 2025-09-07T06:42:00.7202017Z * [new branch] gh/coconutruben/53/head -> origin/gh/coconutruben/53/head 2025-09-07T06:42:00.7202166Z * [new branch] gh/coconutruben/53/orig -> origin/gh/coconutruben/53/orig 2025-09-07T06:42:00.7202459Z * [new branch] gh/coconutruben/54/base -> origin/gh/coconutruben/54/base 2025-09-07T06:42:00.7202614Z * [new branch] gh/coconutruben/54/head -> origin/gh/coconutruben/54/head 2025-09-07T06:42:00.7202783Z * [new branch] gh/coconutruben/54/orig -> origin/gh/coconutruben/54/orig 2025-09-07T06:42:00.7206362Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-09-07T06:42:00.7206555Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-09-07T06:42:00.7206747Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-09-07T06:42:00.7206906Z * [new branch] gh/coconutruben/56/base -> origin/gh/coconutruben/56/base 2025-09-07T06:42:00.7207408Z * [new branch] gh/coconutruben/56/head -> origin/gh/coconutruben/56/head 2025-09-07T06:42:00.7207579Z * [new branch] gh/coconutruben/56/orig -> origin/gh/coconutruben/56/orig 2025-09-07T06:42:00.7209024Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-09-07T06:42:00.7209329Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-09-07T06:42:00.7210426Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-09-07T06:42:00.7214373Z * [new branch] gh/coconutruben/58/base -> origin/gh/coconutruben/58/base 2025-09-07T06:42:00.7214767Z * [new branch] gh/coconutruben/58/head -> origin/gh/coconutruben/58/head 2025-09-07T06:42:00.7214939Z * [new branch] gh/coconutruben/58/orig -> origin/gh/coconutruben/58/orig 2025-09-07T06:42:00.7215093Z * [new branch] gh/coconutruben/59/base -> origin/gh/coconutruben/59/base 2025-09-07T06:42:00.7215250Z * [new branch] gh/coconutruben/59/head -> origin/gh/coconutruben/59/head 2025-09-07T06:42:00.7215557Z * [new branch] gh/coconutruben/59/orig -> origin/gh/coconutruben/59/orig 2025-09-07T06:42:00.7216290Z * [new branch] gh/coconutruben/60/base -> origin/gh/coconutruben/60/base 2025-09-07T06:42:00.7219084Z * [new branch] gh/coconutruben/60/head -> origin/gh/coconutruben/60/head 2025-09-07T06:42:00.7219283Z * [new branch] gh/coconutruben/60/orig -> origin/gh/coconutruben/60/orig 2025-09-07T06:42:00.7219452Z * [new branch] gh/coconutruben/61/base -> origin/gh/coconutruben/61/base 2025-09-07T06:42:00.7219779Z * [new branch] gh/coconutruben/61/head -> origin/gh/coconutruben/61/head 2025-09-07T06:42:00.7220456Z * [new branch] gh/coconutruben/61/orig -> origin/gh/coconutruben/61/orig 2025-09-07T06:42:00.7221751Z * [new branch] gh/coconutruben/62/base -> origin/gh/coconutruben/62/base 2025-09-07T06:42:00.7222111Z * [new branch] gh/coconutruben/62/head -> origin/gh/coconutruben/62/head 2025-09-07T06:42:00.7223240Z * [new branch] gh/coconutruben/62/orig -> origin/gh/coconutruben/62/orig 2025-09-07T06:42:00.7224765Z * [new branch] gh/coconutruben/63/base -> origin/gh/coconutruben/63/base 2025-09-07T06:42:00.7225059Z * [new branch] gh/coconutruben/63/head -> origin/gh/coconutruben/63/head 2025-09-07T06:42:00.7226166Z * [new branch] gh/coconutruben/63/orig -> origin/gh/coconutruben/63/orig 2025-09-07T06:42:00.7231422Z * [new branch] gh/coconutruben/64/base -> origin/gh/coconutruben/64/base 2025-09-07T06:42:00.7231741Z * [new branch] gh/coconutruben/64/head -> origin/gh/coconutruben/64/head 2025-09-07T06:42:00.7231914Z * [new branch] gh/coconutruben/64/orig -> origin/gh/coconutruben/64/orig 2025-09-07T06:42:00.7232067Z * [new branch] gh/coconutruben/65/base -> origin/gh/coconutruben/65/base 2025-09-07T06:42:00.7232429Z * [new branch] gh/coconutruben/65/head -> origin/gh/coconutruben/65/head 2025-09-07T06:42:00.7232589Z * [new branch] gh/coconutruben/65/orig -> origin/gh/coconutruben/65/orig 2025-09-07T06:42:00.7232737Z * [new branch] gh/coconutruben/66/base -> origin/gh/coconutruben/66/base 2025-09-07T06:42:00.7232888Z * [new branch] gh/coconutruben/66/head -> origin/gh/coconutruben/66/head 2025-09-07T06:42:00.7233067Z * [new branch] gh/coconutruben/66/orig -> origin/gh/coconutruben/66/orig 2025-09-07T06:42:00.7235137Z * [new branch] gh/codingwithsurya/12/base -> origin/gh/codingwithsurya/12/base 2025-09-07T06:42:00.7235526Z * [new branch] gh/codingwithsurya/12/head -> origin/gh/codingwithsurya/12/head 2025-09-07T06:42:00.7236010Z * [new branch] gh/codingwithsurya/12/orig -> origin/gh/codingwithsurya/12/orig 2025-09-07T06:42:00.7237557Z * [new branch] gh/codingwithsurya/14/base -> origin/gh/codingwithsurya/14/base 2025-09-07T06:42:00.7237750Z * [new branch] gh/codingwithsurya/14/head -> origin/gh/codingwithsurya/14/head 2025-09-07T06:42:00.7238244Z * [new branch] gh/codingwithsurya/14/orig -> origin/gh/codingwithsurya/14/orig 2025-09-07T06:42:00.7243023Z * [new branch] gh/codingwithsurya/15/base -> origin/gh/codingwithsurya/15/base 2025-09-07T06:42:00.7243236Z * [new branch] gh/codingwithsurya/15/head -> origin/gh/codingwithsurya/15/head 2025-09-07T06:42:00.7243653Z * [new branch] gh/codingwithsurya/15/orig -> origin/gh/codingwithsurya/15/orig 2025-09-07T06:42:00.7243826Z * [new branch] gh/codingwithsurya/16/base -> origin/gh/codingwithsurya/16/base 2025-09-07T06:42:00.7244009Z * [new branch] gh/codingwithsurya/16/head -> origin/gh/codingwithsurya/16/head 2025-09-07T06:42:00.7244164Z * [new branch] gh/codingwithsurya/16/orig -> origin/gh/codingwithsurya/16/orig 2025-09-07T06:42:00.7244396Z * [new branch] gh/codingwithsurya/17/base -> origin/gh/codingwithsurya/17/base 2025-09-07T06:42:00.7245260Z * [new branch] gh/codingwithsurya/17/head -> origin/gh/codingwithsurya/17/head 2025-09-07T06:42:00.7245570Z * [new branch] gh/codingwithsurya/17/orig -> origin/gh/codingwithsurya/17/orig 2025-09-07T06:42:00.7247093Z * [new branch] gh/codingwithsurya/18/base -> origin/gh/codingwithsurya/18/base 2025-09-07T06:42:00.7247295Z * [new branch] gh/codingwithsurya/18/head -> origin/gh/codingwithsurya/18/head 2025-09-07T06:42:00.7248445Z * [new branch] gh/codingwithsurya/18/orig -> origin/gh/codingwithsurya/18/orig 2025-09-07T06:42:00.7249562Z * [new branch] gh/codingwithsurya/19/base -> origin/gh/codingwithsurya/19/base 2025-09-07T06:42:00.7250463Z * [new branch] gh/codingwithsurya/19/head -> origin/gh/codingwithsurya/19/head 2025-09-07T06:42:00.7250810Z * [new branch] gh/codingwithsurya/19/orig -> origin/gh/codingwithsurya/19/orig 2025-09-07T06:42:00.7252227Z * [new branch] gh/codingwithsurya/20/base -> origin/gh/codingwithsurya/20/base 2025-09-07T06:42:00.7252481Z * [new branch] gh/codingwithsurya/20/head -> origin/gh/codingwithsurya/20/head 2025-09-07T06:42:00.7253628Z * [new branch] gh/codingwithsurya/20/orig -> origin/gh/codingwithsurya/20/orig 2025-09-07T06:42:00.7254689Z * [new branch] gh/codingwithsurya/21/base -> origin/gh/codingwithsurya/21/base 2025-09-07T06:42:00.7255685Z * [new branch] gh/codingwithsurya/21/head -> origin/gh/codingwithsurya/21/head 2025-09-07T06:42:00.7256132Z * [new branch] gh/codingwithsurya/21/orig -> origin/gh/codingwithsurya/21/orig 2025-09-07T06:42:00.7257358Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-09-07T06:42:00.7258146Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-09-07T06:42:00.7258749Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-09-07T06:42:00.7259259Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-09-07T06:42:00.7260383Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-09-07T06:42:00.7260659Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-09-07T06:42:00.7261902Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-09-07T06:42:00.7262270Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-09-07T06:42:00.7263935Z * [new branch] gh/davidberard98/382/base -> origin/gh/davidberard98/382/base 2025-09-07T06:42:00.7264518Z * [new branch] gh/davidberard98/382/head -> origin/gh/davidberard98/382/head 2025-09-07T06:42:00.7265560Z * [new branch] gh/davidberard98/382/orig -> origin/gh/davidberard98/382/orig 2025-09-07T06:42:00.7266413Z * [new branch] gh/davidberard98/386/base -> origin/gh/davidberard98/386/base 2025-09-07T06:42:00.7271351Z * [new branch] gh/davidberard98/386/head -> origin/gh/davidberard98/386/head 2025-09-07T06:42:00.7271570Z * [new branch] gh/davidberard98/386/orig -> origin/gh/davidberard98/386/orig 2025-09-07T06:42:00.7271743Z * [new branch] gh/davidberard98/391/base -> origin/gh/davidberard98/391/base 2025-09-07T06:42:00.7272108Z * [new branch] gh/davidberard98/391/head -> origin/gh/davidberard98/391/head 2025-09-07T06:42:00.7272281Z * [new branch] gh/davidberard98/391/orig -> origin/gh/davidberard98/391/orig 2025-09-07T06:42:00.7272457Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-09-07T06:42:00.7272644Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-09-07T06:42:00.7272821Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-09-07T06:42:00.7273906Z * [new branch] gh/davidberard98/394/base -> origin/gh/davidberard98/394/base 2025-09-07T06:42:00.7274359Z * [new branch] gh/davidberard98/394/head -> origin/gh/davidberard98/394/head 2025-09-07T06:42:00.7279779Z * [new branch] gh/davidberard98/394/orig -> origin/gh/davidberard98/394/orig 2025-09-07T06:42:00.7284097Z * [new branch] gh/davidberard98/396/base -> origin/gh/davidberard98/396/base 2025-09-07T06:42:00.7288377Z * [new branch] gh/davidberard98/396/head -> origin/gh/davidberard98/396/head 2025-09-07T06:42:00.7293780Z * [new branch] gh/davidberard98/396/orig -> origin/gh/davidberard98/396/orig 2025-09-07T06:42:00.7295745Z * [new branch] gh/davidberard98/397/base -> origin/gh/davidberard98/397/base 2025-09-07T06:42:00.7295940Z * [new branch] gh/davidberard98/397/head -> origin/gh/davidberard98/397/head 2025-09-07T06:42:00.7296100Z * [new branch] gh/davidberard98/397/orig -> origin/gh/davidberard98/397/orig 2025-09-07T06:42:00.7296247Z * [new branch] gh/davidberard98/398/base -> origin/gh/davidberard98/398/base 2025-09-07T06:42:00.7296401Z * [new branch] gh/davidberard98/398/head -> origin/gh/davidberard98/398/head 2025-09-07T06:42:00.7296557Z * [new branch] gh/davidberard98/398/orig -> origin/gh/davidberard98/398/orig 2025-09-07T06:42:00.7296707Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-09-07T06:42:00.7296862Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-09-07T06:42:00.7297007Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-09-07T06:42:00.7297334Z * [new branch] gh/davidberard98/400/base -> origin/gh/davidberard98/400/base 2025-09-07T06:42:00.7297491Z * [new branch] gh/davidberard98/400/head -> origin/gh/davidberard98/400/head 2025-09-07T06:42:00.7297662Z * [new branch] gh/davidberard98/400/orig -> origin/gh/davidberard98/400/orig 2025-09-07T06:42:00.7297820Z * [new branch] gh/davidberard98/401/base -> origin/gh/davidberard98/401/base 2025-09-07T06:42:00.7297977Z * [new branch] gh/davidberard98/401/head -> origin/gh/davidberard98/401/head 2025-09-07T06:42:00.7298149Z * [new branch] gh/davidberard98/401/orig -> origin/gh/davidberard98/401/orig 2025-09-07T06:42:00.7298306Z * [new branch] gh/davidberard98/402/base -> origin/gh/davidberard98/402/base 2025-09-07T06:42:00.7298471Z * [new branch] gh/davidberard98/402/head -> origin/gh/davidberard98/402/head 2025-09-07T06:42:00.7298628Z * [new branch] gh/davidberard98/402/orig -> origin/gh/davidberard98/402/orig 2025-09-07T06:42:00.7298796Z * [new branch] gh/davidberard98/403/base -> origin/gh/davidberard98/403/base 2025-09-07T06:42:00.7298952Z * [new branch] gh/davidberard98/403/head -> origin/gh/davidberard98/403/head 2025-09-07T06:42:00.7299102Z * [new branch] gh/davidberard98/403/orig -> origin/gh/davidberard98/403/orig 2025-09-07T06:42:00.7299260Z * [new branch] gh/davidberard98/404/base -> origin/gh/davidberard98/404/base 2025-09-07T06:42:00.7299451Z * [new branch] gh/davidberard98/404/head -> origin/gh/davidberard98/404/head 2025-09-07T06:42:00.7299613Z * [new branch] gh/davidberard98/404/orig -> origin/gh/davidberard98/404/orig 2025-09-07T06:42:00.7299764Z * [new branch] gh/davidberard98/405/base -> origin/gh/davidberard98/405/base 2025-09-07T06:42:00.7299916Z * [new branch] gh/davidberard98/405/head -> origin/gh/davidberard98/405/head 2025-09-07T06:42:00.7300082Z * [new branch] gh/davidberard98/405/orig -> origin/gh/davidberard98/405/orig 2025-09-07T06:42:00.7300245Z * [new branch] gh/davidberard98/406/base -> origin/gh/davidberard98/406/base 2025-09-07T06:42:00.7300979Z * [new branch] gh/davidberard98/406/head -> origin/gh/davidberard98/406/head 2025-09-07T06:42:00.7301810Z * [new branch] gh/davidberard98/406/orig -> origin/gh/davidberard98/406/orig 2025-09-07T06:42:00.7303435Z * [new branch] gh/davidberard98/407/base -> origin/gh/davidberard98/407/base 2025-09-07T06:42:00.7303727Z * [new branch] gh/davidberard98/407/head -> origin/gh/davidberard98/407/head 2025-09-07T06:42:00.7304810Z * [new branch] gh/davidberard98/407/orig -> origin/gh/davidberard98/407/orig 2025-09-07T06:42:00.7305827Z * [new branch] gh/davidberard98/408/base -> origin/gh/davidberard98/408/base 2025-09-07T06:42:00.7306136Z * [new branch] gh/davidberard98/408/head -> origin/gh/davidberard98/408/head 2025-09-07T06:42:00.7310408Z * [new branch] gh/davidberard98/408/orig -> origin/gh/davidberard98/408/orig 2025-09-07T06:42:00.7310589Z * [new branch] gh/davidberard98/409/base -> origin/gh/davidberard98/409/base 2025-09-07T06:42:00.7310748Z * [new branch] gh/davidberard98/409/head -> origin/gh/davidberard98/409/head 2025-09-07T06:42:00.7310921Z * [new branch] gh/davidberard98/409/orig -> origin/gh/davidberard98/409/orig 2025-09-07T06:42:00.7311108Z * [new branch] gh/desertfire/594/base -> origin/gh/desertfire/594/base 2025-09-07T06:42:00.7311285Z * [new branch] gh/desertfire/594/head -> origin/gh/desertfire/594/head 2025-09-07T06:42:00.7313012Z * [new branch] gh/desertfire/594/orig -> origin/gh/desertfire/594/orig 2025-09-07T06:42:00.7313266Z * [new branch] gh/desertfire/595/base -> origin/gh/desertfire/595/base 2025-09-07T06:42:00.7313710Z * [new branch] gh/desertfire/595/head -> origin/gh/desertfire/595/head 2025-09-07T06:42:00.7314758Z * [new branch] gh/desertfire/595/orig -> origin/gh/desertfire/595/orig 2025-09-07T06:42:00.7315462Z * [new branch] gh/desertfire/597/base -> origin/gh/desertfire/597/base 2025-09-07T06:42:00.7316140Z * [new branch] gh/desertfire/597/head -> origin/gh/desertfire/597/head 2025-09-07T06:42:00.7317695Z * [new branch] gh/desertfire/597/orig -> origin/gh/desertfire/597/orig 2025-09-07T06:42:00.7318848Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-09-07T06:42:00.7319246Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-09-07T06:42:00.7319867Z * [new branch] gh/drisspg/149/base -> origin/gh/drisspg/149/base 2025-09-07T06:42:00.7328727Z * [new branch] gh/drisspg/149/head -> origin/gh/drisspg/149/head 2025-09-07T06:42:00.7328926Z * [new branch] gh/drisspg/149/orig -> origin/gh/drisspg/149/orig 2025-09-07T06:42:00.7329069Z * [new branch] gh/drisspg/159/base -> origin/gh/drisspg/159/base 2025-09-07T06:42:00.7329214Z * [new branch] gh/drisspg/159/head -> origin/gh/drisspg/159/head 2025-09-07T06:42:00.7329352Z * [new branch] gh/drisspg/159/orig -> origin/gh/drisspg/159/orig 2025-09-07T06:42:00.7331426Z * [new branch] gh/drisspg/166/base -> origin/gh/drisspg/166/base 2025-09-07T06:42:00.7331636Z * [new branch] gh/drisspg/166/head -> origin/gh/drisspg/166/head 2025-09-07T06:42:00.7331818Z * [new branch] gh/drisspg/166/orig -> origin/gh/drisspg/166/orig 2025-09-07T06:42:00.7331975Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-09-07T06:42:00.7332160Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-09-07T06:42:00.7332312Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-09-07T06:42:00.7337453Z * [new branch] gh/drisspg/173/base -> origin/gh/drisspg/173/base 2025-09-07T06:42:00.7337793Z * [new branch] gh/drisspg/173/head -> origin/gh/drisspg/173/head 2025-09-07T06:42:00.7337976Z * [new branch] gh/drisspg/173/orig -> origin/gh/drisspg/173/orig 2025-09-07T06:42:00.7338150Z * [new branch] gh/drisspg/177/base -> origin/gh/drisspg/177/base 2025-09-07T06:42:00.7338302Z * [new branch] gh/drisspg/177/head -> origin/gh/drisspg/177/head 2025-09-07T06:42:00.7338597Z * [new branch] gh/drisspg/177/orig -> origin/gh/drisspg/177/orig 2025-09-07T06:42:00.7338838Z * [new branch] gh/drisspg/178/base -> origin/gh/drisspg/178/base 2025-09-07T06:42:00.7339768Z * [new branch] gh/drisspg/178/head -> origin/gh/drisspg/178/head 2025-09-07T06:42:00.7340499Z * [new branch] gh/drisspg/178/orig -> origin/gh/drisspg/178/orig 2025-09-07T06:42:00.7340847Z * [new branch] gh/drisspg/180/base -> origin/gh/drisspg/180/base 2025-09-07T06:42:00.7341018Z * [new branch] gh/drisspg/180/head -> origin/gh/drisspg/180/head 2025-09-07T06:42:00.7341170Z * [new branch] gh/drisspg/180/orig -> origin/gh/drisspg/180/orig 2025-09-07T06:42:00.7341460Z * [new branch] gh/drisspg/181/base -> origin/gh/drisspg/181/base 2025-09-07T06:42:00.7341643Z * [new branch] gh/drisspg/181/head -> origin/gh/drisspg/181/head 2025-09-07T06:42:00.7341786Z * [new branch] gh/drisspg/181/orig -> origin/gh/drisspg/181/orig 2025-09-07T06:42:00.7342057Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-09-07T06:42:00.7342582Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-09-07T06:42:00.7344001Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-09-07T06:42:00.7344200Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-09-07T06:42:00.7344709Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-09-07T06:42:00.7345473Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-09-07T06:42:00.7346799Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-09-07T06:42:00.7347029Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-09-07T06:42:00.7350441Z * [new branch] gh/drisspg/186/base -> origin/gh/drisspg/186/base 2025-09-07T06:42:00.7355579Z * [new branch] gh/drisspg/186/head -> origin/gh/drisspg/186/head 2025-09-07T06:42:00.7357557Z * [new branch] gh/drisspg/186/orig -> origin/gh/drisspg/186/orig 2025-09-07T06:42:00.7357725Z * [new branch] gh/drisspg/187/base -> origin/gh/drisspg/187/base 2025-09-07T06:42:00.7357883Z * [new branch] gh/drisspg/187/head -> origin/gh/drisspg/187/head 2025-09-07T06:42:00.7358032Z * [new branch] gh/drisspg/187/orig -> origin/gh/drisspg/187/orig 2025-09-07T06:42:00.7358196Z * [new branch] gh/drisspg/188/base -> origin/gh/drisspg/188/base 2025-09-07T06:42:00.7358507Z * [new branch] gh/drisspg/188/head -> origin/gh/drisspg/188/head 2025-09-07T06:42:00.7358661Z * [new branch] gh/drisspg/188/orig -> origin/gh/drisspg/188/orig 2025-09-07T06:42:00.7358822Z * [new branch] gh/drisspg/189/base -> origin/gh/drisspg/189/base 2025-09-07T06:42:00.7358958Z * [new branch] gh/drisspg/189/head -> origin/gh/drisspg/189/head 2025-09-07T06:42:00.7359111Z * [new branch] gh/drisspg/189/orig -> origin/gh/drisspg/189/orig 2025-09-07T06:42:00.7359248Z * [new branch] gh/drisspg/190/base -> origin/gh/drisspg/190/base 2025-09-07T06:42:00.7364654Z * [new branch] gh/drisspg/190/head -> origin/gh/drisspg/190/head 2025-09-07T06:42:00.7366902Z * [new branch] gh/drisspg/190/orig -> origin/gh/drisspg/190/orig 2025-09-07T06:42:00.7367203Z * [new branch] gh/drisspg/191/base -> origin/gh/drisspg/191/base 2025-09-07T06:42:00.7372000Z * [new branch] gh/drisspg/191/head -> origin/gh/drisspg/191/head 2025-09-07T06:42:00.7376910Z * [new branch] gh/drisspg/191/orig -> origin/gh/drisspg/191/orig 2025-09-07T06:42:00.7379789Z * [new branch] gh/drisspg/192/base -> origin/gh/drisspg/192/base 2025-09-07T06:42:00.7380095Z * [new branch] gh/drisspg/192/head -> origin/gh/drisspg/192/head 2025-09-07T06:42:00.7380279Z * [new branch] gh/drisspg/192/orig -> origin/gh/drisspg/192/orig 2025-09-07T06:42:00.7380429Z * [new branch] gh/drisspg/193/base -> origin/gh/drisspg/193/base 2025-09-07T06:42:00.7380711Z * [new branch] gh/drisspg/193/head -> origin/gh/drisspg/193/head 2025-09-07T06:42:00.7380860Z * [new branch] gh/drisspg/193/orig -> origin/gh/drisspg/193/orig 2025-09-07T06:42:00.7381107Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-09-07T06:42:00.7381669Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-09-07T06:42:00.7381864Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-09-07T06:42:00.7382022Z * [new branch] gh/drisspg/195/base -> origin/gh/drisspg/195/base 2025-09-07T06:42:00.7382357Z * [new branch] gh/drisspg/195/head -> origin/gh/drisspg/195/head 2025-09-07T06:42:00.7382506Z * [new branch] gh/drisspg/195/orig -> origin/gh/drisspg/195/orig 2025-09-07T06:42:00.7382651Z * [new branch] gh/drisspg/196/base -> origin/gh/drisspg/196/base 2025-09-07T06:42:00.7382805Z * [new branch] gh/drisspg/196/head -> origin/gh/drisspg/196/head 2025-09-07T06:42:00.7382949Z * [new branch] gh/drisspg/196/orig -> origin/gh/drisspg/196/orig 2025-09-07T06:42:00.7383111Z * [new branch] gh/drisspg/197/base -> origin/gh/drisspg/197/base 2025-09-07T06:42:00.7383256Z * [new branch] gh/drisspg/197/head -> origin/gh/drisspg/197/head 2025-09-07T06:42:00.7383408Z * [new branch] gh/drisspg/197/orig -> origin/gh/drisspg/197/orig 2025-09-07T06:42:00.7383553Z * [new branch] gh/drisspg/198/base -> origin/gh/drisspg/198/base 2025-09-07T06:42:00.7383703Z * [new branch] gh/drisspg/198/head -> origin/gh/drisspg/198/head 2025-09-07T06:42:00.7383853Z * [new branch] gh/drisspg/198/orig -> origin/gh/drisspg/198/orig 2025-09-07T06:42:00.7383994Z * [new branch] gh/drisspg/199/base -> origin/gh/drisspg/199/base 2025-09-07T06:42:00.7384142Z * [new branch] gh/drisspg/199/head -> origin/gh/drisspg/199/head 2025-09-07T06:42:00.7384282Z * [new branch] gh/drisspg/199/orig -> origin/gh/drisspg/199/orig 2025-09-07T06:42:00.7384482Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-09-07T06:42:00.7384642Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-09-07T06:42:00.7384813Z * [new branch] gh/eellison/784/base -> origin/gh/eellison/784/base 2025-09-07T06:42:00.7384968Z * [new branch] gh/eellison/784/head -> origin/gh/eellison/784/head 2025-09-07T06:42:00.7385117Z * [new branch] gh/eellison/784/orig -> origin/gh/eellison/784/orig 2025-09-07T06:42:00.7385277Z * [new branch] gh/eellison/785/base -> origin/gh/eellison/785/base 2025-09-07T06:42:00.7385470Z * [new branch] gh/eellison/785/head -> origin/gh/eellison/785/head 2025-09-07T06:42:00.7386830Z * [new branch] gh/eellison/785/orig -> origin/gh/eellison/785/orig 2025-09-07T06:42:00.7394565Z * [new branch] gh/eellison/789/base -> origin/gh/eellison/789/base 2025-09-07T06:42:00.7398826Z * [new branch] gh/eellison/789/head -> origin/gh/eellison/789/head 2025-09-07T06:42:00.7403171Z * [new branch] gh/eellison/789/orig -> origin/gh/eellison/789/orig 2025-09-07T06:42:00.7409270Z * [new branch] gh/eellison/800/base -> origin/gh/eellison/800/base 2025-09-07T06:42:00.7414308Z * [new branch] gh/eellison/800/head -> origin/gh/eellison/800/head 2025-09-07T06:42:00.7419885Z * [new branch] gh/eellison/800/orig -> origin/gh/eellison/800/orig 2025-09-07T06:42:00.7421593Z * [new branch] gh/eellison/801/base -> origin/gh/eellison/801/base 2025-09-07T06:42:00.7422051Z * [new branch] gh/eellison/801/head -> origin/gh/eellison/801/head 2025-09-07T06:42:00.7422255Z * [new branch] gh/eellison/801/orig -> origin/gh/eellison/801/orig 2025-09-07T06:42:00.7422413Z * [new branch] gh/eellison/802/base -> origin/gh/eellison/802/base 2025-09-07T06:42:00.7422582Z * [new branch] gh/eellison/802/head -> origin/gh/eellison/802/head 2025-09-07T06:42:00.7422747Z * [new branch] gh/eellison/802/orig -> origin/gh/eellison/802/orig 2025-09-07T06:42:00.7422893Z * [new branch] gh/eellison/805/base -> origin/gh/eellison/805/base 2025-09-07T06:42:00.7423042Z * [new branch] gh/eellison/805/head -> origin/gh/eellison/805/head 2025-09-07T06:42:00.7423440Z * [new branch] gh/eellison/805/orig -> origin/gh/eellison/805/orig 2025-09-07T06:42:00.7423599Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-09-07T06:42:00.7423748Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-09-07T06:42:00.7423897Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-09-07T06:42:00.7424058Z * [new branch] gh/eellison/809/base -> origin/gh/eellison/809/base 2025-09-07T06:42:00.7424208Z * [new branch] gh/eellison/809/head -> origin/gh/eellison/809/head 2025-09-07T06:42:00.7424361Z * [new branch] gh/eellison/809/orig -> origin/gh/eellison/809/orig 2025-09-07T06:42:00.7424509Z * [new branch] gh/eellison/813/base -> origin/gh/eellison/813/base 2025-09-07T06:42:00.7424669Z * [new branch] gh/eellison/813/head -> origin/gh/eellison/813/head 2025-09-07T06:42:00.7424821Z * [new branch] gh/eellison/813/orig -> origin/gh/eellison/813/orig 2025-09-07T06:42:00.7424972Z * [new branch] gh/eellison/814/base -> origin/gh/eellison/814/base 2025-09-07T06:42:00.7425132Z * [new branch] gh/eellison/814/head -> origin/gh/eellison/814/head 2025-09-07T06:42:00.7425288Z * [new branch] gh/eellison/814/orig -> origin/gh/eellison/814/orig 2025-09-07T06:42:00.7425575Z * [new branch] gh/eellison/815/base -> origin/gh/eellison/815/base 2025-09-07T06:42:00.7425884Z * [new branch] gh/eellison/815/head -> origin/gh/eellison/815/head 2025-09-07T06:42:00.7426043Z * [new branch] gh/eellison/815/orig -> origin/gh/eellison/815/orig 2025-09-07T06:42:00.7426203Z * [new branch] gh/eellison/816/base -> origin/gh/eellison/816/base 2025-09-07T06:42:00.7426355Z * [new branch] gh/eellison/816/head -> origin/gh/eellison/816/head 2025-09-07T06:42:00.7426537Z * [new branch] gh/eellison/816/orig -> origin/gh/eellison/816/orig 2025-09-07T06:42:00.7426690Z * [new branch] gh/eellison/817/base -> origin/gh/eellison/817/base 2025-09-07T06:42:00.7426851Z * [new branch] gh/eellison/817/head -> origin/gh/eellison/817/head 2025-09-07T06:42:00.7427004Z * [new branch] gh/eellison/817/orig -> origin/gh/eellison/817/orig 2025-09-07T06:42:00.7427164Z * [new branch] gh/eellison/818/base -> origin/gh/eellison/818/base 2025-09-07T06:42:00.7427308Z * [new branch] gh/eellison/818/head -> origin/gh/eellison/818/head 2025-09-07T06:42:00.7427441Z * [new branch] gh/eellison/818/orig -> origin/gh/eellison/818/orig 2025-09-07T06:42:00.7427586Z * [new branch] gh/eellison/819/base -> origin/gh/eellison/819/base 2025-09-07T06:42:00.7427724Z * [new branch] gh/eellison/819/head -> origin/gh/eellison/819/head 2025-09-07T06:42:00.7427864Z * [new branch] gh/eellison/819/orig -> origin/gh/eellison/819/orig 2025-09-07T06:42:00.7427999Z * [new branch] gh/eellison/820/base -> origin/gh/eellison/820/base 2025-09-07T06:42:00.7428132Z * [new branch] gh/eellison/820/head -> origin/gh/eellison/820/head 2025-09-07T06:42:00.7428274Z * [new branch] gh/eellison/820/orig -> origin/gh/eellison/820/orig 2025-09-07T06:42:00.7428409Z * [new branch] gh/eellison/821/base -> origin/gh/eellison/821/base 2025-09-07T06:42:00.7428566Z * [new branch] gh/eellison/821/head -> origin/gh/eellison/821/head 2025-09-07T06:42:00.7428698Z * [new branch] gh/eellison/821/orig -> origin/gh/eellison/821/orig 2025-09-07T06:42:00.7428830Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-09-07T06:42:00.7429018Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-09-07T06:42:00.7429151Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-09-07T06:42:00.7429289Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-09-07T06:42:00.7429423Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-09-07T06:42:00.7429571Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-09-07T06:42:00.7429711Z * [new branch] gh/etaf/132/base -> origin/gh/etaf/132/base 2025-09-07T06:42:00.7430087Z * [new branch] gh/etaf/132/head -> origin/gh/etaf/132/head 2025-09-07T06:42:00.7431923Z * [new branch] gh/etaf/132/orig -> origin/gh/etaf/132/orig 2025-09-07T06:42:00.7432130Z * [new branch] gh/etaf/138/base -> origin/gh/etaf/138/base 2025-09-07T06:42:00.7432334Z * [new branch] gh/etaf/138/head -> origin/gh/etaf/138/head 2025-09-07T06:42:00.7434227Z * [new branch] gh/etaf/138/orig -> origin/gh/etaf/138/orig 2025-09-07T06:42:00.7434575Z * [new branch] gh/etaf/140/base -> origin/gh/etaf/140/base 2025-09-07T06:42:00.7434855Z * [new branch] gh/etaf/140/head -> origin/gh/etaf/140/head 2025-09-07T06:42:00.7437453Z * [new branch] gh/etaf/140/orig -> origin/gh/etaf/140/orig 2025-09-07T06:42:00.7437819Z * [new branch] gh/etaf/143/base -> origin/gh/etaf/143/base 2025-09-07T06:42:00.7438074Z * [new branch] gh/etaf/143/head -> origin/gh/etaf/143/head 2025-09-07T06:42:00.7438232Z * [new branch] gh/etaf/143/orig -> origin/gh/etaf/143/orig 2025-09-07T06:42:00.7439759Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-09-07T06:42:00.7440121Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-09-07T06:42:00.7440618Z * [new branch] gh/etaf/151/base -> origin/gh/etaf/151/base 2025-09-07T06:42:00.7442976Z * [new branch] gh/etaf/151/head -> origin/gh/etaf/151/head 2025-09-07T06:42:00.7443305Z * [new branch] gh/etaf/151/orig -> origin/gh/etaf/151/orig 2025-09-07T06:42:00.7443464Z * [new branch] gh/etaf/152/base -> origin/gh/etaf/152/base 2025-09-07T06:42:00.7443970Z * [new branch] gh/etaf/152/head -> origin/gh/etaf/152/head 2025-09-07T06:42:00.7445337Z * [new branch] gh/etaf/152/orig -> origin/gh/etaf/152/orig 2025-09-07T06:42:00.7445651Z * [new branch] gh/etaf/153/base -> origin/gh/etaf/153/base 2025-09-07T06:42:00.7451073Z * [new branch] gh/etaf/153/head -> origin/gh/etaf/153/head 2025-09-07T06:42:00.7451275Z * [new branch] gh/etaf/153/orig -> origin/gh/etaf/153/orig 2025-09-07T06:42:00.7451425Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-09-07T06:42:00.7451562Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-09-07T06:42:00.7451705Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-09-07T06:42:00.7451835Z * [new branch] gh/etaf/155/base -> origin/gh/etaf/155/base 2025-09-07T06:42:00.7456523Z * [new branch] gh/etaf/155/head -> origin/gh/etaf/155/head 2025-09-07T06:42:00.7456880Z * [new branch] gh/etaf/155/orig -> origin/gh/etaf/155/orig 2025-09-07T06:42:00.7457051Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-09-07T06:42:00.7457186Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-09-07T06:42:00.7458960Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-09-07T06:42:00.7459710Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-09-07T06:42:00.7459893Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-09-07T06:42:00.7460150Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-09-07T06:42:00.7460294Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-09-07T06:42:00.7460450Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-09-07T06:42:00.7460597Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-09-07T06:42:00.7460739Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-09-07T06:42:00.7460871Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-09-07T06:42:00.7461200Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-09-07T06:42:00.7462144Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-09-07T06:42:00.7462687Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-09-07T06:42:00.7463631Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-09-07T06:42:00.7464630Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-09-07T06:42:00.7465187Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-09-07T06:42:00.7466038Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-09-07T06:42:00.7467624Z * [new branch] gh/etaf/162/base -> origin/gh/etaf/162/base 2025-09-07T06:42:00.7467797Z * [new branch] gh/etaf/162/head -> origin/gh/etaf/162/head 2025-09-07T06:42:00.7468961Z * [new branch] gh/etaf/162/orig -> origin/gh/etaf/162/orig 2025-09-07T06:42:00.7469982Z * [new branch] gh/etaf/163/base -> origin/gh/etaf/163/base 2025-09-07T06:42:00.7470475Z * [new branch] gh/etaf/163/head -> origin/gh/etaf/163/head 2025-09-07T06:42:00.7471404Z * [new branch] gh/etaf/163/orig -> origin/gh/etaf/163/orig 2025-09-07T06:42:00.7472472Z * [new branch] gh/etaf/164/base -> origin/gh/etaf/164/base 2025-09-07T06:42:00.7472950Z * [new branch] gh/etaf/164/head -> origin/gh/etaf/164/head 2025-09-07T06:42:00.7473990Z * [new branch] gh/etaf/164/orig -> origin/gh/etaf/164/orig 2025-09-07T06:42:00.7474952Z * [new branch] gh/etaf/165/base -> origin/gh/etaf/165/base 2025-09-07T06:42:00.7475499Z * [new branch] gh/etaf/165/orig -> origin/gh/etaf/165/orig 2025-09-07T06:42:00.7479110Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-09-07T06:42:00.7484266Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-09-07T06:42:00.7491706Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-09-07T06:42:00.7495667Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-09-07T06:42:00.7500288Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-09-07T06:42:00.7502452Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-09-07T06:42:00.7502600Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-09-07T06:42:00.7502758Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-09-07T06:42:00.7502898Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-09-07T06:42:00.7503187Z * [new branch] gh/etaf/169/base -> origin/gh/etaf/169/base 2025-09-07T06:42:00.7503329Z * [new branch] gh/etaf/169/head -> origin/gh/etaf/169/head 2025-09-07T06:42:00.7503476Z * [new branch] gh/etaf/169/orig -> origin/gh/etaf/169/orig 2025-09-07T06:42:00.7503668Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-09-07T06:42:00.7503843Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-09-07T06:42:00.7504025Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-09-07T06:42:00.7504191Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-09-07T06:42:00.7504359Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-09-07T06:42:00.7504522Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-09-07T06:42:00.7504733Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-09-07T06:42:00.7504902Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-09-07T06:42:00.7505062Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-09-07T06:42:00.7505214Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-09-07T06:42:00.7505461Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-09-07T06:42:00.7505621Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-09-07T06:42:00.7505887Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-09-07T06:42:00.7506050Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-09-07T06:42:00.7506197Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-09-07T06:42:00.7506346Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-09-07T06:42:00.7506493Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-09-07T06:42:00.7506645Z * [new branch] gh/ezyang/3074/base -> origin/gh/ezyang/3074/base 2025-09-07T06:42:00.7506798Z * [new branch] gh/ezyang/3074/head -> origin/gh/ezyang/3074/head 2025-09-07T06:42:00.7506952Z * [new branch] gh/ezyang/3074/orig -> origin/gh/ezyang/3074/orig 2025-09-07T06:42:00.7507092Z * [new branch] gh/ezyang/3088/base -> origin/gh/ezyang/3088/base 2025-09-07T06:42:00.7507249Z * [new branch] gh/ezyang/3088/head -> origin/gh/ezyang/3088/head 2025-09-07T06:42:00.7507399Z * [new branch] gh/ezyang/3088/orig -> origin/gh/ezyang/3088/orig 2025-09-07T06:42:00.7507550Z * [new branch] gh/ezyang/3092/base -> origin/gh/ezyang/3092/base 2025-09-07T06:42:00.7507700Z * [new branch] gh/ezyang/3092/head -> origin/gh/ezyang/3092/head 2025-09-07T06:42:00.7511814Z * [new branch] gh/ezyang/3092/orig -> origin/gh/ezyang/3092/orig 2025-09-07T06:42:00.7516064Z * [new branch] gh/ezyang/3103/base -> origin/gh/ezyang/3103/base 2025-09-07T06:42:00.7518206Z * [new branch] gh/ezyang/3103/head -> origin/gh/ezyang/3103/head 2025-09-07T06:42:00.7518676Z * [new branch] gh/ezyang/3103/orig -> origin/gh/ezyang/3103/orig 2025-09-07T06:42:00.7518869Z * [new branch] gh/ezyang/3105/base -> origin/gh/ezyang/3105/base 2025-09-07T06:42:00.7519025Z * [new branch] gh/ezyang/3105/head -> origin/gh/ezyang/3105/head 2025-09-07T06:42:00.7519183Z * [new branch] gh/ezyang/3105/orig -> origin/gh/ezyang/3105/orig 2025-09-07T06:42:00.7519476Z * [new branch] gh/ezyang/3114/base -> origin/gh/ezyang/3114/base 2025-09-07T06:42:00.7519736Z * [new branch] gh/ezyang/3114/head -> origin/gh/ezyang/3114/head 2025-09-07T06:42:00.7519897Z * [new branch] gh/ezyang/3114/orig -> origin/gh/ezyang/3114/orig 2025-09-07T06:42:00.7520046Z * [new branch] gh/ezyang/3116/base -> origin/gh/ezyang/3116/base 2025-09-07T06:42:00.7520202Z * [new branch] gh/ezyang/3116/head -> origin/gh/ezyang/3116/head 2025-09-07T06:42:00.7520355Z * [new branch] gh/ezyang/3116/orig -> origin/gh/ezyang/3116/orig 2025-09-07T06:42:00.7520488Z * [new branch] gh/ezyang/3120/base -> origin/gh/ezyang/3120/base 2025-09-07T06:42:00.7520637Z * [new branch] gh/ezyang/3120/head -> origin/gh/ezyang/3120/head 2025-09-07T06:42:00.7520777Z * [new branch] gh/ezyang/3120/orig -> origin/gh/ezyang/3120/orig 2025-09-07T06:42:00.7522498Z * [new branch] gh/ezyang/3122/base -> origin/gh/ezyang/3122/base 2025-09-07T06:42:00.7525125Z * [new branch] gh/ezyang/3122/head -> origin/gh/ezyang/3122/head 2025-09-07T06:42:00.7525280Z * [new branch] gh/ezyang/3122/orig -> origin/gh/ezyang/3122/orig 2025-09-07T06:42:00.7525414Z * [new branch] gh/ezyang/3123/base -> origin/gh/ezyang/3123/base 2025-09-07T06:42:00.7525562Z * [new branch] gh/ezyang/3123/head -> origin/gh/ezyang/3123/head 2025-09-07T06:42:00.7525886Z * [new branch] gh/ezyang/3123/orig -> origin/gh/ezyang/3123/orig 2025-09-07T06:42:00.7526039Z * [new branch] gh/ezyang/3125/base -> origin/gh/ezyang/3125/base 2025-09-07T06:42:00.7526206Z * [new branch] gh/ezyang/3125/head -> origin/gh/ezyang/3125/head 2025-09-07T06:42:00.7526353Z * [new branch] gh/ezyang/3125/orig -> origin/gh/ezyang/3125/orig 2025-09-07T06:42:00.7531756Z * [new branch] gh/ezyang/3126/base -> origin/gh/ezyang/3126/base 2025-09-07T06:42:00.7536729Z * [new branch] gh/ezyang/3126/head -> origin/gh/ezyang/3126/head 2025-09-07T06:42:00.7536913Z * [new branch] gh/ezyang/3126/orig -> origin/gh/ezyang/3126/orig 2025-09-07T06:42:00.7537091Z * [new branch] gh/ezyang/3127/base -> origin/gh/ezyang/3127/base 2025-09-07T06:42:00.7537243Z * [new branch] gh/ezyang/3127/head -> origin/gh/ezyang/3127/head 2025-09-07T06:42:00.7537433Z * [new branch] gh/ezyang/3127/orig -> origin/gh/ezyang/3127/orig 2025-09-07T06:42:00.7537586Z * [new branch] gh/ezyang/3128/base -> origin/gh/ezyang/3128/base 2025-09-07T06:42:00.7537747Z * [new branch] gh/ezyang/3128/head -> origin/gh/ezyang/3128/head 2025-09-07T06:42:00.7537896Z * [new branch] gh/ezyang/3128/orig -> origin/gh/ezyang/3128/orig 2025-09-07T06:42:00.7538057Z * [new branch] gh/ezyang/3129/base -> origin/gh/ezyang/3129/base 2025-09-07T06:42:00.7538207Z * [new branch] gh/ezyang/3129/head -> origin/gh/ezyang/3129/head 2025-09-07T06:42:00.7538355Z * [new branch] gh/ezyang/3129/orig -> origin/gh/ezyang/3129/orig 2025-09-07T06:42:00.7538512Z * [new branch] gh/ezyang/3130/base -> origin/gh/ezyang/3130/base 2025-09-07T06:42:00.7538664Z * [new branch] gh/ezyang/3130/head -> origin/gh/ezyang/3130/head 2025-09-07T06:42:00.7538819Z * [new branch] gh/ezyang/3130/orig -> origin/gh/ezyang/3130/orig 2025-09-07T06:42:00.7538969Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-09-07T06:42:00.7541029Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-09-07T06:42:00.7541378Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-09-07T06:42:00.7541537Z * [new branch] gh/ezyang/3132/base -> origin/gh/ezyang/3132/base 2025-09-07T06:42:00.7541689Z * [new branch] gh/ezyang/3132/head -> origin/gh/ezyang/3132/head 2025-09-07T06:42:00.7541832Z * [new branch] gh/ezyang/3132/orig -> origin/gh/ezyang/3132/orig 2025-09-07T06:42:00.7542008Z * [new branch] gh/ezyang/3133/base -> origin/gh/ezyang/3133/base 2025-09-07T06:42:00.7542926Z * [new branch] gh/ezyang/3133/head -> origin/gh/ezyang/3133/head 2025-09-07T06:42:00.7543399Z * [new branch] gh/ezyang/3133/orig -> origin/gh/ezyang/3133/orig 2025-09-07T06:42:00.7544616Z * [new branch] gh/ezyang/3134/base -> origin/gh/ezyang/3134/base 2025-09-07T06:42:00.7544985Z * [new branch] gh/ezyang/3134/head -> origin/gh/ezyang/3134/head 2025-09-07T06:42:00.7546219Z * [new branch] gh/ezyang/3134/orig -> origin/gh/ezyang/3134/orig 2025-09-07T06:42:00.7547076Z * [new branch] gh/ezyang/3135/base -> origin/gh/ezyang/3135/base 2025-09-07T06:42:00.7547395Z * [new branch] gh/ezyang/3135/head -> origin/gh/ezyang/3135/head 2025-09-07T06:42:00.7548440Z * [new branch] gh/ezyang/3135/orig -> origin/gh/ezyang/3135/orig 2025-09-07T06:42:00.7549343Z * [new branch] gh/ezyang/3136/base -> origin/gh/ezyang/3136/base 2025-09-07T06:42:00.7549784Z * [new branch] gh/ezyang/3136/head -> origin/gh/ezyang/3136/head 2025-09-07T06:42:00.7550784Z * [new branch] gh/ezyang/3136/orig -> origin/gh/ezyang/3136/orig 2025-09-07T06:42:00.7553144Z * [new branch] gh/ezyang/3137/base -> origin/gh/ezyang/3137/base 2025-09-07T06:42:00.7553323Z * [new branch] gh/ezyang/3137/head -> origin/gh/ezyang/3137/head 2025-09-07T06:42:00.7553529Z * [new branch] gh/ezyang/3137/orig -> origin/gh/ezyang/3137/orig 2025-09-07T06:42:00.7553756Z * [new branch] gh/ezyang/3138/base -> origin/gh/ezyang/3138/base 2025-09-07T06:42:00.7554822Z * [new branch] gh/ezyang/3138/head -> origin/gh/ezyang/3138/head 2025-09-07T06:42:00.7555452Z * [new branch] gh/ezyang/3138/orig -> origin/gh/ezyang/3138/orig 2025-09-07T06:42:00.7556525Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-09-07T06:42:00.7556743Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-09-07T06:42:00.7559102Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-09-07T06:42:00.7559288Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-09-07T06:42:00.7559451Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-09-07T06:42:00.7559856Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-09-07T06:42:00.7565206Z * [new branch] gh/ezyang/3141/base -> origin/gh/ezyang/3141/base 2025-09-07T06:42:00.7565390Z * [new branch] gh/ezyang/3141/head -> origin/gh/ezyang/3141/head 2025-09-07T06:42:00.7565536Z * [new branch] gh/ezyang/3141/orig -> origin/gh/ezyang/3141/orig 2025-09-07T06:42:00.7565669Z * [new branch] gh/ezyang/3142/base -> origin/gh/ezyang/3142/base 2025-09-07T06:42:00.7565824Z * [new branch] gh/ezyang/3142/head -> origin/gh/ezyang/3142/head 2025-09-07T06:42:00.7565970Z * [new branch] gh/ezyang/3142/orig -> origin/gh/ezyang/3142/orig 2025-09-07T06:42:00.7566105Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-09-07T06:42:00.7566251Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-09-07T06:42:00.7566752Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-09-07T06:42:00.7569371Z * [new branch] gh/fadara01/1/base -> origin/gh/fadara01/1/base 2025-09-07T06:42:00.7569716Z * [new branch] gh/fadara01/1/head -> origin/gh/fadara01/1/head 2025-09-07T06:42:00.7569902Z * [new branch] gh/fadara01/1/orig -> origin/gh/fadara01/1/orig 2025-09-07T06:42:00.7575043Z * [new branch] gh/fduwjj/171/base -> origin/gh/fduwjj/171/base 2025-09-07T06:42:00.7575242Z * [new branch] gh/fduwjj/171/head -> origin/gh/fduwjj/171/head 2025-09-07T06:42:00.7575400Z * [new branch] gh/fduwjj/171/orig -> origin/gh/fduwjj/171/orig 2025-09-07T06:42:00.7575546Z * [new branch] gh/fduwjj/175/base -> origin/gh/fduwjj/175/base 2025-09-07T06:42:00.7575681Z * [new branch] gh/fduwjj/175/head -> origin/gh/fduwjj/175/head 2025-09-07T06:42:00.7575833Z * [new branch] gh/fduwjj/175/orig -> origin/gh/fduwjj/175/orig 2025-09-07T06:42:00.7577903Z * [new branch] gh/fduwjj/176/base -> origin/gh/fduwjj/176/base 2025-09-07T06:42:00.7578235Z * [new branch] gh/fduwjj/176/head -> origin/gh/fduwjj/176/head 2025-09-07T06:42:00.7578408Z * [new branch] gh/fduwjj/176/orig -> origin/gh/fduwjj/176/orig 2025-09-07T06:42:00.7578651Z * [new branch] gh/fduwjj/177/base -> origin/gh/fduwjj/177/base 2025-09-07T06:42:00.7580450Z * [new branch] gh/fduwjj/177/head -> origin/gh/fduwjj/177/head 2025-09-07T06:42:00.7580645Z * [new branch] gh/fduwjj/177/orig -> origin/gh/fduwjj/177/orig 2025-09-07T06:42:00.7580798Z * [new branch] gh/fduwjj/178/base -> origin/gh/fduwjj/178/base 2025-09-07T06:42:00.7581366Z * [new branch] gh/fduwjj/178/head -> origin/gh/fduwjj/178/head 2025-09-07T06:42:00.7581943Z * [new branch] gh/fduwjj/178/orig -> origin/gh/fduwjj/178/orig 2025-09-07T06:42:00.7583187Z * [new branch] gh/fduwjj/179/base -> origin/gh/fduwjj/179/base 2025-09-07T06:42:00.7583522Z * [new branch] gh/fduwjj/179/head -> origin/gh/fduwjj/179/head 2025-09-07T06:42:00.7584528Z * [new branch] gh/fduwjj/179/orig -> origin/gh/fduwjj/179/orig 2025-09-07T06:42:00.7585496Z * [new branch] gh/fduwjj/180/base -> origin/gh/fduwjj/180/base 2025-09-07T06:42:00.7586125Z * [new branch] gh/fduwjj/180/head -> origin/gh/fduwjj/180/head 2025-09-07T06:42:00.7586849Z * [new branch] gh/fduwjj/180/orig -> origin/gh/fduwjj/180/orig 2025-09-07T06:42:00.7592271Z * [new branch] gh/fduwjj/181/base -> origin/gh/fduwjj/181/base 2025-09-07T06:42:00.7592471Z * [new branch] gh/fduwjj/181/head -> origin/gh/fduwjj/181/head 2025-09-07T06:42:00.7592637Z * [new branch] gh/fduwjj/181/orig -> origin/gh/fduwjj/181/orig 2025-09-07T06:42:00.7592783Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-09-07T06:42:00.7592933Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-09-07T06:42:00.7593072Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-09-07T06:42:00.7593229Z * [new branch] gh/fduwjj/183/base -> origin/gh/fduwjj/183/base 2025-09-07T06:42:00.7593653Z * [new branch] gh/fduwjj/183/head -> origin/gh/fduwjj/183/head 2025-09-07T06:42:00.7594327Z * [new branch] gh/fduwjj/183/orig -> origin/gh/fduwjj/183/orig 2025-09-07T06:42:00.7599325Z * [new branch] gh/fduwjj/184/base -> origin/gh/fduwjj/184/base 2025-09-07T06:42:00.7599767Z * [new branch] gh/fduwjj/184/head -> origin/gh/fduwjj/184/head 2025-09-07T06:42:00.7599924Z * [new branch] gh/fduwjj/184/orig -> origin/gh/fduwjj/184/orig 2025-09-07T06:42:00.7600068Z * [new branch] gh/fduwjj/185/base -> origin/gh/fduwjj/185/base 2025-09-07T06:42:00.7600216Z * [new branch] gh/fduwjj/185/head -> origin/gh/fduwjj/185/head 2025-09-07T06:42:00.7600354Z * [new branch] gh/fduwjj/185/orig -> origin/gh/fduwjj/185/orig 2025-09-07T06:42:00.7600653Z * [new branch] gh/fduwjj/186/base -> origin/gh/fduwjj/186/base 2025-09-07T06:42:00.7601009Z * [new branch] gh/fduwjj/186/head -> origin/gh/fduwjj/186/head 2025-09-07T06:42:00.7602520Z * [new branch] gh/fduwjj/186/orig -> origin/gh/fduwjj/186/orig 2025-09-07T06:42:00.7602814Z * [new branch] gh/fduwjj/187/base -> origin/gh/fduwjj/187/base 2025-09-07T06:42:00.7603221Z * [new branch] gh/fduwjj/187/head -> origin/gh/fduwjj/187/head 2025-09-07T06:42:00.7610403Z * [new branch] gh/fduwjj/187/orig -> origin/gh/fduwjj/187/orig 2025-09-07T06:42:00.7610592Z * [new branch] gh/fduwjj/188/base -> origin/gh/fduwjj/188/base 2025-09-07T06:42:00.7610748Z * [new branch] gh/fduwjj/188/head -> origin/gh/fduwjj/188/head 2025-09-07T06:42:00.7610894Z * [new branch] gh/fduwjj/188/orig -> origin/gh/fduwjj/188/orig 2025-09-07T06:42:00.7611179Z * [new branch] gh/fduwjj/189/base -> origin/gh/fduwjj/189/base 2025-09-07T06:42:00.7611330Z * [new branch] gh/fduwjj/189/head -> origin/gh/fduwjj/189/head 2025-09-07T06:42:00.7611468Z * [new branch] gh/fduwjj/189/orig -> origin/gh/fduwjj/189/orig 2025-09-07T06:42:00.7611613Z * [new branch] gh/fduwjj/190/base -> origin/gh/fduwjj/190/base 2025-09-07T06:42:00.7611770Z * [new branch] gh/fduwjj/190/head -> origin/gh/fduwjj/190/head 2025-09-07T06:42:00.7611905Z * [new branch] gh/fduwjj/190/orig -> origin/gh/fduwjj/190/orig 2025-09-07T06:42:00.7612527Z * [new branch] gh/fduwjj/191/base -> origin/gh/fduwjj/191/base 2025-09-07T06:42:00.7612937Z * [new branch] gh/fduwjj/191/head -> origin/gh/fduwjj/191/head 2025-09-07T06:42:00.7613098Z * [new branch] gh/fduwjj/191/orig -> origin/gh/fduwjj/191/orig 2025-09-07T06:42:00.7617942Z * [new branch] gh/fegin/306/base -> origin/gh/fegin/306/base 2025-09-07T06:42:00.7618125Z * [new branch] gh/fegin/306/head -> origin/gh/fegin/306/head 2025-09-07T06:42:00.7618280Z * [new branch] gh/fegin/306/orig -> origin/gh/fegin/306/orig 2025-09-07T06:42:00.7618433Z * [new branch] gh/fegin/307/base -> origin/gh/fegin/307/base 2025-09-07T06:42:00.7618578Z * [new branch] gh/fegin/307/head -> origin/gh/fegin/307/head 2025-09-07T06:42:00.7618723Z * [new branch] gh/fegin/307/orig -> origin/gh/fegin/307/orig 2025-09-07T06:42:00.7621074Z * [new branch] gh/fegin/308/base -> origin/gh/fegin/308/base 2025-09-07T06:42:00.7621258Z * [new branch] gh/fegin/308/head -> origin/gh/fegin/308/head 2025-09-07T06:42:00.7621567Z * [new branch] gh/fegin/308/orig -> origin/gh/fegin/308/orig 2025-09-07T06:42:00.7622628Z * [new branch] gh/fegin/309/base -> origin/gh/fegin/309/base 2025-09-07T06:42:00.7622986Z * [new branch] gh/fegin/309/head -> origin/gh/fegin/309/head 2025-09-07T06:42:00.7623992Z * [new branch] gh/fegin/309/orig -> origin/gh/fegin/309/orig 2025-09-07T06:42:00.7624909Z * [new branch] gh/fegin/310/base -> origin/gh/fegin/310/base 2025-09-07T06:42:00.7625278Z * [new branch] gh/fegin/310/head -> origin/gh/fegin/310/head 2025-09-07T06:42:00.7630205Z * [new branch] gh/fegin/310/orig -> origin/gh/fegin/310/orig 2025-09-07T06:42:00.7630416Z * [new branch] gh/fegin/311/base -> origin/gh/fegin/311/base 2025-09-07T06:42:00.7630647Z * [new branch] gh/fegin/311/head -> origin/gh/fegin/311/head 2025-09-07T06:42:00.7630871Z * [new branch] gh/fegin/311/orig -> origin/gh/fegin/311/orig 2025-09-07T06:42:00.7631187Z * [new branch] gh/fegin/312/base -> origin/gh/fegin/312/base 2025-09-07T06:42:00.7631426Z * [new branch] gh/fegin/312/head -> origin/gh/fegin/312/head 2025-09-07T06:42:00.7631572Z * [new branch] gh/fegin/312/orig -> origin/gh/fegin/312/orig 2025-09-07T06:42:00.7635908Z * [new branch] gh/fegin/313/base -> origin/gh/fegin/313/base 2025-09-07T06:42:00.7636138Z * [new branch] gh/fegin/313/head -> origin/gh/fegin/313/head 2025-09-07T06:42:00.7636304Z * [new branch] gh/fegin/313/orig -> origin/gh/fegin/313/orig 2025-09-07T06:42:00.7636468Z * [new branch] gh/fffrog/124/base -> origin/gh/fffrog/124/base 2025-09-07T06:42:00.7636638Z * [new branch] gh/fffrog/124/head -> origin/gh/fffrog/124/head 2025-09-07T06:42:00.7636792Z * [new branch] gh/fffrog/124/orig -> origin/gh/fffrog/124/orig 2025-09-07T06:42:00.7640068Z * [new branch] gh/fffrog/129/base -> origin/gh/fffrog/129/base 2025-09-07T06:42:00.7640271Z * [new branch] gh/fffrog/129/head -> origin/gh/fffrog/129/head 2025-09-07T06:42:00.7640449Z * [new branch] gh/fffrog/129/orig -> origin/gh/fffrog/129/orig 2025-09-07T06:42:00.7640609Z * [new branch] gh/fffrog/130/base -> origin/gh/fffrog/130/base 2025-09-07T06:42:00.7646660Z * [new branch] gh/fffrog/130/head -> origin/gh/fffrog/130/head 2025-09-07T06:42:00.7651496Z * [new branch] gh/fffrog/130/orig -> origin/gh/fffrog/130/orig 2025-09-07T06:42:00.7657242Z * [new branch] gh/fffrog/131/base -> origin/gh/fffrog/131/base 2025-09-07T06:42:00.7659168Z * [new branch] gh/fffrog/131/head -> origin/gh/fffrog/131/head 2025-09-07T06:42:00.7659336Z * [new branch] gh/fffrog/131/orig -> origin/gh/fffrog/131/orig 2025-09-07T06:42:00.7659495Z * [new branch] gh/fffrog/132/base -> origin/gh/fffrog/132/base 2025-09-07T06:42:00.7659644Z * [new branch] gh/fffrog/132/head -> origin/gh/fffrog/132/head 2025-09-07T06:42:00.7659782Z * [new branch] gh/fffrog/132/orig -> origin/gh/fffrog/132/orig 2025-09-07T06:42:00.7659945Z * [new branch] gh/fffrog/133/base -> origin/gh/fffrog/133/base 2025-09-07T06:42:00.7660094Z * [new branch] gh/fffrog/133/head -> origin/gh/fffrog/133/head 2025-09-07T06:42:00.7660239Z * [new branch] gh/fffrog/133/orig -> origin/gh/fffrog/133/orig 2025-09-07T06:42:00.7660377Z * [new branch] gh/fffrog/134/base -> origin/gh/fffrog/134/base 2025-09-07T06:42:00.7660513Z * [new branch] gh/fffrog/134/head -> origin/gh/fffrog/134/head 2025-09-07T06:42:00.7660658Z * [new branch] gh/fffrog/134/orig -> origin/gh/fffrog/134/orig 2025-09-07T06:42:00.7660800Z * [new branch] gh/fffrog/135/base -> origin/gh/fffrog/135/base 2025-09-07T06:42:00.7660948Z * [new branch] gh/fffrog/135/head -> origin/gh/fffrog/135/head 2025-09-07T06:42:00.7661091Z * [new branch] gh/fffrog/135/orig -> origin/gh/fffrog/135/orig 2025-09-07T06:42:00.7661239Z * [new branch] gh/fffrog/136/base -> origin/gh/fffrog/136/base 2025-09-07T06:42:00.7661777Z * [new branch] gh/fffrog/136/head -> origin/gh/fffrog/136/head 2025-09-07T06:42:00.7661918Z * [new branch] gh/fffrog/136/orig -> origin/gh/fffrog/136/orig 2025-09-07T06:42:00.7662066Z * [new branch] gh/fffrog/137/base -> origin/gh/fffrog/137/base 2025-09-07T06:42:00.7662206Z * [new branch] gh/fffrog/137/head -> origin/gh/fffrog/137/head 2025-09-07T06:42:00.7662353Z * [new branch] gh/fffrog/137/orig -> origin/gh/fffrog/137/orig 2025-09-07T06:42:00.7662495Z * [new branch] gh/fffrog/138/base -> origin/gh/fffrog/138/base 2025-09-07T06:42:00.7662645Z * [new branch] gh/fffrog/138/head -> origin/gh/fffrog/138/head 2025-09-07T06:42:00.7662792Z * [new branch] gh/fffrog/138/orig -> origin/gh/fffrog/138/orig 2025-09-07T06:42:00.7662932Z * [new branch] gh/fffrog/139/base -> origin/gh/fffrog/139/base 2025-09-07T06:42:00.7663083Z * [new branch] gh/fffrog/139/head -> origin/gh/fffrog/139/head 2025-09-07T06:42:00.7663222Z * [new branch] gh/fffrog/139/orig -> origin/gh/fffrog/139/orig 2025-09-07T06:42:00.7663491Z * [new branch] gh/fffrog/140/base -> origin/gh/fffrog/140/base 2025-09-07T06:42:00.7663950Z * [new branch] gh/fffrog/140/head -> origin/gh/fffrog/140/head 2025-09-07T06:42:00.7664801Z * [new branch] gh/fffrog/140/orig -> origin/gh/fffrog/140/orig 2025-09-07T06:42:00.7666165Z * [new branch] gh/fffrog/141/base -> origin/gh/fffrog/141/base 2025-09-07T06:42:00.7667495Z * [new branch] gh/fffrog/141/head -> origin/gh/fffrog/141/head 2025-09-07T06:42:00.7667639Z * [new branch] gh/fffrog/141/orig -> origin/gh/fffrog/141/orig 2025-09-07T06:42:00.7668100Z * [new branch] gh/fffrog/142/base -> origin/gh/fffrog/142/base 2025-09-07T06:42:00.7668556Z * [new branch] gh/fffrog/142/head -> origin/gh/fffrog/142/head 2025-09-07T06:42:00.7669918Z * [new branch] gh/fffrog/142/orig -> origin/gh/fffrog/142/orig 2025-09-07T06:42:00.7673014Z * [new branch] gh/fffrog/143/base -> origin/gh/fffrog/143/base 2025-09-07T06:42:00.7673314Z * [new branch] gh/fffrog/143/head -> origin/gh/fffrog/143/head 2025-09-07T06:42:00.7673488Z * [new branch] gh/fffrog/143/orig -> origin/gh/fffrog/143/orig 2025-09-07T06:42:00.7673643Z * [new branch] gh/fffrog/144/base -> origin/gh/fffrog/144/base 2025-09-07T06:42:00.7673794Z * [new branch] gh/fffrog/144/head -> origin/gh/fffrog/144/head 2025-09-07T06:42:00.7679987Z * [new branch] gh/fffrog/144/orig -> origin/gh/fffrog/144/orig 2025-09-07T06:42:00.7680371Z * [new branch] gh/fffrog/145/base -> origin/gh/fffrog/145/base 2025-09-07T06:42:00.7680559Z * [new branch] gh/fffrog/145/head -> origin/gh/fffrog/145/head 2025-09-07T06:42:00.7680721Z * [new branch] gh/fffrog/145/orig -> origin/gh/fffrog/145/orig 2025-09-07T06:42:00.7680857Z * [new branch] gh/fffrog/146/base -> origin/gh/fffrog/146/base 2025-09-07T06:42:00.7681153Z * [new branch] gh/fffrog/146/head -> origin/gh/fffrog/146/head 2025-09-07T06:42:00.7681333Z * [new branch] gh/fffrog/146/orig -> origin/gh/fffrog/146/orig 2025-09-07T06:42:00.7686714Z * [new branch] gh/fffrog/147/base -> origin/gh/fffrog/147/base 2025-09-07T06:42:00.7687055Z * [new branch] gh/fffrog/147/head -> origin/gh/fffrog/147/head 2025-09-07T06:42:00.7687328Z * [new branch] gh/fffrog/147/orig -> origin/gh/fffrog/147/orig 2025-09-07T06:42:00.7687780Z * [new branch] gh/fffrog/148/base -> origin/gh/fffrog/148/base 2025-09-07T06:42:00.7687956Z * [new branch] gh/fffrog/148/head -> origin/gh/fffrog/148/head 2025-09-07T06:42:00.7688634Z * [new branch] gh/fffrog/148/orig -> origin/gh/fffrog/148/orig 2025-09-07T06:42:00.7688809Z * [new branch] gh/fffrog/149/base -> origin/gh/fffrog/149/base 2025-09-07T06:42:00.7689023Z * [new branch] gh/fffrog/149/head -> origin/gh/fffrog/149/head 2025-09-07T06:42:00.7689178Z * [new branch] gh/fffrog/149/orig -> origin/gh/fffrog/149/orig 2025-09-07T06:42:00.7689328Z * [new branch] gh/fffrog/150/base -> origin/gh/fffrog/150/base 2025-09-07T06:42:00.7689473Z * [new branch] gh/fffrog/150/head -> origin/gh/fffrog/150/head 2025-09-07T06:42:00.7689608Z * [new branch] gh/fffrog/150/orig -> origin/gh/fffrog/150/orig 2025-09-07T06:42:00.7689760Z * [new branch] gh/fffrog/151/base -> origin/gh/fffrog/151/base 2025-09-07T06:42:00.7689893Z * [new branch] gh/fffrog/151/head -> origin/gh/fffrog/151/head 2025-09-07T06:42:00.7690035Z * [new branch] gh/fffrog/151/orig -> origin/gh/fffrog/151/orig 2025-09-07T06:42:00.7694084Z * [new branch] gh/fffrog/152/base -> origin/gh/fffrog/152/base 2025-09-07T06:42:00.7694417Z * [new branch] gh/fffrog/152/head -> origin/gh/fffrog/152/head 2025-09-07T06:42:00.7694944Z * [new branch] gh/fffrog/153/base -> origin/gh/fffrog/153/base 2025-09-07T06:42:00.7695101Z * [new branch] gh/fffrog/153/head -> origin/gh/fffrog/153/head 2025-09-07T06:42:00.7695248Z * [new branch] gh/fffrog/153/orig -> origin/gh/fffrog/153/orig 2025-09-07T06:42:00.7695406Z * [new branch] gh/gmagogsfm/1/base -> origin/gh/gmagogsfm/1/base 2025-09-07T06:42:00.7695568Z * [new branch] gh/gmagogsfm/1/head -> origin/gh/gmagogsfm/1/head 2025-09-07T06:42:00.7695830Z * [new branch] gh/gmagogsfm/1/orig -> origin/gh/gmagogsfm/1/orig 2025-09-07T06:42:00.7697060Z * [new branch] gh/gmagogsfm/2/base -> origin/gh/gmagogsfm/2/base 2025-09-07T06:42:00.7697414Z * [new branch] gh/gmagogsfm/2/head -> origin/gh/gmagogsfm/2/head 2025-09-07T06:42:00.7697579Z * [new branch] gh/gmagogsfm/2/orig -> origin/gh/gmagogsfm/2/orig 2025-09-07T06:42:00.7699735Z * [new branch] gh/gmagogsfm/3/base -> origin/gh/gmagogsfm/3/base 2025-09-07T06:42:00.7700038Z * [new branch] gh/gmagogsfm/3/head -> origin/gh/gmagogsfm/3/head 2025-09-07T06:42:00.7700210Z * [new branch] gh/gmagogsfm/3/orig -> origin/gh/gmagogsfm/3/orig 2025-09-07T06:42:00.7702241Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-09-07T06:42:00.7702765Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-09-07T06:42:00.7702935Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-09-07T06:42:00.7703765Z * [new branch] gh/guangyey/135/base -> origin/gh/guangyey/135/base 2025-09-07T06:42:00.7704342Z * [new branch] gh/guangyey/135/head -> origin/gh/guangyey/135/head 2025-09-07T06:42:00.7705325Z * [new branch] gh/guangyey/135/orig -> origin/gh/guangyey/135/orig 2025-09-07T06:42:00.7706366Z * [new branch] gh/guangyey/139/base -> origin/gh/guangyey/139/base 2025-09-07T06:42:00.7706942Z * [new branch] gh/guangyey/139/head -> origin/gh/guangyey/139/head 2025-09-07T06:42:00.7707847Z * [new branch] gh/guangyey/139/orig -> origin/gh/guangyey/139/orig 2025-09-07T06:42:00.7709135Z * [new branch] gh/guangyey/140/base -> origin/gh/guangyey/140/base 2025-09-07T06:42:00.7709439Z * [new branch] gh/guangyey/140/head -> origin/gh/guangyey/140/head 2025-09-07T06:42:00.7709977Z * [new branch] gh/guangyey/140/orig -> origin/gh/guangyey/140/orig 2025-09-07T06:42:00.7714057Z * [new branch] gh/guangyey/142/base -> origin/gh/guangyey/142/base 2025-09-07T06:42:00.7714393Z * [new branch] gh/guangyey/142/head -> origin/gh/guangyey/142/head 2025-09-07T06:42:00.7714956Z * [new branch] gh/guangyey/142/orig -> origin/gh/guangyey/142/orig 2025-09-07T06:42:00.7715176Z * [new branch] gh/guangyey/145/base -> origin/gh/guangyey/145/base 2025-09-07T06:42:00.7715334Z * [new branch] gh/guangyey/145/head -> origin/gh/guangyey/145/head 2025-09-07T06:42:00.7715494Z * [new branch] gh/guangyey/145/orig -> origin/gh/guangyey/145/orig 2025-09-07T06:42:00.7715649Z * [new branch] gh/guangyey/153/base -> origin/gh/guangyey/153/base 2025-09-07T06:42:00.7716033Z * [new branch] gh/guangyey/153/head -> origin/gh/guangyey/153/head 2025-09-07T06:42:00.7720437Z * [new branch] gh/guangyey/153/orig -> origin/gh/guangyey/153/orig 2025-09-07T06:42:00.7720620Z * [new branch] gh/guangyey/159/base -> origin/gh/guangyey/159/base 2025-09-07T06:42:00.7720774Z * [new branch] gh/guangyey/159/head -> origin/gh/guangyey/159/head 2025-09-07T06:42:00.7721146Z * [new branch] gh/guangyey/159/orig -> origin/gh/guangyey/159/orig 2025-09-07T06:42:00.7721299Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-09-07T06:42:00.7728227Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-09-07T06:42:00.7733156Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-09-07T06:42:00.7738205Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-09-07T06:42:00.7738438Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-09-07T06:42:00.7739193Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-09-07T06:42:00.7739514Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-09-07T06:42:00.7739986Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-09-07T06:42:00.7740174Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-09-07T06:42:00.7740330Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-09-07T06:42:00.7740480Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-09-07T06:42:00.7740636Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-09-07T06:42:00.7740783Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-09-07T06:42:00.7740932Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-09-07T06:42:00.7741090Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-09-07T06:42:00.7741237Z * [new branch] gh/guangyey/174/base -> origin/gh/guangyey/174/base 2025-09-07T06:42:00.7741391Z * [new branch] gh/guangyey/174/head -> origin/gh/guangyey/174/head 2025-09-07T06:42:00.7741543Z * [new branch] gh/guangyey/174/orig -> origin/gh/guangyey/174/orig 2025-09-07T06:42:00.7741688Z * [new branch] gh/guangyey/176/base -> origin/gh/guangyey/176/base 2025-09-07T06:42:00.7741837Z * [new branch] gh/guangyey/176/head -> origin/gh/guangyey/176/head 2025-09-07T06:42:00.7741983Z * [new branch] gh/guangyey/176/orig -> origin/gh/guangyey/176/orig 2025-09-07T06:42:00.7742448Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-09-07T06:42:00.7742602Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-09-07T06:42:00.7742764Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-09-07T06:42:00.7743063Z * [new branch] gh/guangyey/181/base -> origin/gh/guangyey/181/base 2025-09-07T06:42:00.7744013Z * [new branch] gh/guangyey/181/head -> origin/gh/guangyey/181/head 2025-09-07T06:42:00.7744498Z * [new branch] gh/guangyey/181/orig -> origin/gh/guangyey/181/orig 2025-09-07T06:42:00.7745866Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-09-07T06:42:00.7746423Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-09-07T06:42:00.7750180Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-09-07T06:42:00.7750396Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-09-07T06:42:00.7750551Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-09-07T06:42:00.7750696Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-09-07T06:42:00.7750849Z * [new branch] gh/guangyey/184/base -> origin/gh/guangyey/184/base 2025-09-07T06:42:00.7751225Z * [new branch] gh/guangyey/184/head -> origin/gh/guangyey/184/head 2025-09-07T06:42:00.7753138Z * [new branch] gh/guangyey/184/orig -> origin/gh/guangyey/184/orig 2025-09-07T06:42:00.7753330Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-09-07T06:42:00.7753487Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-09-07T06:42:00.7754477Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-09-07T06:42:00.7758339Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-09-07T06:42:00.7758660Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-09-07T06:42:00.7758843Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-09-07T06:42:00.7758988Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-09-07T06:42:00.7759160Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-09-07T06:42:00.7759446Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-09-07T06:42:00.7759980Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-09-07T06:42:00.7760995Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-09-07T06:42:00.7761202Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-09-07T06:42:00.7763771Z * [new branch] gh/guangyey/189/base -> origin/gh/guangyey/189/base 2025-09-07T06:42:00.7764084Z * [new branch] gh/guangyey/189/head -> origin/gh/guangyey/189/head 2025-09-07T06:42:00.7764445Z * [new branch] gh/guangyey/189/orig -> origin/gh/guangyey/189/orig 2025-09-07T06:42:00.7764707Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-09-07T06:42:00.7765049Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-09-07T06:42:00.7766506Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-09-07T06:42:00.7766732Z * [new branch] gh/guangyey/191/base -> origin/gh/guangyey/191/base 2025-09-07T06:42:00.7767793Z * [new branch] gh/guangyey/191/head -> origin/gh/guangyey/191/head 2025-09-07T06:42:00.7768122Z * [new branch] gh/guangyey/191/orig -> origin/gh/guangyey/191/orig 2025-09-07T06:42:00.7769374Z * [new branch] gh/guangyey/192/base -> origin/gh/guangyey/192/base 2025-09-07T06:42:00.7769601Z * [new branch] gh/guangyey/192/head -> origin/gh/guangyey/192/head 2025-09-07T06:42:00.7770720Z * [new branch] gh/guangyey/192/orig -> origin/gh/guangyey/192/orig 2025-09-07T06:42:00.7771798Z * [new branch] gh/guangyey/193/base -> origin/gh/guangyey/193/base 2025-09-07T06:42:00.7772088Z * [new branch] gh/guangyey/193/head -> origin/gh/guangyey/193/head 2025-09-07T06:42:00.7773062Z * [new branch] gh/guangyey/193/orig -> origin/gh/guangyey/193/orig 2025-09-07T06:42:00.7773998Z * [new branch] gh/guangyey/194/base -> origin/gh/guangyey/194/base 2025-09-07T06:42:00.7774398Z * [new branch] gh/guangyey/194/head -> origin/gh/guangyey/194/head 2025-09-07T06:42:00.7775769Z * [new branch] gh/guangyey/194/orig -> origin/gh/guangyey/194/orig 2025-09-07T06:42:00.7776055Z * [new branch] gh/guangyey/195/base -> origin/gh/guangyey/195/base 2025-09-07T06:42:00.7777063Z * [new branch] gh/guangyey/195/head -> origin/gh/guangyey/195/head 2025-09-07T06:42:00.7777469Z * [new branch] gh/guangyey/195/orig -> origin/gh/guangyey/195/orig 2025-09-07T06:42:00.7779083Z * [new branch] gh/guangyey/196/base -> origin/gh/guangyey/196/base 2025-09-07T06:42:00.7779427Z * [new branch] gh/guangyey/196/head -> origin/gh/guangyey/196/head 2025-09-07T06:42:00.7780420Z * [new branch] gh/guangyey/196/orig -> origin/gh/guangyey/196/orig 2025-09-07T06:42:00.7781449Z * [new branch] gh/guangyey/197/base -> origin/gh/guangyey/197/base 2025-09-07T06:42:00.7781926Z * [new branch] gh/guangyey/197/head -> origin/gh/guangyey/197/head 2025-09-07T06:42:00.7783434Z * [new branch] gh/guangyey/197/orig -> origin/gh/guangyey/197/orig 2025-09-07T06:42:00.7783714Z * [new branch] gh/guangyey/198/base -> origin/gh/guangyey/198/base 2025-09-07T06:42:00.7784411Z * [new branch] gh/guangyey/198/head -> origin/gh/guangyey/198/head 2025-09-07T06:42:00.7785258Z * [new branch] gh/guangyey/198/orig -> origin/gh/guangyey/198/orig 2025-09-07T06:42:00.7789642Z * [new branch] gh/guangyey/199/base -> origin/gh/guangyey/199/base 2025-09-07T06:42:00.7789884Z * [new branch] gh/guangyey/199/head -> origin/gh/guangyey/199/head 2025-09-07T06:42:00.7790369Z * [new branch] gh/guangyey/199/orig -> origin/gh/guangyey/199/orig 2025-09-07T06:42:00.7790837Z * [new branch] gh/guangyey/200/base -> origin/gh/guangyey/200/base 2025-09-07T06:42:00.7791045Z * [new branch] gh/guangyey/200/head -> origin/gh/guangyey/200/head 2025-09-07T06:42:00.7791199Z * [new branch] gh/guangyey/200/orig -> origin/gh/guangyey/200/orig 2025-09-07T06:42:00.7797385Z * [new branch] gh/guangyey/201/base -> origin/gh/guangyey/201/base 2025-09-07T06:42:00.7801732Z * [new branch] gh/guangyey/201/head -> origin/gh/guangyey/201/head 2025-09-07T06:42:00.7807035Z * [new branch] gh/guangyey/201/orig -> origin/gh/guangyey/201/orig 2025-09-07T06:42:00.7807298Z * [new branch] gh/guangyey/202/base -> origin/gh/guangyey/202/base 2025-09-07T06:42:00.7807573Z * [new branch] gh/guangyey/202/head -> origin/gh/guangyey/202/head 2025-09-07T06:42:00.7807749Z * [new branch] gh/guangyey/202/orig -> origin/gh/guangyey/202/orig 2025-09-07T06:42:00.7807894Z * [new branch] gh/guangyey/203/base -> origin/gh/guangyey/203/base 2025-09-07T06:42:00.7808200Z * [new branch] gh/guangyey/203/head -> origin/gh/guangyey/203/head 2025-09-07T06:42:00.7808352Z * [new branch] gh/guangyey/203/orig -> origin/gh/guangyey/203/orig 2025-09-07T06:42:00.7808510Z * [new branch] gh/guangyey/204/base -> origin/gh/guangyey/204/base 2025-09-07T06:42:00.7808650Z * [new branch] gh/guangyey/204/head -> origin/gh/guangyey/204/head 2025-09-07T06:42:00.7808786Z * [new branch] gh/guangyey/204/orig -> origin/gh/guangyey/204/orig 2025-09-07T06:42:00.7808939Z * [new branch] gh/guangyey/205/base -> origin/gh/guangyey/205/base 2025-09-07T06:42:00.7809078Z * [new branch] gh/guangyey/205/head -> origin/gh/guangyey/205/head 2025-09-07T06:42:00.7809224Z * [new branch] gh/guangyey/205/orig -> origin/gh/guangyey/205/orig 2025-09-07T06:42:00.7809361Z * [new branch] gh/guangyey/206/base -> origin/gh/guangyey/206/base 2025-09-07T06:42:00.7809511Z * [new branch] gh/guangyey/206/head -> origin/gh/guangyey/206/head 2025-09-07T06:42:00.7809654Z * [new branch] gh/guangyey/206/orig -> origin/gh/guangyey/206/orig 2025-09-07T06:42:00.7809785Z * [new branch] gh/guangyey/207/base -> origin/gh/guangyey/207/base 2025-09-07T06:42:00.7809922Z * [new branch] gh/guangyey/207/head -> origin/gh/guangyey/207/head 2025-09-07T06:42:00.7810250Z * [new branch] gh/guangyey/207/orig -> origin/gh/guangyey/207/orig 2025-09-07T06:42:00.7810424Z * [new branch] gh/guangyey/79/base -> origin/gh/guangyey/79/base 2025-09-07T06:42:00.7810559Z * [new branch] gh/guangyey/79/head -> origin/gh/guangyey/79/head 2025-09-07T06:42:00.7810825Z * [new branch] gh/guangyey/79/orig -> origin/gh/guangyey/79/orig 2025-09-07T06:42:00.7811024Z * [new branch] gh/guangyey/89/base -> origin/gh/guangyey/89/base 2025-09-07T06:42:00.7811161Z * [new branch] gh/guangyey/89/head -> origin/gh/guangyey/89/head 2025-09-07T06:42:00.7811414Z * [new branch] gh/guangyey/89/orig -> origin/gh/guangyey/89/orig 2025-09-07T06:42:00.7817476Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-09-07T06:42:00.7817690Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-09-07T06:42:00.7817908Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-09-07T06:42:00.7818071Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-09-07T06:42:00.7818229Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-09-07T06:42:00.7818397Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-09-07T06:42:00.7818581Z * [new branch] gh/guilhermeleobas/124/base -> origin/gh/guilhermeleobas/124/base 2025-09-07T06:42:00.7818782Z * [new branch] gh/guilhermeleobas/124/head -> origin/gh/guilhermeleobas/124/head 2025-09-07T06:42:00.7819124Z * [new branch] gh/guilhermeleobas/124/orig -> origin/gh/guilhermeleobas/124/orig 2025-09-07T06:42:00.7819310Z * [new branch] gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base 2025-09-07T06:42:00.7819475Z * [new branch] gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-09-07T06:42:00.7819805Z * [new branch] gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig 2025-09-07T06:42:00.7819973Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-09-07T06:42:00.7820144Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-09-07T06:42:00.7820438Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-09-07T06:42:00.7820834Z * [new branch] gh/guilhermeleobas/163/base -> origin/gh/guilhermeleobas/163/base 2025-09-07T06:42:00.7823168Z * [new branch] gh/guilhermeleobas/163/head -> origin/gh/guilhermeleobas/163/head 2025-09-07T06:42:00.7823406Z * [new branch] gh/guilhermeleobas/163/orig -> origin/gh/guilhermeleobas/163/orig 2025-09-07T06:42:00.7823713Z * [new branch] gh/guilhermeleobas/164/base -> origin/gh/guilhermeleobas/164/base 2025-09-07T06:42:00.7824076Z * [new branch] gh/guilhermeleobas/164/head -> origin/gh/guilhermeleobas/164/head 2025-09-07T06:42:00.7824417Z * [new branch] gh/guilhermeleobas/164/orig -> origin/gh/guilhermeleobas/164/orig 2025-09-07T06:42:00.7825917Z * [new branch] gh/guilhermeleobas/165/base -> origin/gh/guilhermeleobas/165/base 2025-09-07T06:42:00.7826171Z * [new branch] gh/guilhermeleobas/165/head -> origin/gh/guilhermeleobas/165/head 2025-09-07T06:42:00.7832227Z * [new branch] gh/guilhermeleobas/165/orig -> origin/gh/guilhermeleobas/165/orig 2025-09-07T06:42:00.7832469Z * [new branch] gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base 2025-09-07T06:42:00.7832660Z * [new branch] gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head 2025-09-07T06:42:00.7832835Z * [new branch] gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig 2025-09-07T06:42:00.7833268Z * [new branch] gh/guilhermeleobas/167/base -> origin/gh/guilhermeleobas/167/base 2025-09-07T06:42:00.7833458Z * [new branch] gh/guilhermeleobas/167/head -> origin/gh/guilhermeleobas/167/head 2025-09-07T06:42:00.7833635Z * [new branch] gh/guilhermeleobas/167/orig -> origin/gh/guilhermeleobas/167/orig 2025-09-07T06:42:00.7833811Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-09-07T06:42:00.7833995Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-09-07T06:42:00.7841883Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-09-07T06:42:00.7842117Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-09-07T06:42:00.7842295Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-09-07T06:42:00.7842497Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-09-07T06:42:00.7842670Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-09-07T06:42:00.7842827Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-09-07T06:42:00.7842991Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-09-07T06:42:00.7843174Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-09-07T06:42:00.7843338Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-09-07T06:42:00.7843492Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-09-07T06:42:00.7843646Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-09-07T06:42:00.7843816Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-09-07T06:42:00.7847957Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-09-07T06:42:00.7848312Z * [new branch] gh/guilhermeleobas/192/base -> origin/gh/guilhermeleobas/192/base 2025-09-07T06:42:00.7848566Z * [new branch] gh/guilhermeleobas/192/head -> origin/gh/guilhermeleobas/192/head 2025-09-07T06:42:00.7849080Z * [new branch] gh/guilhermeleobas/192/orig -> origin/gh/guilhermeleobas/192/orig 2025-09-07T06:42:00.7849336Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-09-07T06:42:00.7850005Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-09-07T06:42:00.7850353Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-09-07T06:42:00.7850535Z * [new branch] gh/guilhermeleobas/194/base -> origin/gh/guilhermeleobas/194/base 2025-09-07T06:42:00.7850860Z * [new branch] gh/guilhermeleobas/194/head -> origin/gh/guilhermeleobas/194/head 2025-09-07T06:42:00.7851429Z * [new branch] gh/guilhermeleobas/194/orig -> origin/gh/guilhermeleobas/194/orig 2025-09-07T06:42:00.7851624Z * [new branch] gh/guilhermeleobas/203/base -> origin/gh/guilhermeleobas/203/base 2025-09-07T06:42:00.7851810Z * [new branch] gh/guilhermeleobas/203/head -> origin/gh/guilhermeleobas/203/head 2025-09-07T06:42:00.7851969Z * [new branch] gh/guilhermeleobas/203/orig -> origin/gh/guilhermeleobas/203/orig 2025-09-07T06:42:00.7852139Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-09-07T06:42:00.7854351Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-09-07T06:42:00.7854555Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-09-07T06:42:00.7854860Z * [new branch] gh/guilhermeleobas/205/base -> origin/gh/guilhermeleobas/205/base 2025-09-07T06:42:00.7855034Z * [new branch] gh/guilhermeleobas/205/head -> origin/gh/guilhermeleobas/205/head 2025-09-07T06:42:00.7855193Z * [new branch] gh/guilhermeleobas/205/orig -> origin/gh/guilhermeleobas/205/orig 2025-09-07T06:42:00.7855362Z * [new branch] gh/guilhermeleobas/209/base -> origin/gh/guilhermeleobas/209/base 2025-09-07T06:42:00.7858865Z * [new branch] gh/guilhermeleobas/209/head -> origin/gh/guilhermeleobas/209/head 2025-09-07T06:42:00.7859224Z * [new branch] gh/guilhermeleobas/209/orig -> origin/gh/guilhermeleobas/209/orig 2025-09-07T06:42:00.7859497Z * [new branch] gh/guilhermeleobas/210/base -> origin/gh/guilhermeleobas/210/base 2025-09-07T06:42:00.7859722Z * [new branch] gh/guilhermeleobas/210/head -> origin/gh/guilhermeleobas/210/head 2025-09-07T06:42:00.7859997Z * [new branch] gh/guilhermeleobas/210/orig -> origin/gh/guilhermeleobas/210/orig 2025-09-07T06:42:00.7860190Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-09-07T06:42:00.7860855Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-09-07T06:42:00.7864038Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-09-07T06:42:00.7864260Z * [new branch] gh/guilhermeleobas/214/base -> origin/gh/guilhermeleobas/214/base 2025-09-07T06:42:00.7864462Z * [new branch] gh/guilhermeleobas/214/head -> origin/gh/guilhermeleobas/214/head 2025-09-07T06:42:00.7864646Z * [new branch] gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig 2025-09-07T06:42:00.7864829Z * [new branch] gh/guilhermeleobas/215/base -> origin/gh/guilhermeleobas/215/base 2025-09-07T06:42:00.7865042Z * [new branch] gh/guilhermeleobas/215/head -> origin/gh/guilhermeleobas/215/head 2025-09-07T06:42:00.7865227Z * [new branch] gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig 2025-09-07T06:42:00.7865570Z * [new branch] gh/guilhermeleobas/216/base -> origin/gh/guilhermeleobas/216/base 2025-09-07T06:42:00.7867314Z * [new branch] gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head 2025-09-07T06:42:00.7867645Z * [new branch] gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig 2025-09-07T06:42:00.7873611Z * [new branch] gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base 2025-09-07T06:42:00.7873811Z * [new branch] gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head 2025-09-07T06:42:00.7874010Z * [new branch] gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig 2025-09-07T06:42:00.7878179Z * [new branch] gh/guilhermeleobas/219/base -> origin/gh/guilhermeleobas/219/base 2025-09-07T06:42:00.7878386Z * [new branch] gh/guilhermeleobas/219/head -> origin/gh/guilhermeleobas/219/head 2025-09-07T06:42:00.7878563Z * [new branch] gh/guilhermeleobas/219/orig -> origin/gh/guilhermeleobas/219/orig 2025-09-07T06:42:00.7878755Z * [new branch] gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base 2025-09-07T06:42:00.7878934Z * [new branch] gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head 2025-09-07T06:42:00.7879115Z * [new branch] gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig 2025-09-07T06:42:00.7879287Z * [new branch] gh/guilhermeleobas/221/base -> origin/gh/guilhermeleobas/221/base 2025-09-07T06:42:00.7879467Z * [new branch] gh/guilhermeleobas/221/head -> origin/gh/guilhermeleobas/221/head 2025-09-07T06:42:00.7879645Z * [new branch] gh/guilhermeleobas/221/orig -> origin/gh/guilhermeleobas/221/orig 2025-09-07T06:42:00.7879990Z * [new branch] gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base 2025-09-07T06:42:00.7880184Z * [new branch] gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head 2025-09-07T06:42:00.7883636Z * [new branch] gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig 2025-09-07T06:42:00.7883930Z * [new branch] gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base 2025-09-07T06:42:00.7884126Z * [new branch] gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head 2025-09-07T06:42:00.7884291Z * [new branch] gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig 2025-09-07T06:42:00.7884461Z * [new branch] gh/guilhermeleobas/224/base -> origin/gh/guilhermeleobas/224/base 2025-09-07T06:42:00.7889408Z * [new branch] gh/guilhermeleobas/224/head -> origin/gh/guilhermeleobas/224/head 2025-09-07T06:42:00.7894277Z * [new branch] gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig 2025-09-07T06:42:00.7898536Z * [new branch] gh/guilhermeleobas/225/base -> origin/gh/guilhermeleobas/225/base 2025-09-07T06:42:00.7900457Z * [new branch] gh/guilhermeleobas/225/head -> origin/gh/guilhermeleobas/225/head 2025-09-07T06:42:00.7900655Z * [new branch] gh/guilhermeleobas/225/orig -> origin/gh/guilhermeleobas/225/orig 2025-09-07T06:42:00.7900865Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-09-07T06:42:00.7901060Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-09-07T06:42:00.7901235Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-09-07T06:42:00.7901414Z * [new branch] gh/guilhermeleobas/227/base -> origin/gh/guilhermeleobas/227/base 2025-09-07T06:42:00.7901610Z * [new branch] gh/guilhermeleobas/227/head -> origin/gh/guilhermeleobas/227/head 2025-09-07T06:42:00.7901788Z * [new branch] gh/guilhermeleobas/227/orig -> origin/gh/guilhermeleobas/227/orig 2025-09-07T06:42:00.7901965Z * [new branch] gh/guilhermeleobas/228/base -> origin/gh/guilhermeleobas/228/base 2025-09-07T06:42:00.7902149Z * [new branch] gh/guilhermeleobas/228/head -> origin/gh/guilhermeleobas/228/head 2025-09-07T06:42:00.7902465Z * [new branch] gh/guilhermeleobas/228/orig -> origin/gh/guilhermeleobas/228/orig 2025-09-07T06:42:00.7902642Z * [new branch] gh/guilhermeleobas/229/base -> origin/gh/guilhermeleobas/229/base 2025-09-07T06:42:00.7902820Z * [new branch] gh/guilhermeleobas/229/head -> origin/gh/guilhermeleobas/229/head 2025-09-07T06:42:00.7902995Z * [new branch] gh/guilhermeleobas/229/orig -> origin/gh/guilhermeleobas/229/orig 2025-09-07T06:42:00.7903171Z * [new branch] gh/guilhermeleobas/230/base -> origin/gh/guilhermeleobas/230/base 2025-09-07T06:42:00.7903349Z * [new branch] gh/guilhermeleobas/230/head -> origin/gh/guilhermeleobas/230/head 2025-09-07T06:42:00.7903533Z * [new branch] gh/guilhermeleobas/230/orig -> origin/gh/guilhermeleobas/230/orig 2025-09-07T06:42:00.7903707Z * [new branch] gh/guilhermeleobas/231/base -> origin/gh/guilhermeleobas/231/base 2025-09-07T06:42:00.7903889Z * [new branch] gh/guilhermeleobas/231/head -> origin/gh/guilhermeleobas/231/head 2025-09-07T06:42:00.7904068Z * [new branch] gh/guilhermeleobas/231/orig -> origin/gh/guilhermeleobas/231/orig 2025-09-07T06:42:00.7904258Z * [new branch] gh/guilhermeleobas/232/base -> origin/gh/guilhermeleobas/232/base 2025-09-07T06:42:00.7904437Z * [new branch] gh/guilhermeleobas/232/head -> origin/gh/guilhermeleobas/232/head 2025-09-07T06:42:00.7904611Z * [new branch] gh/guilhermeleobas/232/orig -> origin/gh/guilhermeleobas/232/orig 2025-09-07T06:42:00.7904856Z * [new branch] gh/guilhermeleobas/233/base -> origin/gh/guilhermeleobas/233/base 2025-09-07T06:42:00.7905176Z * [new branch] gh/guilhermeleobas/233/head -> origin/gh/guilhermeleobas/233/head 2025-09-07T06:42:00.7905492Z * [new branch] gh/guilhermeleobas/233/orig -> origin/gh/guilhermeleobas/233/orig 2025-09-07T06:42:00.7911723Z * [new branch] gh/guilhermeleobas/234/base -> origin/gh/guilhermeleobas/234/base 2025-09-07T06:42:00.7913850Z * [new branch] gh/guilhermeleobas/234/head -> origin/gh/guilhermeleobas/234/head 2025-09-07T06:42:00.7914172Z * [new branch] gh/guilhermeleobas/234/orig -> origin/gh/guilhermeleobas/234/orig 2025-09-07T06:42:00.7919689Z * [new branch] gh/guilhermeleobas/235/base -> origin/gh/guilhermeleobas/235/base 2025-09-07T06:42:00.7924513Z * [new branch] gh/guilhermeleobas/235/head -> origin/gh/guilhermeleobas/235/head 2025-09-07T06:42:00.7930140Z * [new branch] gh/guilhermeleobas/235/orig -> origin/gh/guilhermeleobas/235/orig 2025-09-07T06:42:00.7931276Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-09-07T06:42:00.7931637Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-09-07T06:42:00.7931838Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-09-07T06:42:00.7932033Z * [new branch] gh/guilhermeleobas/237/base -> origin/gh/guilhermeleobas/237/base 2025-09-07T06:42:00.7932200Z * [new branch] gh/guilhermeleobas/237/head -> origin/gh/guilhermeleobas/237/head 2025-09-07T06:42:00.7932364Z * [new branch] gh/guilhermeleobas/237/orig -> origin/gh/guilhermeleobas/237/orig 2025-09-07T06:42:00.7932541Z * [new branch] gh/guilhermeleobas/238/base -> origin/gh/guilhermeleobas/238/base 2025-09-07T06:42:00.7932706Z * [new branch] gh/guilhermeleobas/238/head -> origin/gh/guilhermeleobas/238/head 2025-09-07T06:42:00.7932871Z * [new branch] gh/guilhermeleobas/238/orig -> origin/gh/guilhermeleobas/238/orig 2025-09-07T06:42:00.7933033Z * [new branch] gh/guilhermeleobas/239/base -> origin/gh/guilhermeleobas/239/base 2025-09-07T06:42:00.7933201Z * [new branch] gh/guilhermeleobas/239/head -> origin/gh/guilhermeleobas/239/head 2025-09-07T06:42:00.7933564Z * [new branch] gh/guilhermeleobas/239/orig -> origin/gh/guilhermeleobas/239/orig 2025-09-07T06:42:00.7933733Z * [new branch] gh/guilhermeleobas/240/base -> origin/gh/guilhermeleobas/240/base 2025-09-07T06:42:00.7933907Z * [new branch] gh/guilhermeleobas/240/head -> origin/gh/guilhermeleobas/240/head 2025-09-07T06:42:00.7934074Z * [new branch] gh/guilhermeleobas/240/orig -> origin/gh/guilhermeleobas/240/orig 2025-09-07T06:42:00.7934248Z * [new branch] gh/guilhermeleobas/241/base -> origin/gh/guilhermeleobas/241/base 2025-09-07T06:42:00.7934418Z * [new branch] gh/guilhermeleobas/241/head -> origin/gh/guilhermeleobas/241/head 2025-09-07T06:42:00.7934593Z * [new branch] gh/guilhermeleobas/241/orig -> origin/gh/guilhermeleobas/241/orig 2025-09-07T06:42:00.7934764Z * [new branch] gh/guilhermeleobas/242/base -> origin/gh/guilhermeleobas/242/base 2025-09-07T06:42:00.7934935Z * [new branch] gh/guilhermeleobas/242/head -> origin/gh/guilhermeleobas/242/head 2025-09-07T06:42:00.7935112Z * [new branch] gh/guilhermeleobas/242/orig -> origin/gh/guilhermeleobas/242/orig 2025-09-07T06:42:00.7935278Z * [new branch] gh/guilhermeleobas/243/base -> origin/gh/guilhermeleobas/243/base 2025-09-07T06:42:00.7935449Z * [new branch] gh/guilhermeleobas/243/head -> origin/gh/guilhermeleobas/243/head 2025-09-07T06:42:00.7935614Z * [new branch] gh/guilhermeleobas/243/orig -> origin/gh/guilhermeleobas/243/orig 2025-09-07T06:42:00.7935850Z * [new branch] gh/guilhermeleobas/244/base -> origin/gh/guilhermeleobas/244/base 2025-09-07T06:42:00.7936023Z * [new branch] gh/guilhermeleobas/244/head -> origin/gh/guilhermeleobas/244/head 2025-09-07T06:42:00.7936188Z * [new branch] gh/guilhermeleobas/244/orig -> origin/gh/guilhermeleobas/244/orig 2025-09-07T06:42:00.7936358Z * [new branch] gh/guilhermeleobas/245/base -> origin/gh/guilhermeleobas/245/base 2025-09-07T06:42:00.7936531Z * [new branch] gh/guilhermeleobas/245/head -> origin/gh/guilhermeleobas/245/head 2025-09-07T06:42:00.7936703Z * [new branch] gh/guilhermeleobas/245/orig -> origin/gh/guilhermeleobas/245/orig 2025-09-07T06:42:00.7936874Z * [new branch] gh/guilhermeleobas/73/base -> origin/gh/guilhermeleobas/73/base 2025-09-07T06:42:00.7937039Z * [new branch] gh/guilhermeleobas/73/head -> origin/gh/guilhermeleobas/73/head 2025-09-07T06:42:00.7937213Z * [new branch] gh/guilhermeleobas/73/orig -> origin/gh/guilhermeleobas/73/orig 2025-09-07T06:42:00.7937383Z * [new branch] gh/henrylhtsang/140/base -> origin/gh/henrylhtsang/140/base 2025-09-07T06:42:00.7937555Z * [new branch] gh/henrylhtsang/140/head -> origin/gh/henrylhtsang/140/head 2025-09-07T06:42:00.7937868Z * [new branch] gh/henrylhtsang/140/orig -> origin/gh/henrylhtsang/140/orig 2025-09-07T06:42:00.7941325Z * [new branch] gh/henrylhtsang/141/base -> origin/gh/henrylhtsang/141/base 2025-09-07T06:42:00.7941541Z * [new branch] gh/henrylhtsang/141/head -> origin/gh/henrylhtsang/141/head 2025-09-07T06:42:00.7941724Z * [new branch] gh/henrylhtsang/141/orig -> origin/gh/henrylhtsang/141/orig 2025-09-07T06:42:00.7941899Z * [new branch] gh/henrylhtsang/142/base -> origin/gh/henrylhtsang/142/base 2025-09-07T06:42:00.7943106Z * [new branch] gh/henrylhtsang/142/head -> origin/gh/henrylhtsang/142/head 2025-09-07T06:42:00.7943390Z * [new branch] gh/henrylhtsang/142/orig -> origin/gh/henrylhtsang/142/orig 2025-09-07T06:42:00.7943709Z * [new branch] gh/henrylhtsang/143/base -> origin/gh/henrylhtsang/143/base 2025-09-07T06:42:00.7944716Z * [new branch] gh/henrylhtsang/143/head -> origin/gh/henrylhtsang/143/head 2025-09-07T06:42:00.7945191Z * [new branch] gh/henrylhtsang/143/orig -> origin/gh/henrylhtsang/143/orig 2025-09-07T06:42:00.7950662Z * [new branch] gh/henrylhtsang/144/base -> origin/gh/henrylhtsang/144/base 2025-09-07T06:42:00.7950959Z * [new branch] gh/henrylhtsang/144/head -> origin/gh/henrylhtsang/144/head 2025-09-07T06:42:00.7951224Z * [new branch] gh/henrylhtsang/144/orig -> origin/gh/henrylhtsang/144/orig 2025-09-07T06:42:00.7951486Z * [new branch] gh/henrylhtsang/145/base -> origin/gh/henrylhtsang/145/base 2025-09-07T06:42:00.7951754Z * [new branch] gh/henrylhtsang/145/head -> origin/gh/henrylhtsang/145/head 2025-09-07T06:42:00.7952468Z * [new branch] gh/henrylhtsang/145/orig -> origin/gh/henrylhtsang/145/orig 2025-09-07T06:42:00.7957540Z * [new branch] gh/henrylhtsang/146/base -> origin/gh/henrylhtsang/146/base 2025-09-07T06:42:00.7962596Z * [new branch] gh/henrylhtsang/146/head -> origin/gh/henrylhtsang/146/head 2025-09-07T06:42:00.7967315Z * [new branch] gh/henrylhtsang/146/orig -> origin/gh/henrylhtsang/146/orig 2025-09-07T06:42:00.7969736Z * [new branch] gh/henrylhtsang/147/base -> origin/gh/henrylhtsang/147/base 2025-09-07T06:42:00.7969959Z * [new branch] gh/henrylhtsang/147/head -> origin/gh/henrylhtsang/147/head 2025-09-07T06:42:00.7970130Z * [new branch] gh/henrylhtsang/147/orig -> origin/gh/henrylhtsang/147/orig 2025-09-07T06:42:00.7970370Z * [new branch] gh/henrylhtsang/148/base -> origin/gh/henrylhtsang/148/base 2025-09-07T06:42:00.7976977Z * [new branch] gh/henrylhtsang/148/head -> origin/gh/henrylhtsang/148/head 2025-09-07T06:42:00.7979085Z * [new branch] gh/henrylhtsang/148/orig -> origin/gh/henrylhtsang/148/orig 2025-09-07T06:42:00.7979425Z * [new branch] gh/henrylhtsang/149/base -> origin/gh/henrylhtsang/149/base 2025-09-07T06:42:00.7979707Z * [new branch] gh/henrylhtsang/149/head -> origin/gh/henrylhtsang/149/head 2025-09-07T06:42:00.7979905Z * [new branch] gh/henrylhtsang/149/orig -> origin/gh/henrylhtsang/149/orig 2025-09-07T06:42:00.7980168Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-09-07T06:42:00.7980530Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-09-07T06:42:00.7980686Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-09-07T06:42:00.7980953Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-09-07T06:42:00.7981172Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-09-07T06:42:00.7981333Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-09-07T06:42:00.7981477Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-09-07T06:42:00.7981620Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-09-07T06:42:00.7981770Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-09-07T06:42:00.7981919Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-09-07T06:42:00.7982056Z * [new branch] gh/isuruf/141/base -> origin/gh/isuruf/141/base 2025-09-07T06:42:00.7982193Z * [new branch] gh/isuruf/141/head -> origin/gh/isuruf/141/head 2025-09-07T06:42:00.7982354Z * [new branch] gh/isuruf/141/orig -> origin/gh/isuruf/141/orig 2025-09-07T06:42:00.7982492Z * [new branch] gh/isuruf/142/base -> origin/gh/isuruf/142/base 2025-09-07T06:42:00.7982633Z * [new branch] gh/isuruf/142/head -> origin/gh/isuruf/142/head 2025-09-07T06:42:00.7982771Z * [new branch] gh/isuruf/142/orig -> origin/gh/isuruf/142/orig 2025-09-07T06:42:00.7983054Z * [new branch] gh/isuruf/143/base -> origin/gh/isuruf/143/base 2025-09-07T06:42:00.7983191Z * [new branch] gh/isuruf/143/head -> origin/gh/isuruf/143/head 2025-09-07T06:42:00.7983331Z * [new branch] gh/isuruf/143/orig -> origin/gh/isuruf/143/orig 2025-09-07T06:42:00.7983480Z * [new branch] gh/isuruf/144/base -> origin/gh/isuruf/144/base 2025-09-07T06:42:00.7983615Z * [new branch] gh/isuruf/144/head -> origin/gh/isuruf/144/head 2025-09-07T06:42:00.7983763Z * [new branch] gh/isuruf/144/orig -> origin/gh/isuruf/144/orig 2025-09-07T06:42:00.7983899Z * [new branch] gh/isuruf/145/base -> origin/gh/isuruf/145/base 2025-09-07T06:42:00.7984035Z * [new branch] gh/isuruf/145/head -> origin/gh/isuruf/145/head 2025-09-07T06:42:00.7984182Z * [new branch] gh/isuruf/145/orig -> origin/gh/isuruf/145/orig 2025-09-07T06:42:00.7984324Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-09-07T06:42:00.7984469Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-09-07T06:42:00.7984605Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-09-07T06:42:00.7984754Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-09-07T06:42:00.7984901Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-09-07T06:42:00.7985078Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-09-07T06:42:00.7985241Z * [new branch] gh/jamesjwu/150/base -> origin/gh/jamesjwu/150/base 2025-09-07T06:42:00.7985822Z * [new branch] gh/jamesjwu/150/head -> origin/gh/jamesjwu/150/head 2025-09-07T06:42:00.7987111Z * [new branch] gh/jamesjwu/150/orig -> origin/gh/jamesjwu/150/orig 2025-09-07T06:42:00.7987467Z * [new branch] gh/jamesjwu/154/base -> origin/gh/jamesjwu/154/base 2025-09-07T06:42:00.7988528Z * [new branch] gh/jamesjwu/154/head -> origin/gh/jamesjwu/154/head 2025-09-07T06:42:00.7989024Z * [new branch] gh/jamesjwu/154/orig -> origin/gh/jamesjwu/154/orig 2025-09-07T06:42:00.7990051Z * [new branch] gh/jamesjwu/155/base -> origin/gh/jamesjwu/155/base 2025-09-07T06:42:00.7990329Z * [new branch] gh/jamesjwu/155/head -> origin/gh/jamesjwu/155/head 2025-09-07T06:42:00.7994263Z * [new branch] gh/jamesjwu/155/orig -> origin/gh/jamesjwu/155/orig 2025-09-07T06:42:00.7994819Z * [new branch] gh/jamesjwu/159/base -> origin/gh/jamesjwu/159/base 2025-09-07T06:42:00.7994993Z * [new branch] gh/jamesjwu/159/head -> origin/gh/jamesjwu/159/head 2025-09-07T06:42:00.7995151Z * [new branch] gh/jamesjwu/159/orig -> origin/gh/jamesjwu/159/orig 2025-09-07T06:42:00.7995330Z * [new branch] gh/jamesjwu/163/base -> origin/gh/jamesjwu/163/base 2025-09-07T06:42:00.7995490Z * [new branch] gh/jamesjwu/163/head -> origin/gh/jamesjwu/163/head 2025-09-07T06:42:00.8001083Z * [new branch] gh/jamesjwu/163/orig -> origin/gh/jamesjwu/163/orig 2025-09-07T06:42:00.8005942Z * [new branch] gh/jamesjwu/171/base -> origin/gh/jamesjwu/171/base 2025-09-07T06:42:00.8006117Z * [new branch] gh/jamesjwu/171/head -> origin/gh/jamesjwu/171/head 2025-09-07T06:42:00.8006287Z * [new branch] gh/jamesjwu/171/orig -> origin/gh/jamesjwu/171/orig 2025-09-07T06:42:00.8006427Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-09-07T06:42:00.8006565Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-09-07T06:42:00.8006720Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-09-07T06:42:00.8007033Z * [new branch] gh/jamesjwu/181/base -> origin/gh/jamesjwu/181/base 2025-09-07T06:42:00.8007178Z * [new branch] gh/jamesjwu/181/head -> origin/gh/jamesjwu/181/head 2025-09-07T06:42:00.8007313Z * [new branch] gh/jamesjwu/181/orig -> origin/gh/jamesjwu/181/orig 2025-09-07T06:42:00.8007454Z * [new branch] gh/jamesjwu/182/base -> origin/gh/jamesjwu/182/base 2025-09-07T06:42:00.8007601Z * [new branch] gh/jamesjwu/182/head -> origin/gh/jamesjwu/182/head 2025-09-07T06:42:00.8007736Z * [new branch] gh/jamesjwu/182/orig -> origin/gh/jamesjwu/182/orig 2025-09-07T06:42:00.8007881Z * [new branch] gh/jamesjwu/183/base -> origin/gh/jamesjwu/183/base 2025-09-07T06:42:00.8008016Z * [new branch] gh/jamesjwu/183/head -> origin/gh/jamesjwu/183/head 2025-09-07T06:42:00.8008168Z * [new branch] gh/jamesjwu/183/orig -> origin/gh/jamesjwu/183/orig 2025-09-07T06:42:00.8008307Z * [new branch] gh/jamesjwu/184/base -> origin/gh/jamesjwu/184/base 2025-09-07T06:42:00.8008498Z * [new branch] gh/jamesjwu/184/head -> origin/gh/jamesjwu/184/head 2025-09-07T06:42:00.8009554Z * [new branch] gh/jamesjwu/184/orig -> origin/gh/jamesjwu/184/orig 2025-09-07T06:42:00.8010647Z * [new branch] gh/jamesjwu/185/base -> origin/gh/jamesjwu/185/base 2025-09-07T06:42:00.8011093Z * [new branch] gh/jamesjwu/185/head -> origin/gh/jamesjwu/185/head 2025-09-07T06:42:00.8013979Z * [new branch] gh/jamesjwu/185/orig -> origin/gh/jamesjwu/185/orig 2025-09-07T06:42:00.8014166Z * [new branch] gh/jamesjwu/186/base -> origin/gh/jamesjwu/186/base 2025-09-07T06:42:00.8014304Z * [new branch] gh/jamesjwu/186/head -> origin/gh/jamesjwu/186/head 2025-09-07T06:42:00.8014462Z * [new branch] gh/jamesjwu/186/orig -> origin/gh/jamesjwu/186/orig 2025-09-07T06:42:00.8016460Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-09-07T06:42:00.8016796Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-09-07T06:42:00.8016975Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-09-07T06:42:00.8017496Z * [new branch] gh/jamesjwu/188/base -> origin/gh/jamesjwu/188/base 2025-09-07T06:42:00.8018974Z * [new branch] gh/jamesjwu/188/head -> origin/gh/jamesjwu/188/head 2025-09-07T06:42:00.8019162Z * [new branch] gh/jamesjwu/188/orig -> origin/gh/jamesjwu/188/orig 2025-09-07T06:42:00.8019907Z * [new branch] gh/jamesjwu/189/base -> origin/gh/jamesjwu/189/base 2025-09-07T06:42:00.8020835Z * [new branch] gh/jamesjwu/189/head -> origin/gh/jamesjwu/189/head 2025-09-07T06:42:00.8021395Z * [new branch] gh/jamesjwu/189/orig -> origin/gh/jamesjwu/189/orig 2025-09-07T06:42:00.8022964Z * [new branch] gh/jamesjwu/190/base -> origin/gh/jamesjwu/190/base 2025-09-07T06:42:00.8023228Z * [new branch] gh/jamesjwu/190/head -> origin/gh/jamesjwu/190/head 2025-09-07T06:42:00.8024387Z * [new branch] gh/jamesjwu/190/orig -> origin/gh/jamesjwu/190/orig 2025-09-07T06:42:00.8025403Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-09-07T06:42:00.8029610Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-09-07T06:42:00.8029962Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-09-07T06:42:00.8030131Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-09-07T06:42:00.8030282Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-09-07T06:42:00.8030788Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-09-07T06:42:00.8031310Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-09-07T06:42:00.8031511Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-09-07T06:42:00.8035643Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-09-07T06:42:00.8035848Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-09-07T06:42:00.8036404Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-09-07T06:42:00.8036585Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-09-07T06:42:00.8036732Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-09-07T06:42:00.8036950Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-09-07T06:42:00.8037100Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-09-07T06:42:00.8037258Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-09-07T06:42:00.8037584Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-09-07T06:42:00.8037928Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-09-07T06:42:00.8042128Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-09-07T06:42:00.8042271Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-09-07T06:42:00.8042782Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-09-07T06:42:00.8042966Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-09-07T06:42:00.8043156Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-09-07T06:42:00.8043319Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-09-07T06:42:00.8046900Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-09-07T06:42:00.8047194Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-09-07T06:42:00.8047389Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-09-07T06:42:00.8047560Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-09-07T06:42:00.8047703Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-09-07T06:42:00.8047966Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-09-07T06:42:00.8051417Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-09-07T06:42:00.8051641Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-09-07T06:42:00.8051798Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-09-07T06:42:00.8051959Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-09-07T06:42:00.8052112Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-09-07T06:42:00.8052273Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-09-07T06:42:00.8052451Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-09-07T06:42:00.8055402Z * [new branch] gh/janeyx99/296/base -> origin/gh/janeyx99/296/base 2025-09-07T06:42:00.8055668Z * [new branch] gh/janeyx99/296/head -> origin/gh/janeyx99/296/head 2025-09-07T06:42:00.8055822Z * [new branch] gh/janeyx99/296/orig -> origin/gh/janeyx99/296/orig 2025-09-07T06:42:00.8056038Z * [new branch] gh/janeyx99/297/base -> origin/gh/janeyx99/297/base 2025-09-07T06:42:00.8056172Z * [new branch] gh/janeyx99/297/head -> origin/gh/janeyx99/297/head 2025-09-07T06:42:00.8056312Z * [new branch] gh/janeyx99/297/orig -> origin/gh/janeyx99/297/orig 2025-09-07T06:42:00.8059295Z * [new branch] gh/janeyx99/298/base -> origin/gh/janeyx99/298/base 2025-09-07T06:42:00.8059633Z * [new branch] gh/janeyx99/298/head -> origin/gh/janeyx99/298/head 2025-09-07T06:42:00.8059810Z * [new branch] gh/janeyx99/298/orig -> origin/gh/janeyx99/298/orig 2025-09-07T06:42:00.8059966Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-09-07T06:42:00.8060115Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-09-07T06:42:00.8063269Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-09-07T06:42:00.8063443Z * [new branch] gh/janeyx99/300/base -> origin/gh/janeyx99/300/base 2025-09-07T06:42:00.8063602Z * [new branch] gh/janeyx99/300/head -> origin/gh/janeyx99/300/head 2025-09-07T06:42:00.8063752Z * [new branch] gh/janeyx99/300/orig -> origin/gh/janeyx99/300/orig 2025-09-07T06:42:00.8063911Z * [new branch] gh/janeyx99/301/base -> origin/gh/janeyx99/301/base 2025-09-07T06:42:00.8064216Z * [new branch] gh/janeyx99/301/head -> origin/gh/janeyx99/301/head 2025-09-07T06:42:00.8064460Z * [new branch] gh/janeyx99/301/orig -> origin/gh/janeyx99/301/orig 2025-09-07T06:42:00.8065490Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-09-07T06:42:00.8065770Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-09-07T06:42:00.8072954Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-09-07T06:42:00.8073143Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-09-07T06:42:00.8073290Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-09-07T06:42:00.8073429Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-09-07T06:42:00.8073580Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-09-07T06:42:00.8073756Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-09-07T06:42:00.8079050Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-09-07T06:42:00.8083335Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-09-07T06:42:00.8085597Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-09-07T06:42:00.8085773Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-09-07T06:42:00.8086407Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-09-07T06:42:00.8086569Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-09-07T06:42:00.8086717Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-09-07T06:42:00.8087585Z * [new branch] gh/jansel/531/base -> origin/gh/jansel/531/base 2025-09-07T06:42:00.8087913Z * [new branch] gh/jansel/531/head -> origin/gh/jansel/531/head 2025-09-07T06:42:00.8088084Z * [new branch] gh/jansel/531/orig -> origin/gh/jansel/531/orig 2025-09-07T06:42:00.8088268Z * [new branch] gh/jbschlosser/208/head -> origin/gh/jbschlosser/208/head 2025-09-07T06:42:00.8088443Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-09-07T06:42:00.8088744Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-09-07T06:42:00.8088910Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-09-07T06:42:00.8089073Z * [new branch] gh/jbschlosser/248/base -> origin/gh/jbschlosser/248/base 2025-09-07T06:42:00.8089243Z * [new branch] gh/jbschlosser/248/head -> origin/gh/jbschlosser/248/head 2025-09-07T06:42:00.8089405Z * [new branch] gh/jbschlosser/248/orig -> origin/gh/jbschlosser/248/orig 2025-09-07T06:42:00.8089570Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-09-07T06:42:00.8089730Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-09-07T06:42:00.8093242Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-09-07T06:42:00.8093848Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-09-07T06:42:00.8094051Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-09-07T06:42:00.8094213Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-09-07T06:42:00.8094365Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-09-07T06:42:00.8094518Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-09-07T06:42:00.8094851Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-09-07T06:42:00.8100151Z * [new branch] gh/jiayisunx/64/base -> origin/gh/jiayisunx/64/base 2025-09-07T06:42:00.8101882Z * [new branch] gh/jiayisunx/64/head -> origin/gh/jiayisunx/64/head 2025-09-07T06:42:00.8102219Z * [new branch] gh/jiayisunx/64/orig -> origin/gh/jiayisunx/64/orig 2025-09-07T06:42:00.8102489Z * [new branch] gh/jiayisunx/65/base -> origin/gh/jiayisunx/65/base 2025-09-07T06:42:00.8102663Z * [new branch] gh/jiayisunx/65/head -> origin/gh/jiayisunx/65/head 2025-09-07T06:42:00.8102832Z * [new branch] gh/jiayisunx/65/orig -> origin/gh/jiayisunx/65/orig 2025-09-07T06:42:00.8102987Z * [new branch] gh/jiayisunx/66/base -> origin/gh/jiayisunx/66/base 2025-09-07T06:42:00.8103178Z * [new branch] gh/jiayisunx/66/head -> origin/gh/jiayisunx/66/head 2025-09-07T06:42:00.8103401Z * [new branch] gh/jiayisunx/66/orig -> origin/gh/jiayisunx/66/orig 2025-09-07T06:42:00.8103564Z * [new branch] gh/jiayisunx/67/base -> origin/gh/jiayisunx/67/base 2025-09-07T06:42:00.8103723Z * [new branch] gh/jiayisunx/67/head -> origin/gh/jiayisunx/67/head 2025-09-07T06:42:00.8103878Z * [new branch] gh/jiayisunx/67/orig -> origin/gh/jiayisunx/67/orig 2025-09-07T06:42:00.8104044Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-09-07T06:42:00.8104197Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-09-07T06:42:00.8104358Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-09-07T06:42:00.8104512Z * [new branch] gh/jiayisunx/69/base -> origin/gh/jiayisunx/69/base 2025-09-07T06:42:00.8104664Z * [new branch] gh/jiayisunx/69/head -> origin/gh/jiayisunx/69/head 2025-09-07T06:42:00.8104835Z * [new branch] gh/jiayisunx/69/orig -> origin/gh/jiayisunx/69/orig 2025-09-07T06:42:00.8104988Z * [new branch] gh/jiayisunx/70/base -> origin/gh/jiayisunx/70/base 2025-09-07T06:42:00.8105147Z * [new branch] gh/jiayisunx/70/head -> origin/gh/jiayisunx/70/head 2025-09-07T06:42:00.8105326Z * [new branch] gh/jiayisunx/70/orig -> origin/gh/jiayisunx/70/orig 2025-09-07T06:42:00.8106437Z * [new branch] gh/jiayisunx/71/base -> origin/gh/jiayisunx/71/base 2025-09-07T06:42:00.8112484Z * [new branch] gh/jiayisunx/71/head -> origin/gh/jiayisunx/71/head 2025-09-07T06:42:00.8117396Z * [new branch] gh/jiayisunx/71/orig -> origin/gh/jiayisunx/71/orig 2025-09-07T06:42:00.8122604Z * [new branch] gh/jiayisunx/72/base -> origin/gh/jiayisunx/72/base 2025-09-07T06:42:00.8124816Z * [new branch] gh/jiayisunx/72/head -> origin/gh/jiayisunx/72/head 2025-09-07T06:42:00.8125123Z * [new branch] gh/jiayisunx/72/orig -> origin/gh/jiayisunx/72/orig 2025-09-07T06:42:00.8125466Z * [new branch] gh/jiayisunx/73/base -> origin/gh/jiayisunx/73/base 2025-09-07T06:42:00.8125692Z * [new branch] gh/jiayisunx/73/head -> origin/gh/jiayisunx/73/head 2025-09-07T06:42:00.8125959Z * [new branch] gh/jiayisunx/73/orig -> origin/gh/jiayisunx/73/orig 2025-09-07T06:42:00.8126105Z * [new branch] gh/jiayisunx/74/base -> origin/gh/jiayisunx/74/base 2025-09-07T06:42:00.8126255Z * [new branch] gh/jiayisunx/74/head -> origin/gh/jiayisunx/74/head 2025-09-07T06:42:00.8126405Z * [new branch] gh/jiayisunx/74/orig -> origin/gh/jiayisunx/74/orig 2025-09-07T06:42:00.8126538Z * [new branch] gh/jiayisunx/75/base -> origin/gh/jiayisunx/75/base 2025-09-07T06:42:00.8126877Z * [new branch] gh/jiayisunx/75/head -> origin/gh/jiayisunx/75/head 2025-09-07T06:42:00.8127040Z * [new branch] gh/jiayisunx/75/orig -> origin/gh/jiayisunx/75/orig 2025-09-07T06:42:00.8127212Z * [new branch] gh/jiayisunx/76/base -> origin/gh/jiayisunx/76/base 2025-09-07T06:42:00.8127392Z * [new branch] gh/jiayisunx/76/head -> origin/gh/jiayisunx/76/head 2025-09-07T06:42:00.8127557Z * [new branch] gh/jiayisunx/76/orig -> origin/gh/jiayisunx/76/orig 2025-09-07T06:42:00.8127724Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-09-07T06:42:00.8127873Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-09-07T06:42:00.8128038Z * [new branch] gh/justinchuby/111/base -> origin/gh/justinchuby/111/base 2025-09-07T06:42:00.8128189Z * [new branch] gh/justinchuby/111/head -> origin/gh/justinchuby/111/head 2025-09-07T06:42:00.8128349Z * [new branch] gh/justinchuby/111/orig -> origin/gh/justinchuby/111/orig 2025-09-07T06:42:00.8128500Z * [new branch] gh/justinchuby/112/base -> origin/gh/justinchuby/112/base 2025-09-07T06:42:00.8128658Z * [new branch] gh/justinchuby/112/head -> origin/gh/justinchuby/112/head 2025-09-07T06:42:00.8128805Z * [new branch] gh/justinchuby/112/orig -> origin/gh/justinchuby/112/orig 2025-09-07T06:42:00.8128955Z * [new branch] gh/justinchuby/113/base -> origin/gh/justinchuby/113/base 2025-09-07T06:42:00.8129111Z * [new branch] gh/justinchuby/113/head -> origin/gh/justinchuby/113/head 2025-09-07T06:42:00.8129642Z * [new branch] gh/justinchuby/113/orig -> origin/gh/justinchuby/113/orig 2025-09-07T06:42:00.8129808Z * [new branch] gh/justinchuby/114/base -> origin/gh/justinchuby/114/base 2025-09-07T06:42:00.8129968Z * [new branch] gh/justinchuby/114/head -> origin/gh/justinchuby/114/head 2025-09-07T06:42:00.8130231Z * [new branch] gh/justinchuby/114/orig -> origin/gh/justinchuby/114/orig 2025-09-07T06:42:00.8135706Z * [new branch] gh/justinchuby/115/base -> origin/gh/justinchuby/115/base 2025-09-07T06:42:00.8139405Z * [new branch] gh/justinchuby/115/head -> origin/gh/justinchuby/115/head 2025-09-07T06:42:00.8141901Z * [new branch] gh/justinchuby/115/orig -> origin/gh/justinchuby/115/orig 2025-09-07T06:42:00.8142119Z * [new branch] gh/karthickai/1/base -> origin/gh/karthickai/1/base 2025-09-07T06:42:00.8142290Z * [new branch] gh/karthickai/1/head -> origin/gh/karthickai/1/head 2025-09-07T06:42:00.8142438Z * [new branch] gh/karthickai/1/orig -> origin/gh/karthickai/1/orig 2025-09-07T06:42:00.8142594Z * [new branch] gh/karthickai/2/base -> origin/gh/karthickai/2/base 2025-09-07T06:42:00.8142757Z * [new branch] gh/karthickai/2/head -> origin/gh/karthickai/2/head 2025-09-07T06:42:00.8142914Z * [new branch] gh/karthickai/2/orig -> origin/gh/karthickai/2/orig 2025-09-07T06:42:00.8143080Z * [new branch] gh/kurtamohler/32/base -> origin/gh/kurtamohler/32/base 2025-09-07T06:42:00.8143239Z * [new branch] gh/kurtamohler/32/head -> origin/gh/kurtamohler/32/head 2025-09-07T06:42:00.8143414Z * [new branch] gh/kurtamohler/32/orig -> origin/gh/kurtamohler/32/orig 2025-09-07T06:42:00.8143568Z * [new branch] gh/kurtamohler/33/base -> origin/gh/kurtamohler/33/base 2025-09-07T06:42:00.8143725Z * [new branch] gh/kurtamohler/33/head -> origin/gh/kurtamohler/33/head 2025-09-07T06:42:00.8143879Z * [new branch] gh/kurtamohler/33/orig -> origin/gh/kurtamohler/33/orig 2025-09-07T06:42:00.8144043Z * [new branch] gh/kurtamohler/34/base -> origin/gh/kurtamohler/34/base 2025-09-07T06:42:00.8144346Z * [new branch] gh/kurtamohler/34/head -> origin/gh/kurtamohler/34/head 2025-09-07T06:42:00.8144510Z * [new branch] gh/kurtamohler/34/orig -> origin/gh/kurtamohler/34/orig 2025-09-07T06:42:00.8144674Z * [new branch] gh/kurtamohler/41/base -> origin/gh/kurtamohler/41/base 2025-09-07T06:42:00.8145171Z * [new branch] gh/kurtamohler/41/head -> origin/gh/kurtamohler/41/head 2025-09-07T06:42:00.8145347Z * [new branch] gh/kurtamohler/41/orig -> origin/gh/kurtamohler/41/orig 2025-09-07T06:42:00.8153062Z * [new branch] gh/kurtamohler/46/base -> origin/gh/kurtamohler/46/base 2025-09-07T06:42:00.8155350Z * [new branch] gh/kurtamohler/46/head -> origin/gh/kurtamohler/46/head 2025-09-07T06:42:00.8161926Z * [new branch] gh/kurtamohler/46/orig -> origin/gh/kurtamohler/46/orig 2025-09-07T06:42:00.8167239Z * [new branch] gh/kurtamohler/47/base -> origin/gh/kurtamohler/47/base 2025-09-07T06:42:00.8172116Z * [new branch] gh/kurtamohler/47/head -> origin/gh/kurtamohler/47/head 2025-09-07T06:42:00.8177037Z * [new branch] gh/kurtamohler/47/orig -> origin/gh/kurtamohler/47/orig 2025-09-07T06:42:00.8180973Z * [new branch] gh/kurtamohler/48/base -> origin/gh/kurtamohler/48/base 2025-09-07T06:42:00.8181195Z * [new branch] gh/kurtamohler/48/head -> origin/gh/kurtamohler/48/head 2025-09-07T06:42:00.8181698Z * [new branch] gh/kurtamohler/48/orig -> origin/gh/kurtamohler/48/orig 2025-09-07T06:42:00.8181889Z * [new branch] gh/kurtamohler/49/base -> origin/gh/kurtamohler/49/base 2025-09-07T06:42:00.8182049Z * [new branch] gh/kurtamohler/49/head -> origin/gh/kurtamohler/49/head 2025-09-07T06:42:00.8182205Z * [new branch] gh/kurtamohler/49/orig -> origin/gh/kurtamohler/49/orig 2025-09-07T06:42:00.8182383Z * [new branch] gh/kurtamohler/50/base -> origin/gh/kurtamohler/50/base 2025-09-07T06:42:00.8182538Z * [new branch] gh/kurtamohler/50/head -> origin/gh/kurtamohler/50/head 2025-09-07T06:42:00.8182701Z * [new branch] gh/kurtamohler/50/orig -> origin/gh/kurtamohler/50/orig 2025-09-07T06:42:00.8182857Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-09-07T06:42:00.8183164Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-09-07T06:42:00.8183313Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-09-07T06:42:00.8183464Z * [new branch] gh/kwen2501/15/base -> origin/gh/kwen2501/15/base 2025-09-07T06:42:00.8183616Z * [new branch] gh/kwen2501/15/head -> origin/gh/kwen2501/15/head 2025-09-07T06:42:00.8183762Z * [new branch] gh/kwen2501/156/base -> origin/gh/kwen2501/156/base 2025-09-07T06:42:00.8183919Z * [new branch] gh/kwen2501/156/head -> origin/gh/kwen2501/156/head 2025-09-07T06:42:00.8184064Z * [new branch] gh/kwen2501/156/orig -> origin/gh/kwen2501/156/orig 2025-09-07T06:42:00.8184206Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-09-07T06:42:00.8184358Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-09-07T06:42:00.8184505Z * [new branch] gh/kwen2501/186/base -> origin/gh/kwen2501/186/base 2025-09-07T06:42:00.8184656Z * [new branch] gh/kwen2501/186/head -> origin/gh/kwen2501/186/head 2025-09-07T06:42:00.8184798Z * [new branch] gh/kwen2501/186/orig -> origin/gh/kwen2501/186/orig 2025-09-07T06:42:00.8184952Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-09-07T06:42:00.8185142Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-09-07T06:42:00.8185287Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-09-07T06:42:00.8185483Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-09-07T06:42:00.8185631Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-09-07T06:42:00.8186004Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-09-07T06:42:00.8186146Z * [new branch] gh/kwen2501/194/base -> origin/gh/kwen2501/194/base 2025-09-07T06:42:00.8186287Z * [new branch] gh/kwen2501/194/head -> origin/gh/kwen2501/194/head 2025-09-07T06:42:00.8186435Z * [new branch] gh/kwen2501/194/orig -> origin/gh/kwen2501/194/orig 2025-09-07T06:42:00.8186576Z * [new branch] gh/kwen2501/199/base -> origin/gh/kwen2501/199/base 2025-09-07T06:42:00.8186741Z * [new branch] gh/kwen2501/199/head -> origin/gh/kwen2501/199/head 2025-09-07T06:42:00.8186880Z * [new branch] gh/kwen2501/199/orig -> origin/gh/kwen2501/199/orig 2025-09-07T06:42:00.8187031Z * [new branch] gh/kwen2501/200/base -> origin/gh/kwen2501/200/base 2025-09-07T06:42:00.8187183Z * [new branch] gh/kwen2501/200/head -> origin/gh/kwen2501/200/head 2025-09-07T06:42:00.8187325Z * [new branch] gh/kwen2501/200/orig -> origin/gh/kwen2501/200/orig 2025-09-07T06:42:00.8187473Z * [new branch] gh/kwen2501/201/base -> origin/gh/kwen2501/201/base 2025-09-07T06:42:00.8187611Z * [new branch] gh/kwen2501/201/head -> origin/gh/kwen2501/201/head 2025-09-07T06:42:00.8187754Z * [new branch] gh/kwen2501/201/orig -> origin/gh/kwen2501/201/orig 2025-09-07T06:42:00.8187893Z * [new branch] gh/kwen2501/203/base -> origin/gh/kwen2501/203/base 2025-09-07T06:42:00.8188042Z * [new branch] gh/kwen2501/203/head -> origin/gh/kwen2501/203/head 2025-09-07T06:42:00.8188183Z * [new branch] gh/kwen2501/203/orig -> origin/gh/kwen2501/203/orig 2025-09-07T06:42:00.8188321Z * [new branch] gh/kwen2501/204/base -> origin/gh/kwen2501/204/base 2025-09-07T06:42:00.8188467Z * [new branch] gh/kwen2501/204/head -> origin/gh/kwen2501/204/head 2025-09-07T06:42:00.8189349Z * [new branch] gh/kwen2501/204/orig -> origin/gh/kwen2501/204/orig 2025-09-07T06:42:00.8189602Z * [new branch] gh/kwen2501/205/base -> origin/gh/kwen2501/205/base 2025-09-07T06:42:00.8189760Z * [new branch] gh/kwen2501/205/head -> origin/gh/kwen2501/205/head 2025-09-07T06:42:00.8189983Z * [new branch] gh/kwen2501/205/orig -> origin/gh/kwen2501/205/orig 2025-09-07T06:42:00.8190196Z * [new branch] gh/kwen2501/206/base -> origin/gh/kwen2501/206/base 2025-09-07T06:42:00.8196464Z * [new branch] gh/kwen2501/206/head -> origin/gh/kwen2501/206/head 2025-09-07T06:42:00.8202001Z * [new branch] gh/kwen2501/206/orig -> origin/gh/kwen2501/206/orig 2025-09-07T06:42:00.8206978Z * [new branch] gh/kwen2501/207/base -> origin/gh/kwen2501/207/base 2025-09-07T06:42:00.8211998Z * [new branch] gh/kwen2501/207/head -> origin/gh/kwen2501/207/head 2025-09-07T06:42:00.8216940Z * [new branch] gh/kwen2501/207/orig -> origin/gh/kwen2501/207/orig 2025-09-07T06:42:00.8217149Z * [new branch] gh/kwen2501/208/base -> origin/gh/kwen2501/208/base 2025-09-07T06:42:00.8217381Z * [new branch] gh/kwen2501/208/head -> origin/gh/kwen2501/208/head 2025-09-07T06:42:00.8217592Z * [new branch] gh/kwen2501/208/orig -> origin/gh/kwen2501/208/orig 2025-09-07T06:42:00.8217958Z * [new branch] gh/kwen2501/209/base -> origin/gh/kwen2501/209/base 2025-09-07T06:42:00.8218190Z * [new branch] gh/kwen2501/209/head -> origin/gh/kwen2501/209/head 2025-09-07T06:42:00.8218761Z * [new branch] gh/kwen2501/209/orig -> origin/gh/kwen2501/209/orig 2025-09-07T06:42:00.8218932Z * [new branch] gh/kwen2501/210/base -> origin/gh/kwen2501/210/base 2025-09-07T06:42:00.8219082Z * [new branch] gh/kwen2501/210/head -> origin/gh/kwen2501/210/head 2025-09-07T06:42:00.8219230Z * [new branch] gh/kwen2501/210/orig -> origin/gh/kwen2501/210/orig 2025-09-07T06:42:00.8219367Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-09-07T06:42:00.8219514Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-09-07T06:42:00.8219833Z * [new branch] gh/kwen2501/212/base -> origin/gh/kwen2501/212/base 2025-09-07T06:42:00.8220015Z * [new branch] gh/kwen2501/212/head -> origin/gh/kwen2501/212/head 2025-09-07T06:42:00.8220173Z * [new branch] gh/kwen2501/212/orig -> origin/gh/kwen2501/212/orig 2025-09-07T06:42:00.8220326Z * [new branch] gh/kwen2501/213/base -> origin/gh/kwen2501/213/base 2025-09-07T06:42:00.8220496Z * [new branch] gh/kwen2501/213/head -> origin/gh/kwen2501/213/head 2025-09-07T06:42:00.8220637Z * [new branch] gh/kwen2501/213/orig -> origin/gh/kwen2501/213/orig 2025-09-07T06:42:00.8220790Z * [new branch] gh/kwen2501/214/base -> origin/gh/kwen2501/214/base 2025-09-07T06:42:00.8220938Z * [new branch] gh/kwen2501/214/head -> origin/gh/kwen2501/214/head 2025-09-07T06:42:00.8221093Z * [new branch] gh/kwen2501/214/orig -> origin/gh/kwen2501/214/orig 2025-09-07T06:42:00.8221241Z * [new branch] gh/kwen2501/215/base -> origin/gh/kwen2501/215/base 2025-09-07T06:42:00.8221396Z * [new branch] gh/kwen2501/215/head -> origin/gh/kwen2501/215/head 2025-09-07T06:42:00.8221543Z * [new branch] gh/kwen2501/215/orig -> origin/gh/kwen2501/215/orig 2025-09-07T06:42:00.8221687Z * [new branch] gh/kwen2501/216/base -> origin/gh/kwen2501/216/base 2025-09-07T06:42:00.8221836Z * [new branch] gh/kwen2501/216/head -> origin/gh/kwen2501/216/head 2025-09-07T06:42:00.8222186Z * [new branch] gh/kwen2501/216/orig -> origin/gh/kwen2501/216/orig 2025-09-07T06:42:00.8222339Z * [new branch] gh/kwen2501/217/base -> origin/gh/kwen2501/217/base 2025-09-07T06:42:00.8222496Z * [new branch] gh/kwen2501/217/head -> origin/gh/kwen2501/217/head 2025-09-07T06:42:00.8222644Z * [new branch] gh/kwen2501/217/orig -> origin/gh/kwen2501/217/orig 2025-09-07T06:42:00.8222792Z * [new branch] gh/kwen2501/218/base -> origin/gh/kwen2501/218/base 2025-09-07T06:42:00.8222943Z * [new branch] gh/kwen2501/218/head -> origin/gh/kwen2501/218/head 2025-09-07T06:42:00.8223089Z * [new branch] gh/kwen2501/218/orig -> origin/gh/kwen2501/218/orig 2025-09-07T06:42:00.8223237Z * [new branch] gh/kwen2501/219/base -> origin/gh/kwen2501/219/base 2025-09-07T06:42:00.8223385Z * [new branch] gh/kwen2501/219/head -> origin/gh/kwen2501/219/head 2025-09-07T06:42:00.8224301Z * [new branch] gh/kwen2501/219/orig -> origin/gh/kwen2501/219/orig 2025-09-07T06:42:00.8225186Z * [new branch] gh/kwen2501/220/base -> origin/gh/kwen2501/220/base 2025-09-07T06:42:00.8225848Z * [new branch] gh/kwen2501/220/head -> origin/gh/kwen2501/220/head 2025-09-07T06:42:00.8229223Z * [new branch] gh/kwen2501/220/orig -> origin/gh/kwen2501/220/orig 2025-09-07T06:42:00.8229712Z * [new branch] gh/kwen2501/221/base -> origin/gh/kwen2501/221/base 2025-09-07T06:42:00.8229956Z * [new branch] gh/kwen2501/221/head -> origin/gh/kwen2501/221/head 2025-09-07T06:42:00.8236282Z * [new branch] gh/kwen2501/221/orig -> origin/gh/kwen2501/221/orig 2025-09-07T06:42:00.8241139Z * [new branch] gh/kwen2501/222/base -> origin/gh/kwen2501/222/base 2025-09-07T06:42:00.8245279Z * [new branch] gh/kwen2501/222/head -> origin/gh/kwen2501/222/head 2025-09-07T06:42:00.8249584Z * [new branch] gh/kwen2501/222/orig -> origin/gh/kwen2501/222/orig 2025-09-07T06:42:00.8254399Z * [new branch] gh/kwen2501/223/base -> origin/gh/kwen2501/223/base 2025-09-07T06:42:00.8258152Z * [new branch] gh/kwen2501/223/head -> origin/gh/kwen2501/223/head 2025-09-07T06:42:00.8260424Z * [new branch] gh/kwen2501/223/orig -> origin/gh/kwen2501/223/orig 2025-09-07T06:42:00.8260640Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-09-07T06:42:00.8260809Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-09-07T06:42:00.8260954Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-09-07T06:42:00.8261106Z * [new branch] gh/kwen2501/225/base -> origin/gh/kwen2501/225/base 2025-09-07T06:42:00.8261256Z * [new branch] gh/kwen2501/225/head -> origin/gh/kwen2501/225/head 2025-09-07T06:42:00.8261404Z * [new branch] gh/kwen2501/225/orig -> origin/gh/kwen2501/225/orig 2025-09-07T06:42:00.8261599Z * [new branch] gh/kwen2501/226/base -> origin/gh/kwen2501/226/base 2025-09-07T06:42:00.8261761Z * [new branch] gh/kwen2501/226/head -> origin/gh/kwen2501/226/head 2025-09-07T06:42:00.8261907Z * [new branch] gh/kwen2501/226/orig -> origin/gh/kwen2501/226/orig 2025-09-07T06:42:00.8262062Z * [new branch] gh/kwen2501/227/base -> origin/gh/kwen2501/227/base 2025-09-07T06:42:00.8262215Z * [new branch] gh/kwen2501/227/head -> origin/gh/kwen2501/227/head 2025-09-07T06:42:00.8262361Z * [new branch] gh/kwen2501/227/orig -> origin/gh/kwen2501/227/orig 2025-09-07T06:42:00.8262510Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-09-07T06:42:00.8262801Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-09-07T06:42:00.8262947Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-09-07T06:42:00.8263101Z * [new branch] gh/kwen2501/229/base -> origin/gh/kwen2501/229/base 2025-09-07T06:42:00.8263246Z * [new branch] gh/kwen2501/229/head -> origin/gh/kwen2501/229/head 2025-09-07T06:42:00.8263391Z * [new branch] gh/kwen2501/229/orig -> origin/gh/kwen2501/229/orig 2025-09-07T06:42:00.8263548Z * [new branch] gh/kwen2501/230/base -> origin/gh/kwen2501/230/base 2025-09-07T06:42:00.8263702Z * [new branch] gh/kwen2501/230/head -> origin/gh/kwen2501/230/head 2025-09-07T06:42:00.8263848Z * [new branch] gh/kwen2501/230/orig -> origin/gh/kwen2501/230/orig 2025-09-07T06:42:00.8264008Z * [new branch] gh/kwen2501/231/base -> origin/gh/kwen2501/231/base 2025-09-07T06:42:00.8264156Z * [new branch] gh/kwen2501/231/head -> origin/gh/kwen2501/231/head 2025-09-07T06:42:00.8264300Z * [new branch] gh/kwen2501/231/orig -> origin/gh/kwen2501/231/orig 2025-09-07T06:42:00.8264489Z * [new branch] gh/kwen2501/232/base -> origin/gh/kwen2501/232/base 2025-09-07T06:42:00.8264633Z * [new branch] gh/kwen2501/232/head -> origin/gh/kwen2501/232/head 2025-09-07T06:42:00.8264781Z * [new branch] gh/kwen2501/232/orig -> origin/gh/kwen2501/232/orig 2025-09-07T06:42:00.8265030Z * [new branch] gh/laithsakka/156/base -> origin/gh/laithsakka/156/base 2025-09-07T06:42:00.8265196Z * [new branch] gh/laithsakka/156/head -> origin/gh/laithsakka/156/head 2025-09-07T06:42:00.8265391Z * [new branch] gh/laithsakka/156/orig -> origin/gh/laithsakka/156/orig 2025-09-07T06:42:00.8265595Z * [new branch] gh/laithsakka/160/base -> origin/gh/laithsakka/160/base 2025-09-07T06:42:00.8266081Z * [new branch] gh/laithsakka/160/head -> origin/gh/laithsakka/160/head 2025-09-07T06:42:00.8266252Z * [new branch] gh/laithsakka/160/orig -> origin/gh/laithsakka/160/orig 2025-09-07T06:42:00.8266424Z * [new branch] gh/laithsakka/178/base -> origin/gh/laithsakka/178/base 2025-09-07T06:42:00.8266585Z * [new branch] gh/laithsakka/178/head -> origin/gh/laithsakka/178/head 2025-09-07T06:42:00.8266752Z * [new branch] gh/laithsakka/178/orig -> origin/gh/laithsakka/178/orig 2025-09-07T06:42:00.8266953Z * [new branch] gh/laithsakka/191/base -> origin/gh/laithsakka/191/base 2025-09-07T06:42:00.8267124Z * [new branch] gh/laithsakka/191/head -> origin/gh/laithsakka/191/head 2025-09-07T06:42:00.8267284Z * [new branch] gh/laithsakka/191/orig -> origin/gh/laithsakka/191/orig 2025-09-07T06:42:00.8267451Z * [new branch] gh/laithsakka/237/base -> origin/gh/laithsakka/237/base 2025-09-07T06:42:00.8267616Z * [new branch] gh/laithsakka/237/head -> origin/gh/laithsakka/237/head 2025-09-07T06:42:00.8267776Z * [new branch] gh/laithsakka/237/orig -> origin/gh/laithsakka/237/orig 2025-09-07T06:42:00.8267936Z * [new branch] gh/laithsakka/249/base -> origin/gh/laithsakka/249/base 2025-09-07T06:42:00.8268102Z * [new branch] gh/laithsakka/249/head -> origin/gh/laithsakka/249/head 2025-09-07T06:42:00.8268430Z * [new branch] gh/laithsakka/249/orig -> origin/gh/laithsakka/249/orig 2025-09-07T06:42:00.8273593Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-09-07T06:42:00.8273795Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-09-07T06:42:00.8273963Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-09-07T06:42:00.8274345Z * [new branch] gh/laithsakka/254/base -> origin/gh/laithsakka/254/base 2025-09-07T06:42:00.8274529Z * [new branch] gh/laithsakka/254/head -> origin/gh/laithsakka/254/head 2025-09-07T06:42:00.8274718Z * [new branch] gh/laithsakka/254/orig -> origin/gh/laithsakka/254/orig 2025-09-07T06:42:00.8274886Z * [new branch] gh/laithsakka/255/base -> origin/gh/laithsakka/255/base 2025-09-07T06:42:00.8275086Z * [new branch] gh/laithsakka/255/head -> origin/gh/laithsakka/255/head 2025-09-07T06:42:00.8279103Z * [new branch] gh/laithsakka/255/orig -> origin/gh/laithsakka/255/orig 2025-09-07T06:42:00.8279304Z * [new branch] gh/laithsakka/256/base -> origin/gh/laithsakka/256/base 2025-09-07T06:42:00.8279525Z * [new branch] gh/laithsakka/256/head -> origin/gh/laithsakka/256/head 2025-09-07T06:42:00.8279712Z * [new branch] gh/laithsakka/256/orig -> origin/gh/laithsakka/256/orig 2025-09-07T06:42:00.8279874Z * [new branch] gh/laithsakka/257/base -> origin/gh/laithsakka/257/base 2025-09-07T06:42:00.8280036Z * [new branch] gh/laithsakka/257/head -> origin/gh/laithsakka/257/head 2025-09-07T06:42:00.8280199Z * [new branch] gh/laithsakka/257/orig -> origin/gh/laithsakka/257/orig 2025-09-07T06:42:00.8280505Z * [new branch] gh/laithsakka/258/base -> origin/gh/laithsakka/258/base 2025-09-07T06:42:00.8281305Z * [new branch] gh/laithsakka/258/head -> origin/gh/laithsakka/258/head 2025-09-07T06:42:00.8281524Z * [new branch] gh/laithsakka/258/orig -> origin/gh/laithsakka/258/orig 2025-09-07T06:42:00.8285547Z * [new branch] gh/laithsakka/259/base -> origin/gh/laithsakka/259/base 2025-09-07T06:42:00.8285745Z * [new branch] gh/laithsakka/259/head -> origin/gh/laithsakka/259/head 2025-09-07T06:42:00.8285931Z * [new branch] gh/laithsakka/259/orig -> origin/gh/laithsakka/259/orig 2025-09-07T06:42:00.8286088Z * [new branch] gh/laithsakka/260/base -> origin/gh/laithsakka/260/base 2025-09-07T06:42:00.8286255Z * [new branch] gh/laithsakka/260/head -> origin/gh/laithsakka/260/head 2025-09-07T06:42:00.8319171Z * [new branch] gh/laithsakka/260/orig -> origin/gh/laithsakka/260/orig 2025-09-07T06:42:00.8319531Z * [new branch] gh/laithsakka/261/base -> origin/gh/laithsakka/261/base 2025-09-07T06:42:00.8319940Z * [new branch] gh/laithsakka/261/head -> origin/gh/laithsakka/261/head 2025-09-07T06:42:00.8320615Z * [new branch] gh/laithsakka/261/orig -> origin/gh/laithsakka/261/orig 2025-09-07T06:42:00.8321234Z * [new branch] gh/laithsakka/262/base -> origin/gh/laithsakka/262/base 2025-09-07T06:42:00.8321617Z * [new branch] gh/laithsakka/262/head -> origin/gh/laithsakka/262/head 2025-09-07T06:42:00.8321777Z * [new branch] gh/laithsakka/262/orig -> origin/gh/laithsakka/262/orig 2025-09-07T06:42:00.8321931Z * [new branch] gh/laithsakka/263/base -> origin/gh/laithsakka/263/base 2025-09-07T06:42:00.8322069Z * [new branch] gh/laithsakka/263/head -> origin/gh/laithsakka/263/head 2025-09-07T06:42:00.8322232Z * [new branch] gh/laithsakka/263/orig -> origin/gh/laithsakka/263/orig 2025-09-07T06:42:00.8322378Z * [new branch] gh/laithsakka/264/base -> origin/gh/laithsakka/264/base 2025-09-07T06:42:00.8322524Z * [new branch] gh/laithsakka/264/head -> origin/gh/laithsakka/264/head 2025-09-07T06:42:00.8322659Z * [new branch] gh/laithsakka/264/orig -> origin/gh/laithsakka/264/orig 2025-09-07T06:42:00.8322796Z * [new branch] gh/laithsakka/265/base -> origin/gh/laithsakka/265/base 2025-09-07T06:42:00.8323122Z * [new branch] gh/laithsakka/265/head -> origin/gh/laithsakka/265/head 2025-09-07T06:42:00.8323269Z * [new branch] gh/laithsakka/265/orig -> origin/gh/laithsakka/265/orig 2025-09-07T06:42:00.8323425Z * [new branch] gh/laithsakka/266/base -> origin/gh/laithsakka/266/base 2025-09-07T06:42:00.8323572Z * [new branch] gh/laithsakka/266/head -> origin/gh/laithsakka/266/head 2025-09-07T06:42:00.8323730Z * [new branch] gh/laithsakka/266/orig -> origin/gh/laithsakka/266/orig 2025-09-07T06:42:00.8323873Z * [new branch] gh/laithsakka/267/base -> origin/gh/laithsakka/267/base 2025-09-07T06:42:00.8324013Z * [new branch] gh/laithsakka/267/head -> origin/gh/laithsakka/267/head 2025-09-07T06:42:00.8324159Z * [new branch] gh/laithsakka/267/orig -> origin/gh/laithsakka/267/orig 2025-09-07T06:42:00.8324298Z * [new branch] gh/laithsakka/268/base -> origin/gh/laithsakka/268/base 2025-09-07T06:42:00.8324448Z * [new branch] gh/laithsakka/268/head -> origin/gh/laithsakka/268/head 2025-09-07T06:42:00.8324588Z * [new branch] gh/laithsakka/268/orig -> origin/gh/laithsakka/268/orig 2025-09-07T06:42:00.8324744Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-09-07T06:42:00.8324886Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-09-07T06:42:00.8325101Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-09-07T06:42:00.8325251Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-09-07T06:42:00.8325387Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-09-07T06:42:00.8325543Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-09-07T06:42:00.8325681Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-09-07T06:42:00.8325814Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-09-07T06:42:00.8325965Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-09-07T06:42:00.8326103Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-09-07T06:42:00.8326252Z * [new branch] gh/lucaskabela/10/base -> origin/gh/lucaskabela/10/base 2025-09-07T06:42:00.8326389Z * [new branch] gh/lucaskabela/10/head -> origin/gh/lucaskabela/10/head 2025-09-07T06:42:00.8326535Z * [new branch] gh/lucaskabela/10/orig -> origin/gh/lucaskabela/10/orig 2025-09-07T06:42:00.8326673Z * [new branch] gh/lucaskabela/11/base -> origin/gh/lucaskabela/11/base 2025-09-07T06:42:00.8326807Z * [new branch] gh/lucaskabela/11/head -> origin/gh/lucaskabela/11/head 2025-09-07T06:42:00.8326954Z * [new branch] gh/lucaskabela/11/orig -> origin/gh/lucaskabela/11/orig 2025-09-07T06:42:00.8327088Z * [new branch] gh/lucaskabela/12/base -> origin/gh/lucaskabela/12/base 2025-09-07T06:42:00.8327231Z * [new branch] gh/lucaskabela/12/head -> origin/gh/lucaskabela/12/head 2025-09-07T06:42:00.8327371Z * [new branch] gh/lucaskabela/12/orig -> origin/gh/lucaskabela/12/orig 2025-09-07T06:42:00.8327513Z * [new branch] gh/lucaskabela/13/base -> origin/gh/lucaskabela/13/base 2025-09-07T06:42:00.8327649Z * [new branch] gh/lucaskabela/13/head -> origin/gh/lucaskabela/13/head 2025-09-07T06:42:00.8327784Z * [new branch] gh/lucaskabela/13/orig -> origin/gh/lucaskabela/13/orig 2025-09-07T06:42:00.8327925Z * [new branch] gh/lucaskabela/14/base -> origin/gh/lucaskabela/14/base 2025-09-07T06:42:00.8328058Z * [new branch] gh/lucaskabela/14/head -> origin/gh/lucaskabela/14/head 2025-09-07T06:42:00.8328232Z * [new branch] gh/lucaskabela/14/orig -> origin/gh/lucaskabela/14/orig 2025-09-07T06:42:00.8328492Z * [new branch] gh/lucaskabela/15/base -> origin/gh/lucaskabela/15/base 2025-09-07T06:42:00.8328727Z * [new branch] gh/lucaskabela/15/head -> origin/gh/lucaskabela/15/head 2025-09-07T06:42:00.8328967Z * [new branch] gh/lucaskabela/15/orig -> origin/gh/lucaskabela/15/orig 2025-09-07T06:42:00.8333153Z * [new branch] gh/lucaskabela/16/base -> origin/gh/lucaskabela/16/base 2025-09-07T06:42:00.8333348Z * [new branch] gh/lucaskabela/16/head -> origin/gh/lucaskabela/16/head 2025-09-07T06:42:00.8333841Z * [new branch] gh/lucaskabela/16/orig -> origin/gh/lucaskabela/16/orig 2025-09-07T06:42:00.8334041Z * [new branch] gh/lucaskabela/17/base -> origin/gh/lucaskabela/17/base 2025-09-07T06:42:00.8334238Z * [new branch] gh/lucaskabela/17/head -> origin/gh/lucaskabela/17/head 2025-09-07T06:42:00.8334392Z * [new branch] gh/lucaskabela/17/orig -> origin/gh/lucaskabela/17/orig 2025-09-07T06:42:00.8334714Z * [new branch] gh/lucaskabela/2/base -> origin/gh/lucaskabela/2/base 2025-09-07T06:42:00.8334938Z * [new branch] gh/lucaskabela/2/head -> origin/gh/lucaskabela/2/head 2025-09-07T06:42:00.8336689Z * [new branch] gh/lucaskabela/2/orig -> origin/gh/lucaskabela/2/orig 2025-09-07T06:42:00.8337144Z * [new branch] gh/lucaskabela/3/base -> origin/gh/lucaskabela/3/base 2025-09-07T06:42:00.8337557Z * [new branch] gh/lucaskabela/3/head -> origin/gh/lucaskabela/3/head 2025-09-07T06:42:00.8337943Z * [new branch] gh/lucaskabela/3/orig -> origin/gh/lucaskabela/3/orig 2025-09-07T06:42:00.8338934Z * [new branch] gh/lucaskabela/4/base -> origin/gh/lucaskabela/4/base 2025-09-07T06:42:00.8339319Z * [new branch] gh/lucaskabela/4/head -> origin/gh/lucaskabela/4/head 2025-09-07T06:42:00.8340597Z * [new branch] gh/lucaskabela/4/orig -> origin/gh/lucaskabela/4/orig 2025-09-07T06:42:00.8341021Z * [new branch] gh/lucaskabela/5/base -> origin/gh/lucaskabela/5/base 2025-09-07T06:42:00.8341996Z * [new branch] gh/lucaskabela/5/head -> origin/gh/lucaskabela/5/head 2025-09-07T06:42:00.8342364Z * [new branch] gh/lucaskabela/5/orig -> origin/gh/lucaskabela/5/orig 2025-09-07T06:42:00.8343496Z * [new branch] gh/lucaskabela/6/base -> origin/gh/lucaskabela/6/base 2025-09-07T06:42:00.8343903Z * [new branch] gh/lucaskabela/6/head -> origin/gh/lucaskabela/6/head 2025-09-07T06:42:00.8344836Z * [new branch] gh/lucaskabela/6/orig -> origin/gh/lucaskabela/6/orig 2025-09-07T06:42:00.8346234Z * [new branch] gh/lucaskabela/7/base -> origin/gh/lucaskabela/7/base 2025-09-07T06:42:00.8347515Z * [new branch] gh/lucaskabela/7/head -> origin/gh/lucaskabela/7/head 2025-09-07T06:42:00.8347933Z * [new branch] gh/lucaskabela/7/orig -> origin/gh/lucaskabela/7/orig 2025-09-07T06:42:00.8348119Z * [new branch] gh/lucaskabela/8/base -> origin/gh/lucaskabela/8/base 2025-09-07T06:42:00.8354330Z * [new branch] gh/lucaskabela/8/head -> origin/gh/lucaskabela/8/head 2025-09-07T06:42:00.8354553Z * [new branch] gh/lucaskabela/8/orig -> origin/gh/lucaskabela/8/orig 2025-09-07T06:42:00.8354708Z * [new branch] gh/lucaskabela/9/base -> origin/gh/lucaskabela/9/base 2025-09-07T06:42:00.8354870Z * [new branch] gh/lucaskabela/9/head -> origin/gh/lucaskabela/9/head 2025-09-07T06:42:00.8355033Z * [new branch] gh/lucaskabela/9/orig -> origin/gh/lucaskabela/9/orig 2025-09-07T06:42:00.8355284Z * [new branch] gh/lw/3/base -> origin/gh/lw/3/base 2025-09-07T06:42:00.8358342Z * [new branch] gh/lw/3/head -> origin/gh/lw/3/head 2025-09-07T06:42:00.8358510Z * [new branch] gh/lw/3/orig -> origin/gh/lw/3/orig 2025-09-07T06:42:00.8358684Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-09-07T06:42:00.8359375Z * [new branch] gh/malfet/330/base -> origin/gh/malfet/330/base 2025-09-07T06:42:00.8359748Z * [new branch] gh/malfet/330/head -> origin/gh/malfet/330/head 2025-09-07T06:42:00.8359905Z * [new branch] gh/malfet/330/orig -> origin/gh/malfet/330/orig 2025-09-07T06:42:00.8360045Z * [new branch] gh/malfet/396/base -> origin/gh/malfet/396/base 2025-09-07T06:42:00.8362432Z * [new branch] gh/malfet/396/head -> origin/gh/malfet/396/head 2025-09-07T06:42:00.8362646Z * [new branch] gh/malfet/396/orig -> origin/gh/malfet/396/orig 2025-09-07T06:42:00.8362807Z * [new branch] gh/malfet/397/base -> origin/gh/malfet/397/base 2025-09-07T06:42:00.8362951Z * [new branch] gh/malfet/397/head -> origin/gh/malfet/397/head 2025-09-07T06:42:00.8363109Z * [new branch] gh/malfet/397/orig -> origin/gh/malfet/397/orig 2025-09-07T06:42:00.8363264Z * [new branch] gh/malfet/398/base -> origin/gh/malfet/398/base 2025-09-07T06:42:00.8368546Z * [new branch] gh/malfet/398/head -> origin/gh/malfet/398/head 2025-09-07T06:42:00.8368746Z * [new branch] gh/malfet/398/orig -> origin/gh/malfet/398/orig 2025-09-07T06:42:00.8368885Z * [new branch] gh/malfet/399/base -> origin/gh/malfet/399/base 2025-09-07T06:42:00.8369032Z * [new branch] gh/malfet/399/head -> origin/gh/malfet/399/head 2025-09-07T06:42:00.8369176Z * [new branch] gh/malfet/399/orig -> origin/gh/malfet/399/orig 2025-09-07T06:42:00.8369323Z * [new branch] gh/malfet/414/base -> origin/gh/malfet/414/base 2025-09-07T06:42:00.8369458Z * [new branch] gh/malfet/414/head -> origin/gh/malfet/414/head 2025-09-07T06:42:00.8370728Z * [new branch] gh/malfet/414/orig -> origin/gh/malfet/414/orig 2025-09-07T06:42:00.8371410Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-09-07T06:42:00.8371603Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-09-07T06:42:00.8371886Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-09-07T06:42:00.8372035Z * [new branch] gh/malfet/418/base -> origin/gh/malfet/418/base 2025-09-07T06:42:00.8372189Z * [new branch] gh/malfet/418/head -> origin/gh/malfet/418/head 2025-09-07T06:42:00.8376134Z * [new branch] gh/malfet/418/orig -> origin/gh/malfet/418/orig 2025-09-07T06:42:00.8376296Z * [new branch] gh/malfet/475/base -> origin/gh/malfet/475/base 2025-09-07T06:42:00.8376452Z * [new branch] gh/malfet/475/head -> origin/gh/malfet/475/head 2025-09-07T06:42:00.8376604Z * [new branch] gh/malfet/475/orig -> origin/gh/malfet/475/orig 2025-09-07T06:42:00.8376752Z * [new branch] gh/malfet/476/base -> origin/gh/malfet/476/base 2025-09-07T06:42:00.8376910Z * [new branch] gh/malfet/476/head -> origin/gh/malfet/476/head 2025-09-07T06:42:00.8377054Z * [new branch] gh/malfet/476/orig -> origin/gh/malfet/476/orig 2025-09-07T06:42:00.8377226Z * [new branch] gh/malfet/477/base -> origin/gh/malfet/477/base 2025-09-07T06:42:00.8378203Z * [new branch] gh/malfet/477/head -> origin/gh/malfet/477/head 2025-09-07T06:42:00.8381237Z * [new branch] gh/malfet/477/orig -> origin/gh/malfet/477/orig 2025-09-07T06:42:00.8381433Z * [new branch] gh/malfet/478/base -> origin/gh/malfet/478/base 2025-09-07T06:42:00.8382016Z * [new branch] gh/malfet/478/head -> origin/gh/malfet/478/head 2025-09-07T06:42:00.8382196Z * [new branch] gh/malfet/478/orig -> origin/gh/malfet/478/orig 2025-09-07T06:42:00.8382351Z * [new branch] gh/malfet/479/base -> origin/gh/malfet/479/base 2025-09-07T06:42:00.8382527Z * [new branch] gh/malfet/479/head -> origin/gh/malfet/479/head 2025-09-07T06:42:00.8383135Z * [new branch] gh/malfet/479/orig -> origin/gh/malfet/479/orig 2025-09-07T06:42:00.8384201Z * [new branch] gh/malfet/480/base -> origin/gh/malfet/480/base 2025-09-07T06:42:00.8384596Z * [new branch] gh/malfet/480/head -> origin/gh/malfet/480/head 2025-09-07T06:42:00.8385607Z * [new branch] gh/malfet/480/orig -> origin/gh/malfet/480/orig 2025-09-07T06:42:00.8386759Z * [new branch] gh/malfet/481/base -> origin/gh/malfet/481/base 2025-09-07T06:42:00.8391047Z * [new branch] gh/malfet/481/head -> origin/gh/malfet/481/head 2025-09-07T06:42:00.8391348Z * [new branch] gh/malfet/481/orig -> origin/gh/malfet/481/orig 2025-09-07T06:42:00.8391546Z * [new branch] gh/malfet/482/base -> origin/gh/malfet/482/base 2025-09-07T06:42:00.8391842Z * [new branch] gh/malfet/482/head -> origin/gh/malfet/482/head 2025-09-07T06:42:00.8391999Z * [new branch] gh/malfet/482/orig -> origin/gh/malfet/482/orig 2025-09-07T06:42:00.8392146Z * [new branch] gh/malfet/483/base -> origin/gh/malfet/483/base 2025-09-07T06:42:00.8398057Z * [new branch] gh/malfet/483/head -> origin/gh/malfet/483/head 2025-09-07T06:42:00.8398257Z * [new branch] gh/malfet/483/orig -> origin/gh/malfet/483/orig 2025-09-07T06:42:00.8398418Z * [new branch] gh/malfet/484/base -> origin/gh/malfet/484/base 2025-09-07T06:42:00.8398570Z * [new branch] gh/malfet/484/head -> origin/gh/malfet/484/head 2025-09-07T06:42:00.8398730Z * [new branch] gh/malfet/484/orig -> origin/gh/malfet/484/orig 2025-09-07T06:42:00.8398880Z * [new branch] gh/malfet/485/base -> origin/gh/malfet/485/base 2025-09-07T06:42:00.8399039Z * [new branch] gh/malfet/485/head -> origin/gh/malfet/485/head 2025-09-07T06:42:00.8404205Z * [new branch] gh/malfet/485/orig -> origin/gh/malfet/485/orig 2025-09-07T06:42:00.8404528Z * [new branch] gh/malfet/486/base -> origin/gh/malfet/486/base 2025-09-07T06:42:00.8404781Z * [new branch] gh/malfet/486/head -> origin/gh/malfet/486/head 2025-09-07T06:42:00.8405015Z * [new branch] gh/malfet/486/orig -> origin/gh/malfet/486/orig 2025-09-07T06:42:00.8405175Z * [new branch] gh/malfet/487/base -> origin/gh/malfet/487/base 2025-09-07T06:42:00.8405304Z * [new branch] gh/malfet/487/head -> origin/gh/malfet/487/head 2025-09-07T06:42:00.8405445Z * [new branch] gh/malfet/487/orig -> origin/gh/malfet/487/orig 2025-09-07T06:42:00.8405583Z * [new branch] gh/malfet/488/base -> origin/gh/malfet/488/base 2025-09-07T06:42:00.8406424Z * [new branch] gh/malfet/488/head -> origin/gh/malfet/488/head 2025-09-07T06:42:00.8406599Z * [new branch] gh/malfet/488/orig -> origin/gh/malfet/488/orig 2025-09-07T06:42:00.8406805Z * [new branch] gh/malfet/489/base -> origin/gh/malfet/489/base 2025-09-07T06:42:00.8407029Z * [new branch] gh/malfet/489/head -> origin/gh/malfet/489/head 2025-09-07T06:42:00.8407400Z * [new branch] gh/malfet/489/orig -> origin/gh/malfet/489/orig 2025-09-07T06:42:00.8407557Z * [new branch] gh/malfet/490/base -> origin/gh/malfet/490/base 2025-09-07T06:42:00.8407780Z * [new branch] gh/malfet/490/head -> origin/gh/malfet/490/head 2025-09-07T06:42:00.8407939Z * [new branch] gh/malfet/490/orig -> origin/gh/malfet/490/orig 2025-09-07T06:42:00.8412826Z * [new branch] gh/malfet/491/base -> origin/gh/malfet/491/base 2025-09-07T06:42:00.8413162Z * [new branch] gh/malfet/491/head -> origin/gh/malfet/491/head 2025-09-07T06:42:00.8413345Z * [new branch] gh/malfet/491/orig -> origin/gh/malfet/491/orig 2025-09-07T06:42:00.8413492Z * [new branch] gh/malfet/492/base -> origin/gh/malfet/492/base 2025-09-07T06:42:00.8413650Z * [new branch] gh/malfet/492/head -> origin/gh/malfet/492/head 2025-09-07T06:42:00.8413936Z * [new branch] gh/malfet/492/orig -> origin/gh/malfet/492/orig 2025-09-07T06:42:00.8418441Z * [new branch] gh/malfet/493/base -> origin/gh/malfet/493/base 2025-09-07T06:42:00.8418782Z * [new branch] gh/malfet/493/head -> origin/gh/malfet/493/head 2025-09-07T06:42:00.8418962Z * [new branch] gh/malfet/493/orig -> origin/gh/malfet/493/orig 2025-09-07T06:42:00.8419350Z * [new branch] gh/malfet/494/base -> origin/gh/malfet/494/base 2025-09-07T06:42:00.8419799Z * [new branch] gh/malfet/494/head -> origin/gh/malfet/494/head 2025-09-07T06:42:00.8419983Z * [new branch] gh/malfet/494/orig -> origin/gh/malfet/494/orig 2025-09-07T06:42:00.8420125Z * [new branch] gh/malfet/495/base -> origin/gh/malfet/495/base 2025-09-07T06:42:00.8420288Z * [new branch] gh/malfet/495/head -> origin/gh/malfet/495/head 2025-09-07T06:42:00.8420434Z * [new branch] gh/malfet/495/orig -> origin/gh/malfet/495/orig 2025-09-07T06:42:00.8420580Z * [new branch] gh/malfet/496/base -> origin/gh/malfet/496/base 2025-09-07T06:42:00.8420718Z * [new branch] gh/malfet/496/head -> origin/gh/malfet/496/head 2025-09-07T06:42:00.8420854Z * [new branch] gh/malfet/496/orig -> origin/gh/malfet/496/orig 2025-09-07T06:42:00.8421006Z * [new branch] gh/malfet/497/base -> origin/gh/malfet/497/base 2025-09-07T06:42:00.8421433Z * [new branch] gh/malfet/497/head -> origin/gh/malfet/497/head 2025-09-07T06:42:00.8422375Z * [new branch] gh/malfet/497/orig -> origin/gh/malfet/497/orig 2025-09-07T06:42:00.8423479Z * [new branch] gh/malfet/498/base -> origin/gh/malfet/498/base 2025-09-07T06:42:00.8424023Z * [new branch] gh/malfet/498/head -> origin/gh/malfet/498/head 2025-09-07T06:42:00.8424596Z * [new branch] gh/malfet/498/orig -> origin/gh/malfet/498/orig 2025-09-07T06:42:00.8425829Z * [new branch] gh/malfet/499/base -> origin/gh/malfet/499/base 2025-09-07T06:42:00.8426148Z * [new branch] gh/malfet/499/head -> origin/gh/malfet/499/head 2025-09-07T06:42:00.8427114Z * [new branch] gh/malfet/499/orig -> origin/gh/malfet/499/orig 2025-09-07T06:42:00.8429512Z * [new branch] gh/malfet/500/base -> origin/gh/malfet/500/base 2025-09-07T06:42:00.8429697Z * [new branch] gh/malfet/500/head -> origin/gh/malfet/500/head 2025-09-07T06:42:00.8429848Z * [new branch] gh/malfet/500/orig -> origin/gh/malfet/500/orig 2025-09-07T06:42:00.8431819Z * [new branch] gh/malfet/501/base -> origin/gh/malfet/501/base 2025-09-07T06:42:00.8432408Z * [new branch] gh/malfet/501/head -> origin/gh/malfet/501/head 2025-09-07T06:42:00.8432589Z * [new branch] gh/malfet/501/orig -> origin/gh/malfet/501/orig 2025-09-07T06:42:00.8432851Z * [new branch] gh/malfet/502/base -> origin/gh/malfet/502/base 2025-09-07T06:42:00.8433025Z * [new branch] gh/malfet/502/head -> origin/gh/malfet/502/head 2025-09-07T06:42:00.8434422Z * [new branch] gh/malfet/502/orig -> origin/gh/malfet/502/orig 2025-09-07T06:42:00.8434831Z * [new branch] gh/malfet/503/base -> origin/gh/malfet/503/base 2025-09-07T06:42:00.8437428Z * [new branch] gh/malfet/503/head -> origin/gh/malfet/503/head 2025-09-07T06:42:00.8437764Z * [new branch] gh/malfet/503/orig -> origin/gh/malfet/503/orig 2025-09-07T06:42:00.8438007Z * [new branch] gh/malfet/504/base -> origin/gh/malfet/504/base 2025-09-07T06:42:00.8438192Z * [new branch] gh/malfet/504/head -> origin/gh/malfet/504/head 2025-09-07T06:42:00.8438469Z * [new branch] gh/malfet/504/orig -> origin/gh/malfet/504/orig 2025-09-07T06:42:00.8442381Z * [new branch] gh/malfet/505/base -> origin/gh/malfet/505/base 2025-09-07T06:42:00.8442567Z * [new branch] gh/malfet/505/head -> origin/gh/malfet/505/head 2025-09-07T06:42:00.8442718Z * [new branch] gh/malfet/505/orig -> origin/gh/malfet/505/orig 2025-09-07T06:42:00.8443065Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-09-07T06:42:00.8443215Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-09-07T06:42:00.8443374Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-09-07T06:42:00.8443893Z * [new branch] gh/malfet/507/base -> origin/gh/malfet/507/base 2025-09-07T06:42:00.8444489Z * [new branch] gh/malfet/507/head -> origin/gh/malfet/507/head 2025-09-07T06:42:00.8445093Z * [new branch] gh/malfet/507/orig -> origin/gh/malfet/507/orig 2025-09-07T06:42:00.8449229Z * [new branch] gh/malfet/508/base -> origin/gh/malfet/508/base 2025-09-07T06:42:00.8449416Z * [new branch] gh/malfet/508/head -> origin/gh/malfet/508/head 2025-09-07T06:42:00.8449556Z * [new branch] gh/malfet/508/orig -> origin/gh/malfet/508/orig 2025-09-07T06:42:00.8449718Z * [new branch] gh/malfet/509/base -> origin/gh/malfet/509/base 2025-09-07T06:42:00.8449861Z * [new branch] gh/malfet/509/head -> origin/gh/malfet/509/head 2025-09-07T06:42:00.8458491Z * [new branch] gh/malfet/509/orig -> origin/gh/malfet/509/orig 2025-09-07T06:42:00.8461683Z * [new branch] gh/malfet/510/base -> origin/gh/malfet/510/base 2025-09-07T06:42:00.8462427Z * [new branch] gh/malfet/510/head -> origin/gh/malfet/510/head 2025-09-07T06:42:00.8462806Z * [new branch] gh/malfet/510/orig -> origin/gh/malfet/510/orig 2025-09-07T06:42:00.8462994Z * [new branch] gh/malfet/511/base -> origin/gh/malfet/511/base 2025-09-07T06:42:00.8463134Z * [new branch] gh/malfet/511/head -> origin/gh/malfet/511/head 2025-09-07T06:42:00.8463279Z * [new branch] gh/malfet/511/orig -> origin/gh/malfet/511/orig 2025-09-07T06:42:00.8463423Z * [new branch] gh/malfet/512/base -> origin/gh/malfet/512/base 2025-09-07T06:42:00.8463567Z * [new branch] gh/malfet/512/head -> origin/gh/malfet/512/head 2025-09-07T06:42:00.8463703Z * [new branch] gh/malfet/512/orig -> origin/gh/malfet/512/orig 2025-09-07T06:42:00.8463845Z * [new branch] gh/malfet/513/base -> origin/gh/malfet/513/base 2025-09-07T06:42:00.8464137Z * [new branch] gh/malfet/513/head -> origin/gh/malfet/513/head 2025-09-07T06:42:00.8464274Z * [new branch] gh/malfet/513/orig -> origin/gh/malfet/513/orig 2025-09-07T06:42:00.8464433Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-09-07T06:42:00.8464577Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-09-07T06:42:00.8464772Z * [new branch] gh/manuelcandales/10/base -> origin/gh/manuelcandales/10/base 2025-09-07T06:42:00.8464941Z * [new branch] gh/manuelcandales/10/head -> origin/gh/manuelcandales/10/head 2025-09-07T06:42:00.8465111Z * [new branch] gh/manuelcandales/10/orig -> origin/gh/manuelcandales/10/orig 2025-09-07T06:42:00.8465271Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-09-07T06:42:00.8465438Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-09-07T06:42:00.8465615Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-09-07T06:42:00.8466279Z * [new branch] gh/manuelcandales/9/base -> origin/gh/manuelcandales/9/base 2025-09-07T06:42:00.8467974Z * [new branch] gh/manuelcandales/9/head -> origin/gh/manuelcandales/9/head 2025-09-07T06:42:00.8468205Z * [new branch] gh/manuelcandales/9/orig -> origin/gh/manuelcandales/9/orig 2025-09-07T06:42:00.8471084Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-09-07T06:42:00.8471294Z * [new branch] gh/masnesral/204/base -> origin/gh/masnesral/204/base 2025-09-07T06:42:00.8471457Z * [new branch] gh/masnesral/204/head -> origin/gh/masnesral/204/head 2025-09-07T06:42:00.8472033Z * [new branch] gh/masnesral/204/orig -> origin/gh/masnesral/204/orig 2025-09-07T06:42:00.8473284Z * [new branch] gh/masnesral/235/base -> origin/gh/masnesral/235/base 2025-09-07T06:42:00.8473813Z * [new branch] gh/masnesral/235/head -> origin/gh/masnesral/235/head 2025-09-07T06:42:00.8474694Z * [new branch] gh/masnesral/235/orig -> origin/gh/masnesral/235/orig 2025-09-07T06:42:00.8475618Z * [new branch] gh/masnesral/34/base -> origin/gh/masnesral/34/base 2025-09-07T06:42:00.8477592Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-09-07T06:42:00.8477749Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-09-07T06:42:00.8478213Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-09-07T06:42:00.8479345Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-09-07T06:42:00.8479627Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-09-07T06:42:00.8482705Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-09-07T06:42:00.8482929Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-09-07T06:42:00.8483082Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-09-07T06:42:00.8483236Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-09-07T06:42:00.8483394Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-09-07T06:42:00.8484638Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-09-07T06:42:00.8484919Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-09-07T06:42:00.8486001Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-09-07T06:42:00.8486306Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-09-07T06:42:00.8487888Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-09-07T06:42:00.8488270Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-09-07T06:42:00.8489982Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-09-07T06:42:00.8490164Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-09-07T06:42:00.8490589Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-09-07T06:42:00.8491395Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-09-07T06:42:00.8492311Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-09-07T06:42:00.8492753Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-09-07T06:42:00.8493952Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-09-07T06:42:00.8494408Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-09-07T06:42:00.8495603Z * [new branch] gh/mikaylagawarecki/317/base -> origin/gh/mikaylagawarecki/317/base 2025-09-07T06:42:00.8496238Z * [new branch] gh/mikaylagawarecki/317/head -> origin/gh/mikaylagawarecki/317/head 2025-09-07T06:42:00.8496850Z * [new branch] gh/mikaylagawarecki/317/orig -> origin/gh/mikaylagawarecki/317/orig 2025-09-07T06:42:00.8497970Z * [new branch] gh/mikaylagawarecki/320/base -> origin/gh/mikaylagawarecki/320/base 2025-09-07T06:42:00.8499109Z * [new branch] gh/mikaylagawarecki/320/head -> origin/gh/mikaylagawarecki/320/head 2025-09-07T06:42:00.8499289Z * [new branch] gh/mikaylagawarecki/320/orig -> origin/gh/mikaylagawarecki/320/orig 2025-09-07T06:42:00.8501377Z * [new branch] gh/mikaylagawarecki/329/base -> origin/gh/mikaylagawarecki/329/base 2025-09-07T06:42:00.8501608Z * [new branch] gh/mikaylagawarecki/329/head -> origin/gh/mikaylagawarecki/329/head 2025-09-07T06:42:00.8501809Z * [new branch] gh/mikaylagawarecki/329/orig -> origin/gh/mikaylagawarecki/329/orig 2025-09-07T06:42:00.8502838Z * [new branch] gh/mikaylagawarecki/330/base -> origin/gh/mikaylagawarecki/330/base 2025-09-07T06:42:00.8503374Z * [new branch] gh/mikaylagawarecki/330/head -> origin/gh/mikaylagawarecki/330/head 2025-09-07T06:42:00.8504144Z * [new branch] gh/mikaylagawarecki/330/orig -> origin/gh/mikaylagawarecki/330/orig 2025-09-07T06:42:00.8505200Z * [new branch] gh/mikaylagawarecki/331/base -> origin/gh/mikaylagawarecki/331/base 2025-09-07T06:42:00.8506984Z * [new branch] gh/mikaylagawarecki/331/head -> origin/gh/mikaylagawarecki/331/head 2025-09-07T06:42:00.8507193Z * [new branch] gh/mikaylagawarecki/331/orig -> origin/gh/mikaylagawarecki/331/orig 2025-09-07T06:42:00.8507660Z * [new branch] gh/mikaylagawarecki/332/base -> origin/gh/mikaylagawarecki/332/base 2025-09-07T06:42:00.8508492Z * [new branch] gh/mikaylagawarecki/332/head -> origin/gh/mikaylagawarecki/332/head 2025-09-07T06:42:00.8509351Z * [new branch] gh/mikaylagawarecki/332/orig -> origin/gh/mikaylagawarecki/332/orig 2025-09-07T06:42:00.8510594Z * [new branch] gh/mikaylagawarecki/334/base -> origin/gh/mikaylagawarecki/334/base 2025-09-07T06:42:00.8514527Z * [new branch] gh/mikaylagawarecki/334/head -> origin/gh/mikaylagawarecki/334/head 2025-09-07T06:42:00.8514730Z * [new branch] gh/mikaylagawarecki/334/orig -> origin/gh/mikaylagawarecki/334/orig 2025-09-07T06:42:00.8515112Z * [new branch] gh/mikaylagawarecki/335/base -> origin/gh/mikaylagawarecki/335/base 2025-09-07T06:42:00.8515382Z * [new branch] gh/mikaylagawarecki/335/head -> origin/gh/mikaylagawarecki/335/head 2025-09-07T06:42:00.8515566Z * [new branch] gh/mikaylagawarecki/335/orig -> origin/gh/mikaylagawarecki/335/orig 2025-09-07T06:42:00.8515739Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-09-07T06:42:00.8516089Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-09-07T06:42:00.8516473Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-09-07T06:42:00.8517535Z * [new branch] gh/mikaylagawarecki/337/base -> origin/gh/mikaylagawarecki/337/base 2025-09-07T06:42:00.8518073Z * [new branch] gh/mikaylagawarecki/337/head -> origin/gh/mikaylagawarecki/337/head 2025-09-07T06:42:00.8518918Z * [new branch] gh/mikaylagawarecki/337/orig -> origin/gh/mikaylagawarecki/337/orig 2025-09-07T06:42:00.8519949Z * [new branch] gh/mikaylagawarecki/338/base -> origin/gh/mikaylagawarecki/338/base 2025-09-07T06:42:00.8520534Z * [new branch] gh/mikaylagawarecki/338/head -> origin/gh/mikaylagawarecki/338/head 2025-09-07T06:42:00.8521337Z * [new branch] gh/mikaylagawarecki/338/orig -> origin/gh/mikaylagawarecki/338/orig 2025-09-07T06:42:00.8522589Z * [new branch] gh/mikaylagawarecki/339/base -> origin/gh/mikaylagawarecki/339/base 2025-09-07T06:42:00.8523051Z * [new branch] gh/mikaylagawarecki/339/head -> origin/gh/mikaylagawarecki/339/head 2025-09-07T06:42:00.8524160Z * [new branch] gh/mikaylagawarecki/339/orig -> origin/gh/mikaylagawarecki/339/orig 2025-09-07T06:42:00.8525264Z * [new branch] gh/mlazos/1/base -> origin/gh/mlazos/1/base 2025-09-07T06:42:00.8525782Z * [new branch] gh/mlazos/1/head -> origin/gh/mlazos/1/head 2025-09-07T06:42:00.8526648Z * [new branch] gh/mlazos/1/orig -> origin/gh/mlazos/1/orig 2025-09-07T06:42:00.8527636Z * [new branch] gh/mlazos/12/base -> origin/gh/mlazos/12/base 2025-09-07T06:42:00.8528277Z * [new branch] gh/mlazos/12/head -> origin/gh/mlazos/12/head 2025-09-07T06:42:00.8529052Z * [new branch] gh/mlazos/12/orig -> origin/gh/mlazos/12/orig 2025-09-07T06:42:00.8530079Z * [new branch] gh/mlazos/13/base -> origin/gh/mlazos/13/base 2025-09-07T06:42:00.8530536Z * [new branch] gh/mlazos/13/head -> origin/gh/mlazos/13/head 2025-09-07T06:42:00.8531440Z * [new branch] gh/mlazos/13/orig -> origin/gh/mlazos/13/orig 2025-09-07T06:42:00.8532848Z * [new branch] gh/mlazos/14/base -> origin/gh/mlazos/14/base 2025-09-07T06:42:00.8533001Z * [new branch] gh/mlazos/14/head -> origin/gh/mlazos/14/head 2025-09-07T06:42:00.8534149Z * [new branch] gh/mlazos/14/orig -> origin/gh/mlazos/14/orig 2025-09-07T06:42:00.8534414Z * [new branch] gh/mlazos/15/base -> origin/gh/mlazos/15/base 2025-09-07T06:42:00.8535490Z * [new branch] gh/mlazos/15/head -> origin/gh/mlazos/15/head 2025-09-07T06:42:00.8535946Z * [new branch] gh/mlazos/15/orig -> origin/gh/mlazos/15/orig 2025-09-07T06:42:00.8540387Z * [new branch] gh/mlazos/16/base -> origin/gh/mlazos/16/base 2025-09-07T06:42:00.8540534Z * [new branch] gh/mlazos/16/head -> origin/gh/mlazos/16/head 2025-09-07T06:42:00.8540687Z * [new branch] gh/mlazos/16/orig -> origin/gh/mlazos/16/orig 2025-09-07T06:42:00.8540829Z * [new branch] gh/mlazos/17/base -> origin/gh/mlazos/17/base 2025-09-07T06:42:00.8540975Z * [new branch] gh/mlazos/17/head -> origin/gh/mlazos/17/head 2025-09-07T06:42:00.8541123Z * [new branch] gh/mlazos/17/orig -> origin/gh/mlazos/17/orig 2025-09-07T06:42:00.8543619Z * [new branch] gh/mlazos/2/base -> origin/gh/mlazos/2/base 2025-09-07T06:42:00.8543812Z * [new branch] gh/mlazos/2/head -> origin/gh/mlazos/2/head 2025-09-07T06:42:00.8543959Z * [new branch] gh/mlazos/2/orig -> origin/gh/mlazos/2/orig 2025-09-07T06:42:00.8544777Z * [new branch] gh/mlazos/3/base -> origin/gh/mlazos/3/base 2025-09-07T06:42:00.8545147Z * [new branch] gh/mlazos/3/head -> origin/gh/mlazos/3/head 2025-09-07T06:42:00.8546228Z * [new branch] gh/mlazos/3/orig -> origin/gh/mlazos/3/orig 2025-09-07T06:42:00.8554735Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-09-07T06:42:00.8557259Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-09-07T06:42:00.8562978Z * [new branch] gh/muchulee8/62/base -> origin/gh/muchulee8/62/base 2025-09-07T06:42:00.8568124Z * [new branch] gh/muchulee8/62/head -> origin/gh/muchulee8/62/head 2025-09-07T06:42:00.8568313Z * [new branch] gh/muchulee8/62/orig -> origin/gh/muchulee8/62/orig 2025-09-07T06:42:00.8568720Z * [new branch] gh/muchulee8/63/base -> origin/gh/muchulee8/63/base 2025-09-07T06:42:00.8568881Z * [new branch] gh/muchulee8/63/head -> origin/gh/muchulee8/63/head 2025-09-07T06:42:00.8569183Z * [new branch] gh/muchulee8/63/orig -> origin/gh/muchulee8/63/orig 2025-09-07T06:42:00.8569332Z * [new branch] gh/muchulee8/64/base -> origin/gh/muchulee8/64/base 2025-09-07T06:42:00.8569475Z * [new branch] gh/muchulee8/64/head -> origin/gh/muchulee8/64/head 2025-09-07T06:42:00.8569642Z * [new branch] gh/muchulee8/64/orig -> origin/gh/muchulee8/64/orig 2025-09-07T06:42:00.8569790Z * [new branch] gh/muchulee8/65/base -> origin/gh/muchulee8/65/base 2025-09-07T06:42:00.8569942Z * [new branch] gh/muchulee8/65/head -> origin/gh/muchulee8/65/head 2025-09-07T06:42:00.8570082Z * [new branch] gh/muchulee8/65/orig -> origin/gh/muchulee8/65/orig 2025-09-07T06:42:00.8570273Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-09-07T06:42:00.8570441Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-09-07T06:42:00.8570606Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-09-07T06:42:00.8570772Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-09-07T06:42:00.8570926Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-09-07T06:42:00.8571089Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-09-07T06:42:00.8571245Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-09-07T06:42:00.8571406Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-09-07T06:42:00.8571560Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-09-07T06:42:00.8571713Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-09-07T06:42:00.8571872Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-09-07T06:42:00.8572041Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-09-07T06:42:00.8572203Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-09-07T06:42:00.8572355Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-09-07T06:42:00.8572574Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-09-07T06:42:00.8572728Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-09-07T06:42:00.8572890Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-09-07T06:42:00.8573086Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-09-07T06:42:00.8574240Z * [new branch] gh/oulgen/35/base -> origin/gh/oulgen/35/base 2025-09-07T06:42:00.8574792Z * [new branch] gh/oulgen/35/head -> origin/gh/oulgen/35/head 2025-09-07T06:42:00.8575322Z * [new branch] gh/oulgen/35/orig -> origin/gh/oulgen/35/orig 2025-09-07T06:42:00.8576443Z * [new branch] gh/oulgen/48/base -> origin/gh/oulgen/48/base 2025-09-07T06:42:00.8576873Z * [new branch] gh/oulgen/48/head -> origin/gh/oulgen/48/head 2025-09-07T06:42:00.8577816Z * [new branch] gh/oulgen/48/orig -> origin/gh/oulgen/48/orig 2025-09-07T06:42:00.8578846Z * [new branch] gh/oulgen/49/base -> origin/gh/oulgen/49/base 2025-09-07T06:42:00.8582460Z * [new branch] gh/oulgen/49/head -> origin/gh/oulgen/49/head 2025-09-07T06:42:00.8582616Z * [new branch] gh/oulgen/49/orig -> origin/gh/oulgen/49/orig 2025-09-07T06:42:00.8582781Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-09-07T06:42:00.8583085Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-09-07T06:42:00.8583243Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-09-07T06:42:00.8583775Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-09-07T06:42:00.8584286Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-09-07T06:42:00.8585198Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-09-07T06:42:00.8586356Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-09-07T06:42:00.8586708Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-09-07T06:42:00.8587818Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-09-07T06:42:00.8589124Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-09-07T06:42:00.8589657Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-09-07T06:42:00.8590042Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-09-07T06:42:00.8591094Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-09-07T06:42:00.8595847Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-09-07T06:42:00.8598924Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-09-07T06:42:00.8599463Z * [new branch] gh/pearu/113/base -> origin/gh/pearu/113/base 2025-09-07T06:42:00.8599656Z * [new branch] gh/pearu/113/head -> origin/gh/pearu/113/head 2025-09-07T06:42:00.8599802Z * [new branch] gh/pearu/113/orig -> origin/gh/pearu/113/orig 2025-09-07T06:42:00.8599943Z * [new branch] gh/pearu/114/base -> origin/gh/pearu/114/base 2025-09-07T06:42:00.8600122Z * [new branch] gh/pearu/114/head -> origin/gh/pearu/114/head 2025-09-07T06:42:00.8600265Z * [new branch] gh/pearu/114/orig -> origin/gh/pearu/114/orig 2025-09-07T06:42:00.8600413Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-09-07T06:42:00.8600552Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-09-07T06:42:00.8600871Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-09-07T06:42:00.8603134Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-09-07T06:42:00.8603526Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-09-07T06:42:00.8603683Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-09-07T06:42:00.8603832Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-09-07T06:42:00.8603995Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-09-07T06:42:00.8607507Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-09-07T06:42:00.8607661Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-09-07T06:42:00.8607895Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-09-07T06:42:00.8608049Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-09-07T06:42:00.8608184Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-09-07T06:42:00.8611304Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-09-07T06:42:00.8611744Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-09-07T06:42:00.8612072Z * [new branch] gh/qqaatw/29/base -> origin/gh/qqaatw/29/base 2025-09-07T06:42:00.8612387Z * [new branch] gh/qqaatw/29/head -> origin/gh/qqaatw/29/head 2025-09-07T06:42:00.8617681Z * [new branch] gh/qqaatw/29/orig -> origin/gh/qqaatw/29/orig 2025-09-07T06:42:00.8621484Z * [new branch] gh/raymo/refresh-script -> origin/gh/raymo/refresh-script 2025-09-07T06:42:00.8621653Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-09-07T06:42:00.8621838Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-09-07T06:42:00.8621985Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-09-07T06:42:00.8622134Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-09-07T06:42:00.8622284Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-09-07T06:42:00.8622424Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-09-07T06:42:00.8622583Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-09-07T06:42:00.8622715Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-09-07T06:42:00.8625474Z * [new branch] gh/rec/156/base -> origin/gh/rec/156/base 2025-09-07T06:42:00.8625819Z * [new branch] gh/rec/156/head -> origin/gh/rec/156/head 2025-09-07T06:42:00.8629899Z * [new branch] gh/rec/156/orig -> origin/gh/rec/156/orig 2025-09-07T06:42:00.8630079Z * [new branch] gh/rec/160/base -> origin/gh/rec/160/base 2025-09-07T06:42:00.8630746Z * [new branch] gh/rec/160/head -> origin/gh/rec/160/head 2025-09-07T06:42:00.8636287Z * [new branch] gh/rec/160/orig -> origin/gh/rec/160/orig 2025-09-07T06:42:00.8641768Z * [new branch] gh/rec/162/base -> origin/gh/rec/162/base 2025-09-07T06:42:00.8642217Z * [new branch] gh/rec/162/head -> origin/gh/rec/162/head 2025-09-07T06:42:00.8642366Z * [new branch] gh/rec/162/orig -> origin/gh/rec/162/orig 2025-09-07T06:42:00.8642500Z * [new branch] gh/rec/163/base -> origin/gh/rec/163/base 2025-09-07T06:42:00.8642638Z * [new branch] gh/rec/163/head -> origin/gh/rec/163/head 2025-09-07T06:42:00.8642994Z * [new branch] gh/rec/163/orig -> origin/gh/rec/163/orig 2025-09-07T06:42:00.8643134Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-09-07T06:42:00.8643264Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-09-07T06:42:00.8643392Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-09-07T06:42:00.8643529Z * [new branch] gh/rec/165/base -> origin/gh/rec/165/base 2025-09-07T06:42:00.8643672Z * [new branch] gh/rec/165/head -> origin/gh/rec/165/head 2025-09-07T06:42:00.8643811Z * [new branch] gh/rec/165/orig -> origin/gh/rec/165/orig 2025-09-07T06:42:00.8643936Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-09-07T06:42:00.8644072Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-09-07T06:42:00.8644197Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-09-07T06:42:00.8644385Z * [new branch] gh/robert-hardwick/1/base -> origin/gh/robert-hardwick/1/base 2025-09-07T06:42:00.8644559Z * [new branch] gh/robert-hardwick/1/head -> origin/gh/robert-hardwick/1/head 2025-09-07T06:42:00.8644727Z * [new branch] gh/robert-hardwick/1/orig -> origin/gh/robert-hardwick/1/orig 2025-09-07T06:42:00.8645327Z * [new branch] gh/robert-hardwick/2/base -> origin/gh/robert-hardwick/2/base 2025-09-07T06:42:00.8646206Z * [new branch] gh/robert-hardwick/2/head -> origin/gh/robert-hardwick/2/head 2025-09-07T06:42:00.8646644Z * [new branch] gh/robert-hardwick/2/orig -> origin/gh/robert-hardwick/2/orig 2025-09-07T06:42:00.8649245Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-09-07T06:42:00.8650602Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-09-07T06:42:00.8650808Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-09-07T06:42:00.8650972Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-09-07T06:42:00.8651133Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-09-07T06:42:00.8654850Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-09-07T06:42:00.8655012Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-09-07T06:42:00.8655501Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-09-07T06:42:00.8655688Z * [new branch] gh/rtimpe/10/base -> origin/gh/rtimpe/10/base 2025-09-07T06:42:00.8656087Z * [new branch] gh/rtimpe/10/head -> origin/gh/rtimpe/10/head 2025-09-07T06:42:00.8656323Z * [new branch] gh/rtimpe/10/orig -> origin/gh/rtimpe/10/orig 2025-09-07T06:42:00.8660285Z * [new branch] gh/rtimpe/11/base -> origin/gh/rtimpe/11/base 2025-09-07T06:42:00.8660442Z * [new branch] gh/rtimpe/11/head -> origin/gh/rtimpe/11/head 2025-09-07T06:42:00.8660591Z * [new branch] gh/rtimpe/11/orig -> origin/gh/rtimpe/11/orig 2025-09-07T06:42:00.8660743Z * [new branch] gh/rtimpe/12/base -> origin/gh/rtimpe/12/base 2025-09-07T06:42:00.8660892Z * [new branch] gh/rtimpe/12/head -> origin/gh/rtimpe/12/head 2025-09-07T06:42:00.8661947Z * [new branch] gh/rtimpe/12/orig -> origin/gh/rtimpe/12/orig 2025-09-07T06:42:00.8662825Z * [new branch] gh/rtimpe/13/base -> origin/gh/rtimpe/13/base 2025-09-07T06:42:00.8663400Z * [new branch] gh/rtimpe/13/head -> origin/gh/rtimpe/13/head 2025-09-07T06:42:00.8664329Z * [new branch] gh/rtimpe/13/orig -> origin/gh/rtimpe/13/orig 2025-09-07T06:42:00.8665996Z * [new branch] gh/rtimpe/14/base -> origin/gh/rtimpe/14/base 2025-09-07T06:42:00.8666323Z * [new branch] gh/rtimpe/14/head -> origin/gh/rtimpe/14/head 2025-09-07T06:42:00.8672610Z * [new branch] gh/rtimpe/14/orig -> origin/gh/rtimpe/14/orig 2025-09-07T06:42:00.8674578Z * [new branch] gh/rtimpe/15/base -> origin/gh/rtimpe/15/base 2025-09-07T06:42:00.8674740Z * [new branch] gh/rtimpe/15/head -> origin/gh/rtimpe/15/head 2025-09-07T06:42:00.8675065Z * [new branch] gh/rtimpe/15/orig -> origin/gh/rtimpe/15/orig 2025-09-07T06:42:00.8675237Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-09-07T06:42:00.8675398Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-09-07T06:42:00.8675547Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-09-07T06:42:00.8675706Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-09-07T06:42:00.8675849Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-09-07T06:42:00.8676008Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-09-07T06:42:00.8676154Z * [new branch] gh/rtimpe/9/base -> origin/gh/rtimpe/9/base 2025-09-07T06:42:00.8678698Z * [new branch] gh/rtimpe/9/head -> origin/gh/rtimpe/9/head 2025-09-07T06:42:00.8679243Z * [new branch] gh/rtimpe/9/orig -> origin/gh/rtimpe/9/orig 2025-09-07T06:42:00.8679443Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-09-07T06:42:00.8679606Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-09-07T06:42:00.8679764Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-09-07T06:42:00.8683194Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-09-07T06:42:00.8683355Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-09-07T06:42:00.8683519Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-09-07T06:42:00.8683669Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-09-07T06:42:00.8683836Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-09-07T06:42:00.8685943Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-09-07T06:42:00.8686157Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-09-07T06:42:00.8687196Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-09-07T06:42:00.8687568Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-09-07T06:42:00.8694078Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-09-07T06:42:00.8694567Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-09-07T06:42:00.8694750Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-09-07T06:42:00.8694963Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-09-07T06:42:00.8695141Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-09-07T06:42:00.8695317Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-09-07T06:42:00.8695478Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-09-07T06:42:00.8698261Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-09-07T06:42:00.8698593Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-09-07T06:42:00.8698752Z * [new branch] gh/sarckk/2/base -> origin/gh/sarckk/2/base 2025-09-07T06:42:00.8698887Z * [new branch] gh/sarckk/2/head -> origin/gh/sarckk/2/head 2025-09-07T06:42:00.8699028Z * [new branch] gh/sarckk/2/orig -> origin/gh/sarckk/2/orig 2025-09-07T06:42:00.8699474Z * [new branch] gh/seemethere/35/base -> origin/gh/seemethere/35/base 2025-09-07T06:42:00.8699644Z * [new branch] gh/seemethere/35/head -> origin/gh/seemethere/35/head 2025-09-07T06:42:00.8700779Z * [new branch] gh/seemethere/35/orig -> origin/gh/seemethere/35/orig 2025-09-07T06:42:00.8702034Z * [new branch] gh/seemethere/37/base -> origin/gh/seemethere/37/base 2025-09-07T06:42:00.8702223Z * [new branch] gh/seemethere/37/head -> origin/gh/seemethere/37/head 2025-09-07T06:42:00.8703352Z * [new branch] gh/seemethere/37/orig -> origin/gh/seemethere/37/orig 2025-09-07T06:42:00.8703843Z * [new branch] gh/seemethere/43/base -> origin/gh/seemethere/43/base 2025-09-07T06:42:00.8704757Z * [new branch] gh/seemethere/43/head -> origin/gh/seemethere/43/head 2025-09-07T06:42:00.8705325Z * [new branch] gh/seemethere/43/orig -> origin/gh/seemethere/43/orig 2025-09-07T06:42:00.8706701Z * [new branch] gh/seemethere/44/base -> origin/gh/seemethere/44/base 2025-09-07T06:42:00.8707035Z * [new branch] gh/seemethere/44/head -> origin/gh/seemethere/44/head 2025-09-07T06:42:00.8708021Z * [new branch] gh/seemethere/44/orig -> origin/gh/seemethere/44/orig 2025-09-07T06:42:00.8709156Z * [new branch] gh/seemethere/48/base -> origin/gh/seemethere/48/base 2025-09-07T06:42:00.8712158Z * [new branch] gh/seemethere/48/head -> origin/gh/seemethere/48/head 2025-09-07T06:42:00.8712821Z * [new branch] gh/seemethere/48/orig -> origin/gh/seemethere/48/orig 2025-09-07T06:42:00.8713011Z * [new branch] gh/seemethere/49/base -> origin/gh/seemethere/49/base 2025-09-07T06:42:00.8713181Z * [new branch] gh/seemethere/49/head -> origin/gh/seemethere/49/head 2025-09-07T06:42:00.8716596Z * [new branch] gh/seemethere/49/orig -> origin/gh/seemethere/49/orig 2025-09-07T06:42:00.8716819Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-09-07T06:42:00.8716995Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-09-07T06:42:00.8717188Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-09-07T06:42:00.8717356Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-09-07T06:42:00.8717542Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-09-07T06:42:00.8717709Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-09-07T06:42:00.8724475Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-09-07T06:42:00.8724671Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-09-07T06:42:00.8725207Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-09-07T06:42:00.8725401Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-09-07T06:42:00.8725566Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-09-07T06:42:00.8725719Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-09-07T06:42:00.8725874Z * [new branch] gh/seemethere/56/base -> origin/gh/seemethere/56/base 2025-09-07T06:42:00.8726252Z * [new branch] gh/seemethere/56/head -> origin/gh/seemethere/56/head 2025-09-07T06:42:00.8726409Z * [new branch] gh/seemethere/56/orig -> origin/gh/seemethere/56/orig 2025-09-07T06:42:00.8726555Z * [new branch] gh/seemethere/57/base -> origin/gh/seemethere/57/base 2025-09-07T06:42:00.8726694Z * [new branch] gh/seemethere/57/head -> origin/gh/seemethere/57/head 2025-09-07T06:42:00.8726850Z * [new branch] gh/seemethere/57/orig -> origin/gh/seemethere/57/orig 2025-09-07T06:42:00.8729976Z * [new branch] gh/seemethere/58/base -> origin/gh/seemethere/58/base 2025-09-07T06:42:00.8730637Z * [new branch] gh/seemethere/58/head -> origin/gh/seemethere/58/head 2025-09-07T06:42:00.8730997Z * [new branch] gh/seemethere/58/orig -> origin/gh/seemethere/58/orig 2025-09-07T06:42:00.8731362Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-09-07T06:42:00.8731573Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-09-07T06:42:00.8731858Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-09-07T06:42:00.8736883Z * [new branch] gh/seemethere/60/base -> origin/gh/seemethere/60/base 2025-09-07T06:42:00.8737086Z * [new branch] gh/seemethere/60/head -> origin/gh/seemethere/60/head 2025-09-07T06:42:00.8737477Z * [new branch] gh/seemethere/60/orig -> origin/gh/seemethere/60/orig 2025-09-07T06:42:00.8737638Z * [new branch] gh/seemethere/61/base -> origin/gh/seemethere/61/base 2025-09-07T06:42:00.8737803Z * [new branch] gh/seemethere/61/head -> origin/gh/seemethere/61/head 2025-09-07T06:42:00.8737952Z * [new branch] gh/seemethere/61/orig -> origin/gh/seemethere/61/orig 2025-09-07T06:42:00.8738124Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-09-07T06:42:00.8739208Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-09-07T06:42:00.8739605Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-09-07T06:42:00.8739809Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-09-07T06:42:00.8739985Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-09-07T06:42:00.8740170Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-09-07T06:42:00.8740343Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-09-07T06:42:00.8740524Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-09-07T06:42:00.8741657Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-09-07T06:42:00.8742823Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-09-07T06:42:00.8743748Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-09-07T06:42:00.8744695Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-09-07T06:42:00.8750516Z * [new branch] gh/shunting314/211/base -> origin/gh/shunting314/211/base 2025-09-07T06:42:00.8750887Z * [new branch] gh/shunting314/211/head -> origin/gh/shunting314/211/head 2025-09-07T06:42:00.8751080Z * [new branch] gh/shunting314/211/orig -> origin/gh/shunting314/211/orig 2025-09-07T06:42:00.8751232Z * [new branch] gh/shunting314/212/base -> origin/gh/shunting314/212/base 2025-09-07T06:42:00.8751389Z * [new branch] gh/shunting314/212/head -> origin/gh/shunting314/212/head 2025-09-07T06:42:00.8751843Z * [new branch] gh/shunting314/212/orig -> origin/gh/shunting314/212/orig 2025-09-07T06:42:00.8752007Z * [new branch] gh/shunting314/213/base -> origin/gh/shunting314/213/base 2025-09-07T06:42:00.8752167Z * [new branch] gh/shunting314/213/head -> origin/gh/shunting314/213/head 2025-09-07T06:42:00.8752318Z * [new branch] gh/shunting314/213/orig -> origin/gh/shunting314/213/orig 2025-09-07T06:42:00.8752479Z * [new branch] gh/shunting314/214/base -> origin/gh/shunting314/214/base 2025-09-07T06:42:00.8752745Z * [new branch] gh/shunting314/214/head -> origin/gh/shunting314/214/head 2025-09-07T06:42:00.8754502Z * [new branch] gh/shunting314/214/orig -> origin/gh/shunting314/214/orig 2025-09-07T06:42:00.8754858Z * [new branch] gh/shunting314/215/base -> origin/gh/shunting314/215/base 2025-09-07T06:42:00.8757964Z * [new branch] gh/shunting314/215/head -> origin/gh/shunting314/215/head 2025-09-07T06:42:00.8758319Z * [new branch] gh/shunting314/215/orig -> origin/gh/shunting314/215/orig 2025-09-07T06:42:00.8758493Z * [new branch] gh/shunting314/216/base -> origin/gh/shunting314/216/base 2025-09-07T06:42:00.8758636Z * [new branch] gh/shunting314/216/head -> origin/gh/shunting314/216/head 2025-09-07T06:42:00.8758878Z * [new branch] gh/shunting314/216/orig -> origin/gh/shunting314/216/orig 2025-09-07T06:42:00.8760393Z * [new branch] gh/shunting314/217/base -> origin/gh/shunting314/217/base 2025-09-07T06:42:00.8760791Z * [new branch] gh/shunting314/217/head -> origin/gh/shunting314/217/head 2025-09-07T06:42:00.8761076Z * [new branch] gh/shunting314/217/orig -> origin/gh/shunting314/217/orig 2025-09-07T06:42:00.8766315Z * [new branch] gh/shunting314/218/base -> origin/gh/shunting314/218/base 2025-09-07T06:42:00.8766564Z * [new branch] gh/shunting314/218/head -> origin/gh/shunting314/218/head 2025-09-07T06:42:00.8766738Z * [new branch] gh/shunting314/218/orig -> origin/gh/shunting314/218/orig 2025-09-07T06:42:00.8766890Z * [new branch] gh/shunting314/219/base -> origin/gh/shunting314/219/base 2025-09-07T06:42:00.8767040Z * [new branch] gh/shunting314/219/head -> origin/gh/shunting314/219/head 2025-09-07T06:42:00.8767200Z * [new branch] gh/shunting314/219/orig -> origin/gh/shunting314/219/orig 2025-09-07T06:42:00.8767530Z * [new branch] gh/shunting314/220/base -> origin/gh/shunting314/220/base 2025-09-07T06:42:00.8769576Z * [new branch] gh/shunting314/220/head -> origin/gh/shunting314/220/head 2025-09-07T06:42:00.8776964Z * [new branch] gh/shunting314/220/orig -> origin/gh/shunting314/220/orig 2025-09-07T06:42:00.8780760Z * [new branch] gh/shunting314/221/base -> origin/gh/shunting314/221/base 2025-09-07T06:42:00.8781132Z * [new branch] gh/shunting314/221/head -> origin/gh/shunting314/221/head 2025-09-07T06:42:00.8781330Z * [new branch] gh/shunting314/221/orig -> origin/gh/shunting314/221/orig 2025-09-07T06:42:00.8781504Z * [new branch] gh/shunting314/222/base -> origin/gh/shunting314/222/base 2025-09-07T06:42:00.8781669Z * [new branch] gh/shunting314/222/head -> origin/gh/shunting314/222/head 2025-09-07T06:42:00.8781970Z * [new branch] gh/shunting314/222/orig -> origin/gh/shunting314/222/orig 2025-09-07T06:42:00.8782780Z * [new branch] gh/shunting314/223/base -> origin/gh/shunting314/223/base 2025-09-07T06:42:00.8783392Z * [new branch] gh/shunting314/223/head -> origin/gh/shunting314/223/head 2025-09-07T06:42:00.8784013Z * [new branch] gh/shunting314/223/orig -> origin/gh/shunting314/223/orig 2025-09-07T06:42:00.8784420Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-09-07T06:42:00.8784573Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-09-07T06:42:00.8784735Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-09-07T06:42:00.8784898Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-09-07T06:42:00.8785059Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-09-07T06:42:00.8785235Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-09-07T06:42:00.8785396Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-09-07T06:42:00.8785546Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-09-07T06:42:00.8785927Z * [new branch] gh/sinhaanhsul/1/base -> origin/gh/sinhaanhsul/1/base 2025-09-07T06:42:00.8786110Z * [new branch] gh/sinhaanhsul/1/head -> origin/gh/sinhaanhsul/1/head 2025-09-07T06:42:00.8786281Z * [new branch] gh/skarjala/17/base -> origin/gh/skarjala/17/base 2025-09-07T06:42:00.8786440Z * [new branch] gh/skarjala/17/head -> origin/gh/skarjala/17/head 2025-09-07T06:42:00.8790657Z * [new branch] gh/skarjala/17/orig -> origin/gh/skarjala/17/orig 2025-09-07T06:42:00.8791886Z * [new branch] gh/skarjala/18/base -> origin/gh/skarjala/18/base 2025-09-07T06:42:00.8792206Z * [new branch] gh/skarjala/18/head -> origin/gh/skarjala/18/head 2025-09-07T06:42:00.8792372Z * [new branch] gh/skarjala/18/orig -> origin/gh/skarjala/18/orig 2025-09-07T06:42:00.8792526Z * [new branch] gh/skarjala/19/base -> origin/gh/skarjala/19/base 2025-09-07T06:42:00.8792679Z * [new branch] gh/skarjala/19/head -> origin/gh/skarjala/19/head 2025-09-07T06:42:00.8792837Z * [new branch] gh/skarjala/19/orig -> origin/gh/skarjala/19/orig 2025-09-07T06:42:00.8795816Z * [new branch] gh/slayton58/1/base -> origin/gh/slayton58/1/base 2025-09-07T06:42:00.8796005Z * [new branch] gh/slayton58/1/head -> origin/gh/slayton58/1/head 2025-09-07T06:42:00.8796146Z * [new branch] gh/slayton58/1/orig -> origin/gh/slayton58/1/orig 2025-09-07T06:42:00.8796307Z * [new branch] gh/slayton58/2/base -> origin/gh/slayton58/2/base 2025-09-07T06:42:00.8796496Z * [new branch] gh/slayton58/2/head -> origin/gh/slayton58/2/head 2025-09-07T06:42:00.8799551Z * [new branch] gh/slayton58/2/orig -> origin/gh/slayton58/2/orig 2025-09-07T06:42:00.8799732Z * [new branch] gh/slayton58/3/base -> origin/gh/slayton58/3/base 2025-09-07T06:42:00.8799890Z * [new branch] gh/slayton58/3/head -> origin/gh/slayton58/3/head 2025-09-07T06:42:00.8800045Z * [new branch] gh/slayton58/3/orig -> origin/gh/slayton58/3/orig 2025-09-07T06:42:00.8800196Z * [new branch] gh/slayton58/4/base -> origin/gh/slayton58/4/base 2025-09-07T06:42:00.8802996Z * [new branch] gh/slayton58/4/head -> origin/gh/slayton58/4/head 2025-09-07T06:42:00.8803184Z * [new branch] gh/slayton58/4/orig -> origin/gh/slayton58/4/orig 2025-09-07T06:42:00.8803351Z * [new branch] gh/slayton58/5/base -> origin/gh/slayton58/5/base 2025-09-07T06:42:00.8803504Z * [new branch] gh/slayton58/5/head -> origin/gh/slayton58/5/head 2025-09-07T06:42:00.8803653Z * [new branch] gh/slayton58/5/orig -> origin/gh/slayton58/5/orig 2025-09-07T06:42:00.8808152Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-09-07T06:42:00.8808467Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-09-07T06:42:00.8808635Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-09-07T06:42:00.8808783Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-09-07T06:42:00.8808930Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-09-07T06:42:00.8809087Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-09-07T06:42:00.8813166Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-09-07T06:42:00.8813332Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-09-07T06:42:00.8813479Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-09-07T06:42:00.8813635Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-09-07T06:42:00.8813788Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-09-07T06:42:00.8817710Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-09-07T06:42:00.8817990Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-09-07T06:42:00.8818154Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-09-07T06:42:00.8818311Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-09-07T06:42:00.8818649Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-09-07T06:42:00.8818807Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-09-07T06:42:00.8818998Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-09-07T06:42:00.8822695Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-09-07T06:42:00.8822906Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-09-07T06:42:00.8823244Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-09-07T06:42:00.8823402Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-09-07T06:42:00.8823995Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-09-07T06:42:00.8824147Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-09-07T06:42:00.8825604Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-09-07T06:42:00.8825937Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-09-07T06:42:00.8831821Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-09-07T06:42:00.8832249Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-09-07T06:42:00.8832497Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-09-07T06:42:00.8832742Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-09-07T06:42:00.8832929Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-09-07T06:42:00.8833173Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-09-07T06:42:00.8833364Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-09-07T06:42:00.8833518Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-09-07T06:42:00.8833670Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-09-07T06:42:00.8833964Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-09-07T06:42:00.8839293Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-09-07T06:42:00.8839504Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-09-07T06:42:00.8839664Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-09-07T06:42:00.8839824Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-09-07T06:42:00.8839979Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-09-07T06:42:00.8840181Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-09-07T06:42:00.8842569Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-09-07T06:42:00.8842739Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-09-07T06:42:00.8842899Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-09-07T06:42:00.8843064Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-09-07T06:42:00.8843222Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-09-07T06:42:00.8843384Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-09-07T06:42:00.8847306Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-09-07T06:42:00.8847835Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-09-07T06:42:00.8848229Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-09-07T06:42:00.8848400Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-09-07T06:42:00.8848558Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-09-07T06:42:00.8848759Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-09-07T06:42:00.8852468Z * [new branch] gh/soulitzer/362/base -> origin/gh/soulitzer/362/base 2025-09-07T06:42:00.8853052Z * [new branch] gh/soulitzer/362/head -> origin/gh/soulitzer/362/head 2025-09-07T06:42:00.8853257Z * [new branch] gh/soulitzer/362/orig -> origin/gh/soulitzer/362/orig 2025-09-07T06:42:00.8853410Z * [new branch] gh/soulitzer/372/base -> origin/gh/soulitzer/372/base 2025-09-07T06:42:00.8853597Z * [new branch] gh/soulitzer/372/head -> origin/gh/soulitzer/372/head 2025-09-07T06:42:00.8853744Z * [new branch] gh/soulitzer/372/orig -> origin/gh/soulitzer/372/orig 2025-09-07T06:42:00.8855630Z * [new branch] gh/soulitzer/373/base -> origin/gh/soulitzer/373/base 2025-09-07T06:42:00.8855817Z * [new branch] gh/soulitzer/373/head -> origin/gh/soulitzer/373/head 2025-09-07T06:42:00.8856093Z * [new branch] gh/soulitzer/373/orig -> origin/gh/soulitzer/373/orig 2025-09-07T06:42:00.8856263Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-09-07T06:42:00.8861287Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-09-07T06:42:00.8861550Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-09-07T06:42:00.8861721Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-09-07T06:42:00.8861877Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-09-07T06:42:00.8862035Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-09-07T06:42:00.8862181Z * [new branch] gh/soulitzer/376/base -> origin/gh/soulitzer/376/base 2025-09-07T06:42:00.8862329Z * [new branch] gh/soulitzer/376/head -> origin/gh/soulitzer/376/head 2025-09-07T06:42:00.8862623Z * [new branch] gh/soulitzer/376/orig -> origin/gh/soulitzer/376/orig 2025-09-07T06:42:00.8862779Z * [new branch] gh/soulitzer/377/base -> origin/gh/soulitzer/377/base 2025-09-07T06:42:00.8862939Z * [new branch] gh/soulitzer/377/head -> origin/gh/soulitzer/377/head 2025-09-07T06:42:00.8863949Z * [new branch] gh/soulitzer/377/orig -> origin/gh/soulitzer/377/orig 2025-09-07T06:42:00.8865035Z * [new branch] gh/soulitzer/378/base -> origin/gh/soulitzer/378/base 2025-09-07T06:42:00.8865565Z * [new branch] gh/soulitzer/378/head -> origin/gh/soulitzer/378/head 2025-09-07T06:42:00.8866692Z * [new branch] gh/soulitzer/378/orig -> origin/gh/soulitzer/378/orig 2025-09-07T06:42:00.8869863Z * [new branch] gh/soulitzer/379/base -> origin/gh/soulitzer/379/base 2025-09-07T06:42:00.8870021Z * [new branch] gh/soulitzer/379/head -> origin/gh/soulitzer/379/head 2025-09-07T06:42:00.8870298Z * [new branch] gh/soulitzer/379/orig -> origin/gh/soulitzer/379/orig 2025-09-07T06:42:00.8870464Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-09-07T06:42:00.8873801Z * [new branch] gh/swolchok/767/base -> origin/gh/swolchok/767/base 2025-09-07T06:42:00.8873954Z * [new branch] gh/swolchok/767/head -> origin/gh/swolchok/767/head 2025-09-07T06:42:00.8874255Z * [new branch] gh/swolchok/767/orig -> origin/gh/swolchok/767/orig 2025-09-07T06:42:00.8874410Z * [new branch] gh/swolchok/768/base -> origin/gh/swolchok/768/base 2025-09-07T06:42:00.8876899Z * [new branch] gh/swolchok/768/head -> origin/gh/swolchok/768/head 2025-09-07T06:42:00.8877062Z * [new branch] gh/swolchok/768/orig -> origin/gh/swolchok/768/orig 2025-09-07T06:42:00.8877224Z * [new branch] gh/swolchok/769/base -> origin/gh/swolchok/769/base 2025-09-07T06:42:00.8877375Z * [new branch] gh/swolchok/769/head -> origin/gh/swolchok/769/head 2025-09-07T06:42:00.8883005Z * [new branch] gh/swolchok/769/orig -> origin/gh/swolchok/769/orig 2025-09-07T06:42:00.8883204Z * [new branch] gh/swolchok/771/base -> origin/gh/swolchok/771/base 2025-09-07T06:42:00.8883352Z * [new branch] gh/swolchok/771/head -> origin/gh/swolchok/771/head 2025-09-07T06:42:00.8883528Z * [new branch] gh/swolchok/771/orig -> origin/gh/swolchok/771/orig 2025-09-07T06:42:00.8883685Z * [new branch] gh/swolchok/772/base -> origin/gh/swolchok/772/base 2025-09-07T06:42:00.8883838Z * [new branch] gh/swolchok/772/head -> origin/gh/swolchok/772/head 2025-09-07T06:42:00.8883995Z * [new branch] gh/swolchok/772/orig -> origin/gh/swolchok/772/orig 2025-09-07T06:42:00.8886333Z * [new branch] gh/swolchok/773/base -> origin/gh/swolchok/773/base 2025-09-07T06:42:00.8886655Z * [new branch] gh/swolchok/773/head -> origin/gh/swolchok/773/head 2025-09-07T06:42:00.8886825Z * [new branch] gh/swolchok/773/orig -> origin/gh/swolchok/773/orig 2025-09-07T06:42:00.8886984Z * [new branch] gh/swolchok/786/base -> origin/gh/swolchok/786/base 2025-09-07T06:42:00.8887149Z * [new branch] gh/swolchok/786/head -> origin/gh/swolchok/786/head 2025-09-07T06:42:00.8887327Z * [new branch] gh/swolchok/786/orig -> origin/gh/swolchok/786/orig 2025-09-07T06:42:00.8887732Z * [new branch] gh/swolchok/787/base -> origin/gh/swolchok/787/base 2025-09-07T06:42:00.8888612Z * [new branch] gh/swolchok/787/head -> origin/gh/swolchok/787/head 2025-09-07T06:42:00.8889037Z * [new branch] gh/swolchok/787/orig -> origin/gh/swolchok/787/orig 2025-09-07T06:42:00.8890357Z * [new branch] gh/swolchok/788/base -> origin/gh/swolchok/788/base 2025-09-07T06:42:00.8890693Z * [new branch] gh/swolchok/788/head -> origin/gh/swolchok/788/head 2025-09-07T06:42:00.8891837Z * [new branch] gh/swolchok/788/orig -> origin/gh/swolchok/788/orig 2025-09-07T06:42:00.8892557Z * [new branch] gh/swolchok/789/base -> origin/gh/swolchok/789/base 2025-09-07T06:42:00.8893207Z * [new branch] gh/swolchok/789/head -> origin/gh/swolchok/789/head 2025-09-07T06:42:00.8894130Z * [new branch] gh/swolchok/789/orig -> origin/gh/swolchok/789/orig 2025-09-07T06:42:00.8895039Z * [new branch] gh/swolchok/790/base -> origin/gh/swolchok/790/base 2025-09-07T06:42:00.8895465Z * [new branch] gh/swolchok/790/head -> origin/gh/swolchok/790/head 2025-09-07T06:42:00.8896543Z * [new branch] gh/swolchok/790/orig -> origin/gh/swolchok/790/orig 2025-09-07T06:42:00.8897514Z * [new branch] gh/swolchok/791/base -> origin/gh/swolchok/791/base 2025-09-07T06:42:00.8898153Z * [new branch] gh/swolchok/791/head -> origin/gh/swolchok/791/head 2025-09-07T06:42:00.8898738Z * [new branch] gh/swolchok/791/orig -> origin/gh/swolchok/791/orig 2025-09-07T06:42:00.8899945Z * [new branch] gh/swolchok/792/base -> origin/gh/swolchok/792/base 2025-09-07T06:42:00.8900325Z * [new branch] gh/swolchok/792/head -> origin/gh/swolchok/792/head 2025-09-07T06:42:00.8901139Z * [new branch] gh/swolchok/792/orig -> origin/gh/swolchok/792/orig 2025-09-07T06:42:00.8902251Z * [new branch] gh/swolchok/793/base -> origin/gh/swolchok/793/base 2025-09-07T06:42:00.8902535Z * [new branch] gh/swolchok/793/head -> origin/gh/swolchok/793/head 2025-09-07T06:42:00.8903555Z * [new branch] gh/swolchok/793/orig -> origin/gh/swolchok/793/orig 2025-09-07T06:42:00.8904816Z * [new branch] gh/swolchok/794/base -> origin/gh/swolchok/794/base 2025-09-07T06:42:00.8905044Z * [new branch] gh/swolchok/794/head -> origin/gh/swolchok/794/head 2025-09-07T06:42:00.8906219Z * [new branch] gh/swolchok/794/orig -> origin/gh/swolchok/794/orig 2025-09-07T06:42:00.8907886Z * [new branch] gh/swolchok/795/base -> origin/gh/swolchok/795/base 2025-09-07T06:42:00.8908414Z * [new branch] gh/swolchok/795/head -> origin/gh/swolchok/795/head 2025-09-07T06:42:00.8909382Z * [new branch] gh/swolchok/795/orig -> origin/gh/swolchok/795/orig 2025-09-07T06:42:00.8911089Z * [new branch] gh/swolchok/796/base -> origin/gh/swolchok/796/base 2025-09-07T06:42:00.8911405Z * [new branch] gh/swolchok/796/head -> origin/gh/swolchok/796/head 2025-09-07T06:42:00.8911709Z * [new branch] gh/swolchok/796/orig -> origin/gh/swolchok/796/orig 2025-09-07T06:42:00.8912898Z * [new branch] gh/swolchok/797/base -> origin/gh/swolchok/797/base 2025-09-07T06:42:00.8913678Z * [new branch] gh/swolchok/797/head -> origin/gh/swolchok/797/head 2025-09-07T06:42:00.8914081Z * [new branch] gh/swolchok/797/orig -> origin/gh/swolchok/797/orig 2025-09-07T06:42:00.8915416Z * [new branch] gh/swolchok/798/base -> origin/gh/swolchok/798/base 2025-09-07T06:42:00.8915870Z * [new branch] gh/swolchok/798/head -> origin/gh/swolchok/798/head 2025-09-07T06:42:00.8917837Z * [new branch] gh/swolchok/798/orig -> origin/gh/swolchok/798/orig 2025-09-07T06:42:00.8918545Z * [new branch] gh/swolchok/799/base -> origin/gh/swolchok/799/base 2025-09-07T06:42:00.8919345Z * [new branch] gh/swolchok/799/head -> origin/gh/swolchok/799/head 2025-09-07T06:42:00.8935719Z * [new branch] gh/swolchok/799/orig -> origin/gh/swolchok/799/orig 2025-09-07T06:42:00.8935902Z * [new branch] gh/swolchok/800/base -> origin/gh/swolchok/800/base 2025-09-07T06:42:00.8936045Z * [new branch] gh/swolchok/800/head -> origin/gh/swolchok/800/head 2025-09-07T06:42:00.8936331Z * [new branch] gh/swolchok/800/orig -> origin/gh/swolchok/800/orig 2025-09-07T06:42:00.8936491Z * [new branch] gh/swolchok/801/base -> origin/gh/swolchok/801/base 2025-09-07T06:42:00.8936659Z * [new branch] gh/swolchok/801/head -> origin/gh/swolchok/801/head 2025-09-07T06:42:00.8936968Z * [new branch] gh/swolchok/801/orig -> origin/gh/swolchok/801/orig 2025-09-07T06:42:00.8937147Z * [new branch] gh/swolchok/802/base -> origin/gh/swolchok/802/base 2025-09-07T06:42:00.8937292Z * [new branch] gh/swolchok/802/head -> origin/gh/swolchok/802/head 2025-09-07T06:42:00.8937452Z * [new branch] gh/swolchok/802/orig -> origin/gh/swolchok/802/orig 2025-09-07T06:42:00.8937594Z * [new branch] gh/swolchok/803/base -> origin/gh/swolchok/803/base 2025-09-07T06:42:00.8937737Z * [new branch] gh/swolchok/803/head -> origin/gh/swolchok/803/head 2025-09-07T06:42:00.8937888Z * [new branch] gh/swolchok/803/orig -> origin/gh/swolchok/803/orig 2025-09-07T06:42:00.8938221Z * [new branch] gh/swolchok/804/base -> origin/gh/swolchok/804/base 2025-09-07T06:42:00.8938384Z * [new branch] gh/swolchok/804/head -> origin/gh/swolchok/804/head 2025-09-07T06:42:00.8938523Z * [new branch] gh/swolchok/804/orig -> origin/gh/swolchok/804/orig 2025-09-07T06:42:00.8938679Z * [new branch] gh/swolchok/805/base -> origin/gh/swolchok/805/base 2025-09-07T06:42:00.8938823Z * [new branch] gh/swolchok/805/head -> origin/gh/swolchok/805/head 2025-09-07T06:42:00.8938964Z * [new branch] gh/swolchok/805/orig -> origin/gh/swolchok/805/orig 2025-09-07T06:42:00.8939112Z * [new branch] gh/swolchok/806/base -> origin/gh/swolchok/806/base 2025-09-07T06:42:00.8939252Z * [new branch] gh/swolchok/806/head -> origin/gh/swolchok/806/head 2025-09-07T06:42:00.8939400Z * [new branch] gh/swolchok/806/orig -> origin/gh/swolchok/806/orig 2025-09-07T06:42:00.8942934Z * [new branch] gh/swolchok/807/base -> origin/gh/swolchok/807/base 2025-09-07T06:42:00.8943123Z * [new branch] gh/swolchok/807/head -> origin/gh/swolchok/807/head 2025-09-07T06:42:00.8943775Z * [new branch] gh/swolchok/807/orig -> origin/gh/swolchok/807/orig 2025-09-07T06:42:00.8943953Z * [new branch] gh/swolchok/808/base -> origin/gh/swolchok/808/base 2025-09-07T06:42:00.8944139Z * [new branch] gh/swolchok/808/head -> origin/gh/swolchok/808/head 2025-09-07T06:42:00.8944295Z * [new branch] gh/swolchok/808/orig -> origin/gh/swolchok/808/orig 2025-09-07T06:42:00.8944463Z * [new branch] gh/swolchok/809/base -> origin/gh/swolchok/809/base 2025-09-07T06:42:00.8947306Z * [new branch] gh/swolchok/809/head -> origin/gh/swolchok/809/head 2025-09-07T06:42:00.8947499Z * [new branch] gh/swolchok/809/orig -> origin/gh/swolchok/809/orig 2025-09-07T06:42:00.8947669Z * [new branch] gh/swolchok/810/base -> origin/gh/swolchok/810/base 2025-09-07T06:42:00.8954963Z * [new branch] gh/swolchok/810/head -> origin/gh/swolchok/810/head 2025-09-07T06:42:00.8959194Z * [new branch] gh/swolchok/810/orig -> origin/gh/swolchok/810/orig 2025-09-07T06:42:00.8961122Z * [new branch] gh/swolchok/811/base -> origin/gh/swolchok/811/base 2025-09-07T06:42:00.8961732Z * [new branch] gh/swolchok/811/head -> origin/gh/swolchok/811/head 2025-09-07T06:42:00.8961887Z * [new branch] gh/swolchok/811/orig -> origin/gh/swolchok/811/orig 2025-09-07T06:42:00.8962037Z * [new branch] gh/swolchok/812/base -> origin/gh/swolchok/812/base 2025-09-07T06:42:00.8962182Z * [new branch] gh/swolchok/812/head -> origin/gh/swolchok/812/head 2025-09-07T06:42:00.8962317Z * [new branch] gh/swolchok/812/orig -> origin/gh/swolchok/812/orig 2025-09-07T06:42:00.8962464Z * [new branch] gh/swolchok/813/base -> origin/gh/swolchok/813/base 2025-09-07T06:42:00.8962606Z * [new branch] gh/swolchok/813/head -> origin/gh/swolchok/813/head 2025-09-07T06:42:00.8962749Z * [new branch] gh/swolchok/813/orig -> origin/gh/swolchok/813/orig 2025-09-07T06:42:00.8962900Z * [new branch] gh/swolchok/814/base -> origin/gh/swolchok/814/base 2025-09-07T06:42:00.8963045Z * [new branch] gh/swolchok/814/head -> origin/gh/swolchok/814/head 2025-09-07T06:42:00.8963191Z * [new branch] gh/swolchok/814/orig -> origin/gh/swolchok/814/orig 2025-09-07T06:42:00.8963335Z * [new branch] gh/swolchok/815/base -> origin/gh/swolchok/815/base 2025-09-07T06:42:00.8963482Z * [new branch] gh/swolchok/815/head -> origin/gh/swolchok/815/head 2025-09-07T06:42:00.8963674Z * [new branch] gh/swolchok/815/orig -> origin/gh/swolchok/815/orig 2025-09-07T06:42:00.8963823Z * [new branch] gh/swolchok/816/base -> origin/gh/swolchok/816/base 2025-09-07T06:42:00.8966359Z * [new branch] gh/swolchok/816/head -> origin/gh/swolchok/816/head 2025-09-07T06:42:00.8966538Z * [new branch] gh/swolchok/816/orig -> origin/gh/swolchok/816/orig 2025-09-07T06:42:00.8966852Z * [new branch] gh/swolchok/817/base -> origin/gh/swolchok/817/base 2025-09-07T06:42:00.8967015Z * [new branch] gh/swolchok/817/head -> origin/gh/swolchok/817/head 2025-09-07T06:42:00.8967312Z * [new branch] gh/swolchok/817/orig -> origin/gh/swolchok/817/orig 2025-09-07T06:42:00.8971855Z * [new branch] gh/swolchok/818/base -> origin/gh/swolchok/818/base 2025-09-07T06:42:00.8972378Z * [new branch] gh/swolchok/818/head -> origin/gh/swolchok/818/head 2025-09-07T06:42:00.8972607Z * [new branch] gh/swolchok/818/orig -> origin/gh/swolchok/818/orig 2025-09-07T06:42:00.8972762Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-09-07T06:42:00.8972918Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-09-07T06:42:00.8973060Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-09-07T06:42:00.8973240Z * [new branch] gh/swolchok/820/base -> origin/gh/swolchok/820/base 2025-09-07T06:42:00.8973389Z * [new branch] gh/swolchok/820/head -> origin/gh/swolchok/820/head 2025-09-07T06:42:00.8973536Z * [new branch] gh/swolchok/820/orig -> origin/gh/swolchok/820/orig 2025-09-07T06:42:00.8973685Z * [new branch] gh/swolchok/821/base -> origin/gh/swolchok/821/base 2025-09-07T06:42:00.8973834Z * [new branch] gh/swolchok/821/head -> origin/gh/swolchok/821/head 2025-09-07T06:42:00.8973991Z * [new branch] gh/swolchok/821/orig -> origin/gh/swolchok/821/orig 2025-09-07T06:42:00.8977362Z * [new branch] gh/swolchok/822/base -> origin/gh/swolchok/822/base 2025-09-07T06:42:00.8977574Z * [new branch] gh/swolchok/822/head -> origin/gh/swolchok/822/head 2025-09-07T06:42:00.8977864Z * [new branch] gh/swolchok/822/orig -> origin/gh/swolchok/822/orig 2025-09-07T06:42:00.8978287Z * [new branch] gh/swolchok/823/base -> origin/gh/swolchok/823/base 2025-09-07T06:42:00.8979460Z * [new branch] gh/swolchok/823/head -> origin/gh/swolchok/823/head 2025-09-07T06:42:00.8981141Z * [new branch] gh/swolchok/823/orig -> origin/gh/swolchok/823/orig 2025-09-07T06:42:00.8981429Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-09-07T06:42:00.8981590Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-09-07T06:42:00.8981755Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-09-07T06:42:00.8981898Z * [new branch] gh/swolchok/825/base -> origin/gh/swolchok/825/base 2025-09-07T06:42:00.8982052Z * [new branch] gh/swolchok/825/head -> origin/gh/swolchok/825/head 2025-09-07T06:42:00.8982206Z * [new branch] gh/swolchok/825/orig -> origin/gh/swolchok/825/orig 2025-09-07T06:42:00.8982380Z * [new branch] gh/swolchok/826/base -> origin/gh/swolchok/826/base 2025-09-07T06:42:00.8982647Z * [new branch] gh/swolchok/826/head -> origin/gh/swolchok/826/head 2025-09-07T06:42:00.8983562Z * [new branch] gh/swolchok/826/orig -> origin/gh/swolchok/826/orig 2025-09-07T06:42:00.8984537Z * [new branch] gh/swolchok/827/base -> origin/gh/swolchok/827/base 2025-09-07T06:42:00.8986398Z * [new branch] gh/swolchok/827/head -> origin/gh/swolchok/827/head 2025-09-07T06:42:00.8986751Z * [new branch] gh/swolchok/827/orig -> origin/gh/swolchok/827/orig 2025-09-07T06:42:00.8987040Z * [new branch] gh/swolchok/828/base -> origin/gh/swolchok/828/base 2025-09-07T06:42:00.8989713Z * [new branch] gh/swolchok/828/head -> origin/gh/swolchok/828/head 2025-09-07T06:42:00.8990127Z * [new branch] gh/swolchok/828/orig -> origin/gh/swolchok/828/orig 2025-09-07T06:42:00.8990469Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-09-07T06:42:00.8990775Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-09-07T06:42:00.8993019Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-09-07T06:42:00.8993203Z * [new branch] gh/swolchok/830/base -> origin/gh/swolchok/830/base 2025-09-07T06:42:00.8993381Z * [new branch] gh/swolchok/830/head -> origin/gh/swolchok/830/head 2025-09-07T06:42:00.8998378Z * [new branch] gh/swolchok/830/orig -> origin/gh/swolchok/830/orig 2025-09-07T06:42:00.9002952Z * [new branch] gh/swolchok/831/base -> origin/gh/swolchok/831/base 2025-09-07T06:42:00.9009105Z * [new branch] gh/swolchok/831/head -> origin/gh/swolchok/831/head 2025-09-07T06:42:00.9009714Z * [new branch] gh/swolchok/831/orig -> origin/gh/swolchok/831/orig 2025-09-07T06:42:00.9009871Z * [new branch] gh/swolchok/832/base -> origin/gh/swolchok/832/base 2025-09-07T06:42:00.9010019Z * [new branch] gh/swolchok/832/head -> origin/gh/swolchok/832/head 2025-09-07T06:42:00.9010175Z * [new branch] gh/swolchok/832/orig -> origin/gh/swolchok/832/orig 2025-09-07T06:42:00.9010333Z * [new branch] gh/syed-ahmed/3/base -> origin/gh/syed-ahmed/3/base 2025-09-07T06:42:00.9010494Z * [new branch] gh/syed-ahmed/3/head -> origin/gh/syed-ahmed/3/head 2025-09-07T06:42:00.9010636Z * [new branch] gh/syed-ahmed/3/orig -> origin/gh/syed-ahmed/3/orig 2025-09-07T06:42:00.9010776Z * [new branch] gh/syed-ahmed/4/base -> origin/gh/syed-ahmed/4/base 2025-09-07T06:42:00.9010926Z * [new branch] gh/syed-ahmed/4/head -> origin/gh/syed-ahmed/4/head 2025-09-07T06:42:00.9011214Z * [new branch] gh/syed-ahmed/4/orig -> origin/gh/syed-ahmed/4/orig 2025-09-07T06:42:00.9011359Z * [new branch] gh/syed-ahmed/5/base -> origin/gh/syed-ahmed/5/base 2025-09-07T06:42:00.9011498Z * [new branch] gh/syed-ahmed/5/head -> origin/gh/syed-ahmed/5/head 2025-09-07T06:42:00.9011650Z * [new branch] gh/syed-ahmed/5/orig -> origin/gh/syed-ahmed/5/orig 2025-09-07T06:42:00.9011807Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-09-07T06:42:00.9011947Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-09-07T06:42:00.9012092Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-09-07T06:42:00.9012237Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-09-07T06:42:00.9012379Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-09-07T06:42:00.9012516Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-09-07T06:42:00.9012658Z * [new branch] gh/tianyu-l/3/base -> origin/gh/tianyu-l/3/base 2025-09-07T06:42:00.9012803Z * [new branch] gh/tianyu-l/3/head -> origin/gh/tianyu-l/3/head 2025-09-07T06:42:00.9013183Z * [new branch] gh/tianyu-l/3/orig -> origin/gh/tianyu-l/3/orig 2025-09-07T06:42:00.9017438Z * [new branch] gh/tianyu-l/4/base -> origin/gh/tianyu-l/4/base 2025-09-07T06:42:00.9022893Z * [new branch] gh/tianyu-l/4/head -> origin/gh/tianyu-l/4/head 2025-09-07T06:42:00.9028248Z * [new branch] gh/tianyu-l/4/orig -> origin/gh/tianyu-l/4/orig 2025-09-07T06:42:00.9033204Z * [new branch] gh/tugsbayasgalan/1/base -> origin/gh/tugsbayasgalan/1/base 2025-09-07T06:42:00.9037183Z * [new branch] gh/tugsbayasgalan/1/head -> origin/gh/tugsbayasgalan/1/head 2025-09-07T06:42:00.9040512Z * [new branch] gh/tugsbayasgalan/1/orig -> origin/gh/tugsbayasgalan/1/orig 2025-09-07T06:42:00.9040731Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-09-07T06:42:00.9041054Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-09-07T06:42:00.9042421Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-09-07T06:42:00.9042647Z * [new branch] gh/tugsbayasgalan/11/base -> origin/gh/tugsbayasgalan/11/base 2025-09-07T06:42:00.9042833Z * [new branch] gh/tugsbayasgalan/11/head -> origin/gh/tugsbayasgalan/11/head 2025-09-07T06:42:00.9043009Z * [new branch] gh/tugsbayasgalan/11/orig -> origin/gh/tugsbayasgalan/11/orig 2025-09-07T06:42:00.9043207Z * [new branch] gh/tugsbayasgalan/12/base -> origin/gh/tugsbayasgalan/12/base 2025-09-07T06:42:00.9043387Z * [new branch] gh/tugsbayasgalan/12/head -> origin/gh/tugsbayasgalan/12/head 2025-09-07T06:42:00.9043561Z * [new branch] gh/tugsbayasgalan/12/orig -> origin/gh/tugsbayasgalan/12/orig 2025-09-07T06:42:00.9043744Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-09-07T06:42:00.9043953Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-09-07T06:42:00.9044136Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-09-07T06:42:00.9044323Z * [new branch] gh/tugsbayasgalan/14/base -> origin/gh/tugsbayasgalan/14/base 2025-09-07T06:42:00.9044564Z * [new branch] gh/tugsbayasgalan/14/head -> origin/gh/tugsbayasgalan/14/head 2025-09-07T06:42:00.9044740Z * [new branch] gh/tugsbayasgalan/14/orig -> origin/gh/tugsbayasgalan/14/orig 2025-09-07T06:42:00.9045151Z * [new branch] gh/tugsbayasgalan/15/base -> origin/gh/tugsbayasgalan/15/base 2025-09-07T06:42:00.9045321Z * [new branch] gh/tugsbayasgalan/15/head -> origin/gh/tugsbayasgalan/15/head 2025-09-07T06:42:00.9045492Z * [new branch] gh/tugsbayasgalan/15/orig -> origin/gh/tugsbayasgalan/15/orig 2025-09-07T06:42:00.9045676Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-09-07T06:42:00.9045841Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-09-07T06:42:00.9046004Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-09-07T06:42:00.9046171Z * [new branch] gh/tugsbayasgalan/3/base -> origin/gh/tugsbayasgalan/3/base 2025-09-07T06:42:00.9046329Z * [new branch] gh/tugsbayasgalan/3/head -> origin/gh/tugsbayasgalan/3/head 2025-09-07T06:42:00.9046499Z * [new branch] gh/tugsbayasgalan/3/orig -> origin/gh/tugsbayasgalan/3/orig 2025-09-07T06:42:00.9046657Z * [new branch] gh/tugsbayasgalan/4/base -> origin/gh/tugsbayasgalan/4/base 2025-09-07T06:42:00.9046822Z * [new branch] gh/tugsbayasgalan/4/head -> origin/gh/tugsbayasgalan/4/head 2025-09-07T06:42:00.9046997Z * [new branch] gh/tugsbayasgalan/4/orig -> origin/gh/tugsbayasgalan/4/orig 2025-09-07T06:42:00.9047152Z * [new branch] gh/tugsbayasgalan/5/base -> origin/gh/tugsbayasgalan/5/base 2025-09-07T06:42:00.9047371Z * [new branch] gh/tugsbayasgalan/5/head -> origin/gh/tugsbayasgalan/5/head 2025-09-07T06:42:00.9047538Z * [new branch] gh/tugsbayasgalan/5/orig -> origin/gh/tugsbayasgalan/5/orig 2025-09-07T06:42:00.9047700Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-09-07T06:42:00.9047856Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-09-07T06:42:00.9048020Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-09-07T06:42:00.9048899Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-09-07T06:42:00.9049089Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-09-07T06:42:00.9049246Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-09-07T06:42:00.9049537Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-09-07T06:42:00.9049732Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-09-07T06:42:00.9049892Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-09-07T06:42:00.9055989Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-09-07T06:42:00.9056343Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-09-07T06:42:00.9056602Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-09-07T06:42:00.9056748Z * [new branch] gh/v0i0/1/base -> origin/gh/v0i0/1/base 2025-09-07T06:42:00.9057011Z * [new branch] gh/v0i0/1/head -> origin/gh/v0i0/1/head 2025-09-07T06:42:00.9057135Z * [new branch] gh/v0i0/1/orig -> origin/gh/v0i0/1/orig 2025-09-07T06:42:00.9057283Z * [new branch] gh/v0i0/4/base -> origin/gh/v0i0/4/base 2025-09-07T06:42:00.9057420Z * [new branch] gh/v0i0/4/head -> origin/gh/v0i0/4/head 2025-09-07T06:42:00.9059481Z * [new branch] gh/v0i0/4/orig -> origin/gh/v0i0/4/orig 2025-09-07T06:42:00.9059726Z * [new branch] gh/v0i0/6/base -> origin/gh/v0i0/6/base 2025-09-07T06:42:00.9059977Z * [new branch] gh/v0i0/6/head -> origin/gh/v0i0/6/head 2025-09-07T06:42:00.9060260Z * [new branch] gh/v0i0/6/orig -> origin/gh/v0i0/6/orig 2025-09-07T06:42:00.9060515Z * [new branch] gh/v0i0/7/base -> origin/gh/v0i0/7/base 2025-09-07T06:42:00.9060748Z * [new branch] gh/v0i0/7/head -> origin/gh/v0i0/7/head 2025-09-07T06:42:00.9060878Z * [new branch] gh/v0i0/7/orig -> origin/gh/v0i0/7/orig 2025-09-07T06:42:00.9061013Z * [new branch] gh/v0i0/8/base -> origin/gh/v0i0/8/base 2025-09-07T06:42:00.9061393Z * [new branch] gh/v0i0/8/head -> origin/gh/v0i0/8/head 2025-09-07T06:42:00.9062988Z * [new branch] gh/v0i0/8/orig -> origin/gh/v0i0/8/orig 2025-09-07T06:42:00.9063158Z * [new branch] gh/v0i0/9/base -> origin/gh/v0i0/9/base 2025-09-07T06:42:00.9064009Z * [new branch] gh/v0i0/9/head -> origin/gh/v0i0/9/head 2025-09-07T06:42:00.9064384Z * [new branch] gh/v0i0/9/orig -> origin/gh/v0i0/9/orig 2025-09-07T06:42:00.9068669Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-09-07T06:42:00.9069298Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-09-07T06:42:00.9069443Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-09-07T06:42:00.9069637Z * [new branch] gh/vkuzo/4/base -> origin/gh/vkuzo/4/base 2025-09-07T06:42:00.9069930Z * [new branch] gh/vkuzo/4/head -> origin/gh/vkuzo/4/head 2025-09-07T06:42:00.9070266Z * [new branch] gh/vkuzo/4/orig -> origin/gh/vkuzo/4/orig 2025-09-07T06:42:00.9076633Z * [new branch] gh/vkuzo/5/base -> origin/gh/vkuzo/5/base 2025-09-07T06:42:00.9076936Z * [new branch] gh/vkuzo/5/head -> origin/gh/vkuzo/5/head 2025-09-07T06:42:00.9077285Z * [new branch] gh/vkuzo/5/orig -> origin/gh/vkuzo/5/orig 2025-09-07T06:42:00.9077428Z * [new branch] gh/vkuzo/6/base -> origin/gh/vkuzo/6/base 2025-09-07T06:42:00.9077573Z * [new branch] gh/vkuzo/6/head -> origin/gh/vkuzo/6/head 2025-09-07T06:42:00.9077718Z * [new branch] gh/vkuzo/6/orig -> origin/gh/vkuzo/6/orig 2025-09-07T06:42:00.9077855Z * [new branch] gh/vkuzo/7/base -> origin/gh/vkuzo/7/base 2025-09-07T06:42:00.9078004Z * [new branch] gh/vkuzo/7/head -> origin/gh/vkuzo/7/head 2025-09-07T06:42:00.9078139Z * [new branch] gh/vkuzo/7/orig -> origin/gh/vkuzo/7/orig 2025-09-07T06:42:00.9079504Z * [new branch] gh/wconstab/419/base -> origin/gh/wconstab/419/base 2025-09-07T06:42:00.9082785Z * [new branch] gh/wconstab/419/head -> origin/gh/wconstab/419/head 2025-09-07T06:42:00.9083320Z * [new branch] gh/wconstab/419/orig -> origin/gh/wconstab/419/orig 2025-09-07T06:42:00.9083514Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-09-07T06:42:00.9083663Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-09-07T06:42:00.9083818Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-09-07T06:42:00.9083964Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-09-07T06:42:00.9087221Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-09-07T06:42:00.9087478Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-09-07T06:42:00.9087639Z * [new branch] gh/wconstab/438/base -> origin/gh/wconstab/438/base 2025-09-07T06:42:00.9087790Z * [new branch] gh/wconstab/438/head -> origin/gh/wconstab/438/head 2025-09-07T06:42:00.9088096Z * [new branch] gh/wconstab/438/orig -> origin/gh/wconstab/438/orig 2025-09-07T06:42:00.9088245Z * [new branch] gh/wconstab/440/base -> origin/gh/wconstab/440/base 2025-09-07T06:42:00.9090943Z * [new branch] gh/wconstab/440/head -> origin/gh/wconstab/440/head 2025-09-07T06:42:00.9091629Z * [new branch] gh/wconstab/440/orig -> origin/gh/wconstab/440/orig 2025-09-07T06:42:00.9092153Z * [new branch] gh/wconstab/441/base -> origin/gh/wconstab/441/base 2025-09-07T06:42:00.9092607Z * [new branch] gh/wconstab/441/head -> origin/gh/wconstab/441/head 2025-09-07T06:42:00.9092891Z * [new branch] gh/wconstab/441/orig -> origin/gh/wconstab/441/orig 2025-09-07T06:42:00.9093183Z * [new branch] gh/wconstab/442/base -> origin/gh/wconstab/442/base 2025-09-07T06:42:00.9093631Z * [new branch] gh/wconstab/442/head -> origin/gh/wconstab/442/head 2025-09-07T06:42:00.9094588Z * [new branch] gh/wconstab/442/orig -> origin/gh/wconstab/442/orig 2025-09-07T06:42:00.9095917Z * [new branch] gh/wconstab/443/base -> origin/gh/wconstab/443/base 2025-09-07T06:42:00.9096102Z * [new branch] gh/wconstab/443/head -> origin/gh/wconstab/443/head 2025-09-07T06:42:00.9097350Z * [new branch] gh/wconstab/443/orig -> origin/gh/wconstab/443/orig 2025-09-07T06:42:00.9097649Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-09-07T06:42:00.9098480Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-09-07T06:42:00.9098974Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-09-07T06:42:00.9100125Z * [new branch] gh/wconstab/445/base -> origin/gh/wconstab/445/base 2025-09-07T06:42:00.9100693Z * [new branch] gh/wconstab/445/head -> origin/gh/wconstab/445/head 2025-09-07T06:42:00.9101618Z * [new branch] gh/wconstab/445/orig -> origin/gh/wconstab/445/orig 2025-09-07T06:42:00.9102932Z * [new branch] gh/wconstab/446/base -> origin/gh/wconstab/446/base 2025-09-07T06:42:00.9103827Z * [new branch] gh/wconstab/446/head -> origin/gh/wconstab/446/head 2025-09-07T06:42:00.9104885Z * [new branch] gh/wconstab/446/orig -> origin/gh/wconstab/446/orig 2025-09-07T06:42:00.9106214Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-09-07T06:42:00.9106497Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-09-07T06:42:00.9107802Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-09-07T06:42:00.9109117Z * [new branch] gh/weifengpy/27/base -> origin/gh/weifengpy/27/base 2025-09-07T06:42:00.9109409Z * [new branch] gh/weifengpy/27/head -> origin/gh/weifengpy/27/head 2025-09-07T06:42:00.9110576Z * [new branch] gh/weifengpy/27/orig -> origin/gh/weifengpy/27/orig 2025-09-07T06:42:00.9111480Z * [new branch] gh/weifengpy/30/base -> origin/gh/weifengpy/30/base 2025-09-07T06:42:00.9111933Z * [new branch] gh/weifengpy/30/head -> origin/gh/weifengpy/30/head 2025-09-07T06:42:00.9112811Z * [new branch] gh/weifengpy/30/orig -> origin/gh/weifengpy/30/orig 2025-09-07T06:42:00.9114162Z * [new branch] gh/williamwen42/196/base -> origin/gh/williamwen42/196/base 2025-09-07T06:42:00.9114696Z * [new branch] gh/williamwen42/196/head -> origin/gh/williamwen42/196/head 2025-09-07T06:42:00.9115931Z * [new branch] gh/williamwen42/196/orig -> origin/gh/williamwen42/196/orig 2025-09-07T06:42:00.9116939Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-09-07T06:42:00.9117409Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-09-07T06:42:00.9118350Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-09-07T06:42:00.9119308Z * [new branch] gh/williamwen42/258/base -> origin/gh/williamwen42/258/base 2025-09-07T06:42:00.9119947Z * [new branch] gh/williamwen42/258/head -> origin/gh/williamwen42/258/head 2025-09-07T06:42:00.9121345Z * [new branch] gh/williamwen42/258/orig -> origin/gh/williamwen42/258/orig 2025-09-07T06:42:00.9121575Z * [new branch] gh/williamwen42/266/base -> origin/gh/williamwen42/266/base 2025-09-07T06:42:00.9122739Z * [new branch] gh/williamwen42/266/head -> origin/gh/williamwen42/266/head 2025-09-07T06:42:00.9123045Z * [new branch] gh/williamwen42/266/orig -> origin/gh/williamwen42/266/orig 2025-09-07T06:42:00.9124358Z * [new branch] gh/williamwen42/267/base -> origin/gh/williamwen42/267/base 2025-09-07T06:42:00.9124923Z * [new branch] gh/williamwen42/267/head -> origin/gh/williamwen42/267/head 2025-09-07T06:42:00.9125828Z * [new branch] gh/williamwen42/267/orig -> origin/gh/williamwen42/267/orig 2025-09-07T06:42:00.9126810Z * [new branch] gh/williamwen42/270/base -> origin/gh/williamwen42/270/base 2025-09-07T06:42:00.9127352Z * [new branch] gh/williamwen42/270/head -> origin/gh/williamwen42/270/head 2025-09-07T06:42:00.9128341Z * [new branch] gh/williamwen42/270/orig -> origin/gh/williamwen42/270/orig 2025-09-07T06:42:00.9129310Z * [new branch] gh/williamwen42/271/base -> origin/gh/williamwen42/271/base 2025-09-07T06:42:00.9129605Z * [new branch] gh/williamwen42/271/head -> origin/gh/williamwen42/271/head 2025-09-07T06:42:00.9130767Z * [new branch] gh/williamwen42/271/orig -> origin/gh/williamwen42/271/orig 2025-09-07T06:42:00.9131701Z * [new branch] gh/williamwen42/272/base -> origin/gh/williamwen42/272/base 2025-09-07T06:42:00.9131995Z * [new branch] gh/williamwen42/272/head -> origin/gh/williamwen42/272/head 2025-09-07T06:42:00.9133221Z * [new branch] gh/williamwen42/272/orig -> origin/gh/williamwen42/272/orig 2025-09-07T06:42:00.9134136Z * [new branch] gh/williamwen42/274/base -> origin/gh/williamwen42/274/base 2025-09-07T06:42:00.9134535Z * [new branch] gh/williamwen42/274/head -> origin/gh/williamwen42/274/head 2025-09-07T06:42:00.9135542Z * [new branch] gh/williamwen42/274/orig -> origin/gh/williamwen42/274/orig 2025-09-07T06:42:00.9136364Z * [new branch] gh/williamwen42/275/base -> origin/gh/williamwen42/275/base 2025-09-07T06:42:00.9136920Z * [new branch] gh/williamwen42/275/head -> origin/gh/williamwen42/275/head 2025-09-07T06:42:00.9138081Z * [new branch] gh/williamwen42/276/base -> origin/gh/williamwen42/276/base 2025-09-07T06:42:00.9138360Z * [new branch] gh/williamwen42/276/head -> origin/gh/williamwen42/276/head 2025-09-07T06:42:00.9139351Z * [new branch] gh/williamwen42/276/orig -> origin/gh/williamwen42/276/orig 2025-09-07T06:42:00.9140426Z * [new branch] gh/williamwen42/277/base -> origin/gh/williamwen42/277/base 2025-09-07T06:42:00.9140870Z * [new branch] gh/williamwen42/277/head -> origin/gh/williamwen42/277/head 2025-09-07T06:42:00.9141778Z * [new branch] gh/williamwen42/277/orig -> origin/gh/williamwen42/277/orig 2025-09-07T06:42:00.9142942Z * [new branch] gh/williamwen42/278/base -> origin/gh/williamwen42/278/base 2025-09-07T06:42:00.9143160Z * [new branch] gh/williamwen42/278/head -> origin/gh/williamwen42/278/head 2025-09-07T06:42:00.9144277Z * [new branch] gh/williamwen42/278/orig -> origin/gh/williamwen42/278/orig 2025-09-07T06:42:00.9145401Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-09-07T06:42:00.9145590Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-09-07T06:42:00.9146837Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-09-07T06:42:00.9147782Z * [new branch] gh/williamwen42/280/base -> origin/gh/williamwen42/280/base 2025-09-07T06:42:00.9148344Z * [new branch] gh/williamwen42/280/head -> origin/gh/williamwen42/280/head 2025-09-07T06:42:00.9149252Z * [new branch] gh/williamwen42/280/orig -> origin/gh/williamwen42/280/orig 2025-09-07T06:42:00.9150235Z * [new branch] gh/williamwen42/281/base -> origin/gh/williamwen42/281/base 2025-09-07T06:42:00.9150664Z * [new branch] gh/williamwen42/281/head -> origin/gh/williamwen42/281/head 2025-09-07T06:42:00.9151540Z * [new branch] gh/williamwen42/281/orig -> origin/gh/williamwen42/281/orig 2025-09-07T06:42:00.9152884Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-09-07T06:42:00.9153366Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-09-07T06:42:00.9154374Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-09-07T06:42:00.9155518Z * [new branch] gh/williamwen42/283/base -> origin/gh/williamwen42/283/base 2025-09-07T06:42:00.9156375Z * [new branch] gh/williamwen42/283/head -> origin/gh/williamwen42/283/head 2025-09-07T06:42:00.9156899Z * [new branch] gh/williamwen42/283/orig -> origin/gh/williamwen42/283/orig 2025-09-07T06:42:00.9158350Z * [new branch] gh/williamwen42/284/base -> origin/gh/williamwen42/284/base 2025-09-07T06:42:00.9158735Z * [new branch] gh/williamwen42/284/head -> origin/gh/williamwen42/284/head 2025-09-07T06:42:00.9159801Z * [new branch] gh/williamwen42/284/orig -> origin/gh/williamwen42/284/orig 2025-09-07T06:42:00.9160810Z * [new branch] gh/williamwen42/285/base -> origin/gh/williamwen42/285/base 2025-09-07T06:42:00.9161157Z * [new branch] gh/williamwen42/285/head -> origin/gh/williamwen42/285/head 2025-09-07T06:42:00.9162099Z * [new branch] gh/williamwen42/285/orig -> origin/gh/williamwen42/285/orig 2025-09-07T06:42:00.9163188Z * [new branch] gh/williamwen42/286/base -> origin/gh/williamwen42/286/base 2025-09-07T06:42:00.9163375Z * [new branch] gh/williamwen42/286/head -> origin/gh/williamwen42/286/head 2025-09-07T06:42:00.9164313Z * [new branch] gh/williamwen42/286/orig -> origin/gh/williamwen42/286/orig 2025-09-07T06:42:00.9165537Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-09-07T06:42:00.9167424Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-09-07T06:42:00.9167634Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-09-07T06:42:00.9167797Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-09-07T06:42:00.9168946Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-09-07T06:42:00.9169118Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-09-07T06:42:00.9170521Z * [new branch] gh/williamwen42/289/base -> origin/gh/williamwen42/289/base 2025-09-07T06:42:00.9171011Z * [new branch] gh/williamwen42/289/head -> origin/gh/williamwen42/289/head 2025-09-07T06:42:00.9172032Z * [new branch] gh/williamwen42/289/orig -> origin/gh/williamwen42/289/orig 2025-09-07T06:42:00.9173355Z * [new branch] gh/wychi/1/base -> origin/gh/wychi/1/base 2025-09-07T06:42:00.9173949Z * [new branch] gh/wychi/1/head -> origin/gh/wychi/1/head 2025-09-07T06:42:00.9174927Z * [new branch] gh/wychi/1/orig -> origin/gh/wychi/1/orig 2025-09-07T06:42:00.9176747Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-09-07T06:42:00.9176921Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-09-07T06:42:00.9181483Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-09-07T06:42:00.9181675Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-09-07T06:42:00.9181823Z * [new branch] gh/xmfan/18/base -> origin/gh/xmfan/18/base 2025-09-07T06:42:00.9181974Z * [new branch] gh/xmfan/18/head -> origin/gh/xmfan/18/head 2025-09-07T06:42:00.9182163Z * [new branch] gh/xmfan/229/base -> origin/gh/xmfan/229/base 2025-09-07T06:42:00.9182320Z * [new branch] gh/xmfan/229/head -> origin/gh/xmfan/229/head 2025-09-07T06:42:00.9182903Z * [new branch] gh/xmfan/229/orig -> origin/gh/xmfan/229/orig 2025-09-07T06:42:00.9184002Z * [new branch] gh/xmfan/237/base -> origin/gh/xmfan/237/base 2025-09-07T06:42:00.9184446Z * [new branch] gh/xmfan/237/head -> origin/gh/xmfan/237/head 2025-09-07T06:42:00.9185492Z * [new branch] gh/xmfan/237/orig -> origin/gh/xmfan/237/orig 2025-09-07T06:42:00.9186083Z * [new branch] gh/xmfan/244/base -> origin/gh/xmfan/244/base 2025-09-07T06:42:00.9187255Z * [new branch] gh/xmfan/244/head -> origin/gh/xmfan/244/head 2025-09-07T06:42:00.9187838Z * [new branch] gh/xmfan/244/orig -> origin/gh/xmfan/244/orig 2025-09-07T06:42:00.9188979Z * [new branch] gh/xmfan/246/base -> origin/gh/xmfan/246/base 2025-09-07T06:42:00.9189266Z * [new branch] gh/xmfan/246/head -> origin/gh/xmfan/246/head 2025-09-07T06:42:00.9190308Z * [new branch] gh/xmfan/246/orig -> origin/gh/xmfan/246/orig 2025-09-07T06:42:00.9191183Z * [new branch] gh/xmfan/253/base -> origin/gh/xmfan/253/base 2025-09-07T06:42:00.9191759Z * [new branch] gh/xmfan/253/head -> origin/gh/xmfan/253/head 2025-09-07T06:42:00.9192627Z * [new branch] gh/xmfan/253/orig -> origin/gh/xmfan/253/orig 2025-09-07T06:42:00.9195009Z * [new branch] gh/xmfan/254/base -> origin/gh/xmfan/254/base 2025-09-07T06:42:00.9195181Z * [new branch] gh/xmfan/254/head -> origin/gh/xmfan/254/head 2025-09-07T06:42:00.9195340Z * [new branch] gh/xmfan/254/orig -> origin/gh/xmfan/254/orig 2025-09-07T06:42:00.9195486Z * [new branch] gh/xmfan/260/base -> origin/gh/xmfan/260/base 2025-09-07T06:42:00.9196718Z * [new branch] gh/xmfan/260/head -> origin/gh/xmfan/260/head 2025-09-07T06:42:00.9197040Z * [new branch] gh/xmfan/260/orig -> origin/gh/xmfan/260/orig 2025-09-07T06:42:00.9199236Z * [new branch] gh/xmfan/262/base -> origin/gh/xmfan/262/base 2025-09-07T06:42:00.9199574Z * [new branch] gh/xmfan/262/head -> origin/gh/xmfan/262/head 2025-09-07T06:42:00.9199763Z * [new branch] gh/xmfan/262/orig -> origin/gh/xmfan/262/orig 2025-09-07T06:42:00.9200032Z * [new branch] gh/xmfan/263/base -> origin/gh/xmfan/263/base 2025-09-07T06:42:00.9201835Z * [new branch] gh/xmfan/263/head -> origin/gh/xmfan/263/head 2025-09-07T06:42:00.9202023Z * [new branch] gh/xmfan/263/orig -> origin/gh/xmfan/263/orig 2025-09-07T06:42:00.9205407Z * [new branch] gh/xmfan/264/base -> origin/gh/xmfan/264/base 2025-09-07T06:42:00.9205884Z * [new branch] gh/xmfan/264/head -> origin/gh/xmfan/264/head 2025-09-07T06:42:00.9206026Z * [new branch] gh/xmfan/264/orig -> origin/gh/xmfan/264/orig 2025-09-07T06:42:00.9206164Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-09-07T06:42:00.9206321Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-09-07T06:42:00.9206467Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-09-07T06:42:00.9211657Z * [new branch] gh/xmfan/276/base -> origin/gh/xmfan/276/base 2025-09-07T06:42:00.9211846Z * [new branch] gh/xmfan/276/head -> origin/gh/xmfan/276/head 2025-09-07T06:42:00.9211999Z * [new branch] gh/xmfan/276/orig -> origin/gh/xmfan/276/orig 2025-09-07T06:42:00.9212150Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-09-07T06:42:00.9212574Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-09-07T06:42:00.9212711Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-09-07T06:42:00.9216858Z * [new branch] gh/xmfan/278/base -> origin/gh/xmfan/278/base 2025-09-07T06:42:00.9217309Z * [new branch] gh/xmfan/278/head -> origin/gh/xmfan/278/head 2025-09-07T06:42:00.9217675Z * [new branch] gh/xmfan/278/orig -> origin/gh/xmfan/278/orig 2025-09-07T06:42:00.9217830Z * [new branch] gh/xmfan/279/base -> origin/gh/xmfan/279/base 2025-09-07T06:42:00.9217969Z * [new branch] gh/xmfan/279/head -> origin/gh/xmfan/279/head 2025-09-07T06:42:00.9218187Z * [new branch] gh/xmfan/279/orig -> origin/gh/xmfan/279/orig 2025-09-07T06:42:00.9222018Z * [new branch] gh/xmfan/280/base -> origin/gh/xmfan/280/base 2025-09-07T06:42:00.9222678Z * [new branch] gh/xmfan/280/head -> origin/gh/xmfan/280/head 2025-09-07T06:42:00.9226257Z * [new branch] gh/xmfan/280/orig -> origin/gh/xmfan/280/orig 2025-09-07T06:42:00.9226432Z * [new branch] gh/xmfan/281/base -> origin/gh/xmfan/281/base 2025-09-07T06:42:00.9226591Z * [new branch] gh/xmfan/281/head -> origin/gh/xmfan/281/head 2025-09-07T06:42:00.9226773Z * [new branch] gh/xmfan/281/orig -> origin/gh/xmfan/281/orig 2025-09-07T06:42:00.9226912Z * [new branch] gh/xmfan/282/base -> origin/gh/xmfan/282/base 2025-09-07T06:42:00.9227078Z * [new branch] gh/xmfan/282/head -> origin/gh/xmfan/282/head 2025-09-07T06:42:00.9227234Z * [new branch] gh/xmfan/283/base -> origin/gh/xmfan/283/base 2025-09-07T06:42:00.9227387Z * [new branch] gh/xmfan/283/head -> origin/gh/xmfan/283/head 2025-09-07T06:42:00.9227530Z * [new branch] gh/xmfan/283/orig -> origin/gh/xmfan/283/orig 2025-09-07T06:42:00.9227717Z * [new branch] gh/xuanzhang816/14/base -> origin/gh/xuanzhang816/14/base 2025-09-07T06:42:00.9234032Z * [new branch] gh/xuanzhang816/14/head -> origin/gh/xuanzhang816/14/head 2025-09-07T06:42:00.9238342Z * [new branch] gh/xuanzhang816/14/orig -> origin/gh/xuanzhang816/14/orig 2025-09-07T06:42:00.9245243Z * [new branch] gh/xuanzhang816/19/base -> origin/gh/xuanzhang816/19/base 2025-09-07T06:42:00.9251283Z * [new branch] gh/xuanzhang816/19/head -> origin/gh/xuanzhang816/19/head 2025-09-07T06:42:00.9253545Z * [new branch] gh/xuanzhang816/19/orig -> origin/gh/xuanzhang816/19/orig 2025-09-07T06:42:00.9253906Z * [new branch] gh/xuanzhang816/22/base -> origin/gh/xuanzhang816/22/base 2025-09-07T06:42:00.9254396Z * [new branch] gh/xuanzhang816/22/head -> origin/gh/xuanzhang816/22/head 2025-09-07T06:42:00.9260647Z * [new branch] gh/xuanzhang816/22/orig -> origin/gh/xuanzhang816/22/orig 2025-09-07T06:42:00.9263059Z * [new branch] gh/xuanzhang816/23/base -> origin/gh/xuanzhang816/23/base 2025-09-07T06:42:00.9263262Z * [new branch] gh/xuanzhang816/23/head -> origin/gh/xuanzhang816/23/head 2025-09-07T06:42:00.9263431Z * [new branch] gh/xuanzhang816/23/orig -> origin/gh/xuanzhang816/23/orig 2025-09-07T06:42:00.9263600Z * [new branch] gh/xuanzhang816/24/base -> origin/gh/xuanzhang816/24/base 2025-09-07T06:42:00.9263754Z * [new branch] gh/xuanzhang816/24/head -> origin/gh/xuanzhang816/24/head 2025-09-07T06:42:00.9263920Z * [new branch] gh/xuanzhang816/24/orig -> origin/gh/xuanzhang816/24/orig 2025-09-07T06:42:00.9264078Z * [new branch] gh/xuanzhang816/25/base -> origin/gh/xuanzhang816/25/base 2025-09-07T06:42:00.9264245Z * [new branch] gh/xuanzhang816/25/head -> origin/gh/xuanzhang816/25/head 2025-09-07T06:42:00.9264394Z * [new branch] gh/xuanzhang816/25/orig -> origin/gh/xuanzhang816/25/orig 2025-09-07T06:42:00.9264558Z * [new branch] gh/xuanzhang816/26/base -> origin/gh/xuanzhang816/26/base 2025-09-07T06:42:00.9264705Z * [new branch] gh/xuanzhang816/26/head -> origin/gh/xuanzhang816/26/head 2025-09-07T06:42:00.9265050Z * [new branch] gh/xuanzhang816/26/orig -> origin/gh/xuanzhang816/26/orig 2025-09-07T06:42:00.9265228Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-09-07T06:42:00.9265375Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-09-07T06:42:00.9265520Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-09-07T06:42:00.9265899Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-09-07T06:42:00.9266068Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-09-07T06:42:00.9266216Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-09-07T06:42:00.9266357Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-09-07T06:42:00.9266504Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-09-07T06:42:00.9266650Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-09-07T06:42:00.9266815Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-09-07T06:42:00.9266956Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-09-07T06:42:00.9267105Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-09-07T06:42:00.9267250Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-09-07T06:42:00.9267402Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-09-07T06:42:00.9267549Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-09-07T06:42:00.9267686Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-09-07T06:42:00.9267832Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-09-07T06:42:00.9267972Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-09-07T06:42:00.9268110Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-09-07T06:42:00.9268257Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-09-07T06:42:00.9268395Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-09-07T06:42:00.9268589Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-09-07T06:42:00.9268726Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-09-07T06:42:00.9268873Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-09-07T06:42:00.9269009Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-09-07T06:42:00.9269154Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-09-07T06:42:00.9269293Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-09-07T06:42:00.9269422Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-09-07T06:42:00.9269559Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-09-07T06:42:00.9269693Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-09-07T06:42:00.9269835Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-09-07T06:42:00.9269972Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-09-07T06:42:00.9270257Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-09-07T06:42:00.9276950Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-09-07T06:42:00.9281943Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-09-07T06:42:00.9283029Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-09-07T06:42:00.9283350Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-09-07T06:42:00.9283528Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-09-07T06:42:00.9283692Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-09-07T06:42:00.9283834Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-09-07T06:42:00.9283978Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-09-07T06:42:00.9284115Z * [new branch] gh/yanbing-j/36/base -> origin/gh/yanbing-j/36/base 2025-09-07T06:42:00.9284258Z * [new branch] gh/yanbing-j/36/head -> origin/gh/yanbing-j/36/head 2025-09-07T06:42:00.9284400Z * [new branch] gh/yanbing-j/36/orig -> origin/gh/yanbing-j/36/orig 2025-09-07T06:42:00.9284532Z * [new branch] gh/yanbing-j/37/base -> origin/gh/yanbing-j/37/base 2025-09-07T06:42:00.9284667Z * [new branch] gh/yanbing-j/37/head -> origin/gh/yanbing-j/37/head 2025-09-07T06:42:00.9284796Z * [new branch] gh/yanbing-j/37/orig -> origin/gh/yanbing-j/37/orig 2025-09-07T06:42:00.9284943Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-09-07T06:42:00.9285073Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-09-07T06:42:00.9285204Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-09-07T06:42:00.9285344Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-09-07T06:42:00.9286702Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-09-07T06:42:00.9287405Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-09-07T06:42:00.9287750Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-09-07T06:42:00.9287916Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-09-07T06:42:00.9288618Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-09-07T06:42:00.9289093Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-09-07T06:42:00.9289265Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-09-07T06:42:00.9289412Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-09-07T06:42:00.9289693Z * [new branch] gh/yangw-dev/16/base -> origin/gh/yangw-dev/16/base 2025-09-07T06:42:00.9290009Z * [new branch] gh/yangw-dev/16/head -> origin/gh/yangw-dev/16/head 2025-09-07T06:42:00.9290175Z * [new branch] gh/yangw-dev/16/orig -> origin/gh/yangw-dev/16/orig 2025-09-07T06:42:00.9290314Z * [new branch] gh/yangw-dev/17/base -> origin/gh/yangw-dev/17/base 2025-09-07T06:42:00.9290466Z * [new branch] gh/yangw-dev/17/head -> origin/gh/yangw-dev/17/head 2025-09-07T06:42:00.9290710Z * [new branch] gh/yangw-dev/17/orig -> origin/gh/yangw-dev/17/orig 2025-09-07T06:42:00.9292753Z * [new branch] gh/yangw-dev/18/base -> origin/gh/yangw-dev/18/base 2025-09-07T06:42:00.9293075Z * [new branch] gh/yangw-dev/18/head -> origin/gh/yangw-dev/18/head 2025-09-07T06:42:00.9293281Z * [new branch] gh/yangw-dev/18/orig -> origin/gh/yangw-dev/18/orig 2025-09-07T06:42:00.9295601Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-09-07T06:42:00.9296163Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-09-07T06:42:00.9296507Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-09-07T06:42:00.9296763Z * [new branch] gh/yangw-dev/20/base -> origin/gh/yangw-dev/20/base 2025-09-07T06:42:00.9297464Z * [new branch] gh/yangw-dev/20/head -> origin/gh/yangw-dev/20/head 2025-09-07T06:42:00.9297873Z * [new branch] gh/yangw-dev/20/orig -> origin/gh/yangw-dev/20/orig 2025-09-07T06:42:00.9301752Z * [new branch] gh/yangw-dev/21/base -> origin/gh/yangw-dev/21/base 2025-09-07T06:42:00.9301945Z * [new branch] gh/yangw-dev/21/head -> origin/gh/yangw-dev/21/head 2025-09-07T06:42:00.9302097Z * [new branch] gh/yangw-dev/21/orig -> origin/gh/yangw-dev/21/orig 2025-09-07T06:42:00.9302252Z * [new branch] gh/yangw-dev/22/base -> origin/gh/yangw-dev/22/base 2025-09-07T06:42:00.9302430Z * [new branch] gh/yangw-dev/22/head -> origin/gh/yangw-dev/22/head 2025-09-07T06:42:00.9302634Z * [new branch] gh/yangw-dev/22/orig -> origin/gh/yangw-dev/22/orig 2025-09-07T06:42:00.9303676Z * [new branch] gh/yangw-dev/23/base -> origin/gh/yangw-dev/23/base 2025-09-07T06:42:00.9304159Z * [new branch] gh/yangw-dev/23/head -> origin/gh/yangw-dev/23/head 2025-09-07T06:42:00.9304683Z * [new branch] gh/yangw-dev/23/orig -> origin/gh/yangw-dev/23/orig 2025-09-07T06:42:00.9305962Z * [new branch] gh/yangw-dev/24/base -> origin/gh/yangw-dev/24/base 2025-09-07T06:42:00.9306402Z * [new branch] gh/yangw-dev/24/head -> origin/gh/yangw-dev/24/head 2025-09-07T06:42:00.9310580Z * [new branch] gh/yangw-dev/24/orig -> origin/gh/yangw-dev/24/orig 2025-09-07T06:42:00.9310757Z * [new branch] gh/yangw-dev/25/base -> origin/gh/yangw-dev/25/base 2025-09-07T06:42:00.9311072Z * [new branch] gh/yangw-dev/25/head -> origin/gh/yangw-dev/25/head 2025-09-07T06:42:00.9311248Z * [new branch] gh/yangw-dev/25/orig -> origin/gh/yangw-dev/25/orig 2025-09-07T06:42:00.9311398Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-09-07T06:42:00.9311555Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-09-07T06:42:00.9311866Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-09-07T06:42:00.9313363Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-09-07T06:42:00.9313511Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-09-07T06:42:00.9314106Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-09-07T06:42:00.9318926Z * [new branch] gh/ydwu4/233/base -> origin/gh/ydwu4/233/base 2025-09-07T06:42:00.9319122Z * [new branch] gh/ydwu4/233/head -> origin/gh/ydwu4/233/head 2025-09-07T06:42:00.9319270Z * [new branch] gh/ydwu4/233/orig -> origin/gh/ydwu4/233/orig 2025-09-07T06:42:00.9319415Z * [new branch] gh/ydwu4/246/base -> origin/gh/ydwu4/246/base 2025-09-07T06:42:00.9319547Z * [new branch] gh/ydwu4/246/head -> origin/gh/ydwu4/246/head 2025-09-07T06:42:00.9319870Z * [new branch] gh/ydwu4/246/orig -> origin/gh/ydwu4/246/orig 2025-09-07T06:42:00.9321098Z * [new branch] gh/ydwu4/253/base -> origin/gh/ydwu4/253/base 2025-09-07T06:42:00.9321276Z * [new branch] gh/ydwu4/253/head -> origin/gh/ydwu4/253/head 2025-09-07T06:42:00.9322375Z * [new branch] gh/ydwu4/253/orig -> origin/gh/ydwu4/253/orig 2025-09-07T06:42:00.9324969Z * [new branch] gh/ydwu4/255/base -> origin/gh/ydwu4/255/base 2025-09-07T06:42:00.9325421Z * [new branch] gh/ydwu4/255/head -> origin/gh/ydwu4/255/head 2025-09-07T06:42:00.9325581Z * [new branch] gh/ydwu4/255/orig -> origin/gh/ydwu4/255/orig 2025-09-07T06:42:00.9325787Z * [new branch] gh/ydwu4/259/base -> origin/gh/ydwu4/259/base 2025-09-07T06:42:00.9326947Z * [new branch] gh/ydwu4/259/head -> origin/gh/ydwu4/259/head 2025-09-07T06:42:00.9327309Z * [new branch] gh/ydwu4/259/orig -> origin/gh/ydwu4/259/orig 2025-09-07T06:42:00.9329539Z * [new branch] gh/ydwu4/262/base -> origin/gh/ydwu4/262/base 2025-09-07T06:42:00.9329730Z * [new branch] gh/ydwu4/262/head -> origin/gh/ydwu4/262/head 2025-09-07T06:42:00.9330480Z * [new branch] gh/ydwu4/262/orig -> origin/gh/ydwu4/262/orig 2025-09-07T06:42:00.9330984Z * [new branch] gh/ydwu4/263/base -> origin/gh/ydwu4/263/base 2025-09-07T06:42:00.9331187Z * [new branch] gh/ydwu4/263/head -> origin/gh/ydwu4/263/head 2025-09-07T06:42:00.9334468Z * [new branch] gh/ydwu4/263/orig -> origin/gh/ydwu4/263/orig 2025-09-07T06:42:00.9335050Z * [new branch] gh/ydwu4/269/base -> origin/gh/ydwu4/269/base 2025-09-07T06:42:00.9335197Z * [new branch] gh/ydwu4/269/head -> origin/gh/ydwu4/269/head 2025-09-07T06:42:00.9335458Z * [new branch] gh/ydwu4/269/orig -> origin/gh/ydwu4/269/orig 2025-09-07T06:42:00.9335803Z * [new branch] gh/ydwu4/270/base -> origin/gh/ydwu4/270/base 2025-09-07T06:42:00.9336018Z * [new branch] gh/ydwu4/270/head -> origin/gh/ydwu4/270/head 2025-09-07T06:42:00.9336899Z * [new branch] gh/ydwu4/270/orig -> origin/gh/ydwu4/270/orig 2025-09-07T06:42:00.9340820Z * [new branch] gh/ydwu4/272/base -> origin/gh/ydwu4/272/base 2025-09-07T06:42:00.9341015Z * [new branch] gh/ydwu4/272/head -> origin/gh/ydwu4/272/head 2025-09-07T06:42:00.9341169Z * [new branch] gh/ydwu4/272/orig -> origin/gh/ydwu4/272/orig 2025-09-07T06:42:00.9341303Z * [new branch] gh/ydwu4/275/base -> origin/gh/ydwu4/275/base 2025-09-07T06:42:00.9341436Z * [new branch] gh/ydwu4/275/head -> origin/gh/ydwu4/275/head 2025-09-07T06:42:00.9341793Z * [new branch] gh/ydwu4/275/orig -> origin/gh/ydwu4/275/orig 2025-09-07T06:42:00.9342247Z * [new branch] gh/ydwu4/276/base -> origin/gh/ydwu4/276/base 2025-09-07T06:42:00.9342867Z * [new branch] gh/ydwu4/276/head -> origin/gh/ydwu4/276/head 2025-09-07T06:42:00.9344252Z * [new branch] gh/ydwu4/276/orig -> origin/gh/ydwu4/276/orig 2025-09-07T06:42:00.9345382Z * [new branch] gh/ydwu4/279/base -> origin/gh/ydwu4/279/base 2025-09-07T06:42:00.9346738Z * [new branch] gh/ydwu4/279/head -> origin/gh/ydwu4/279/head 2025-09-07T06:42:00.9346903Z * [new branch] gh/ydwu4/279/orig -> origin/gh/ydwu4/279/orig 2025-09-07T06:42:00.9353342Z * [new branch] gh/ydwu4/283/base -> origin/gh/ydwu4/283/base 2025-09-07T06:42:00.9357210Z * [new branch] gh/ydwu4/283/head -> origin/gh/ydwu4/283/head 2025-09-07T06:42:00.9361410Z * [new branch] gh/ydwu4/283/orig -> origin/gh/ydwu4/283/orig 2025-09-07T06:42:00.9363472Z * [new branch] gh/ydwu4/289/base -> origin/gh/ydwu4/289/base 2025-09-07T06:42:00.9363622Z * [new branch] gh/ydwu4/289/head -> origin/gh/ydwu4/289/head 2025-09-07T06:42:00.9363780Z * [new branch] gh/ydwu4/289/orig -> origin/gh/ydwu4/289/orig 2025-09-07T06:42:00.9364084Z * [new branch] gh/ydwu4/290/base -> origin/gh/ydwu4/290/base 2025-09-07T06:42:00.9364224Z * [new branch] gh/ydwu4/290/head -> origin/gh/ydwu4/290/head 2025-09-07T06:42:00.9364350Z * [new branch] gh/ydwu4/290/orig -> origin/gh/ydwu4/290/orig 2025-09-07T06:42:00.9364484Z * [new branch] gh/ydwu4/291/base -> origin/gh/ydwu4/291/base 2025-09-07T06:42:00.9364610Z * [new branch] gh/ydwu4/291/head -> origin/gh/ydwu4/291/head 2025-09-07T06:42:00.9364743Z * [new branch] gh/ydwu4/291/orig -> origin/gh/ydwu4/291/orig 2025-09-07T06:42:00.9364876Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-09-07T06:42:00.9365003Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-09-07T06:42:00.9365136Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-09-07T06:42:00.9365265Z * [new branch] gh/ydwu4/293/base -> origin/gh/ydwu4/293/base 2025-09-07T06:42:00.9365398Z * [new branch] gh/ydwu4/293/head -> origin/gh/ydwu4/293/head 2025-09-07T06:42:00.9365522Z * [new branch] gh/ydwu4/293/orig -> origin/gh/ydwu4/293/orig 2025-09-07T06:42:00.9368208Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-09-07T06:42:00.9368462Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-09-07T06:42:00.9368619Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-09-07T06:42:00.9368757Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-09-07T06:42:00.9368889Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-09-07T06:42:00.9369111Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-09-07T06:42:00.9374211Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-09-07T06:42:00.9378915Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-09-07T06:42:00.9383508Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-09-07T06:42:00.9383697Z * [new branch] gh/ydwu4/300/base -> origin/gh/ydwu4/300/base 2025-09-07T06:42:00.9384022Z * [new branch] gh/ydwu4/300/head -> origin/gh/ydwu4/300/head 2025-09-07T06:42:00.9384177Z * [new branch] gh/ydwu4/300/orig -> origin/gh/ydwu4/300/orig 2025-09-07T06:42:00.9384326Z * [new branch] gh/ydwu4/301/base -> origin/gh/ydwu4/301/base 2025-09-07T06:42:00.9384479Z * [new branch] gh/ydwu4/301/head -> origin/gh/ydwu4/301/head 2025-09-07T06:42:00.9384614Z * [new branch] gh/ydwu4/301/orig -> origin/gh/ydwu4/301/orig 2025-09-07T06:42:00.9384773Z * [new branch] gh/ydwu4/302/base -> origin/gh/ydwu4/302/base 2025-09-07T06:42:00.9384912Z * [new branch] gh/ydwu4/302/head -> origin/gh/ydwu4/302/head 2025-09-07T06:42:00.9385066Z * [new branch] gh/ydwu4/302/orig -> origin/gh/ydwu4/302/orig 2025-09-07T06:42:00.9385202Z * [new branch] gh/ydwu4/303/base -> origin/gh/ydwu4/303/base 2025-09-07T06:42:00.9385344Z * [new branch] gh/ydwu4/303/head -> origin/gh/ydwu4/303/head 2025-09-07T06:42:00.9385488Z * [new branch] gh/ydwu4/303/orig -> origin/gh/ydwu4/303/orig 2025-09-07T06:42:00.9385634Z * [new branch] gh/ydwu4/304/base -> origin/gh/ydwu4/304/base 2025-09-07T06:42:00.9386029Z * [new branch] gh/ydwu4/304/head -> origin/gh/ydwu4/304/head 2025-09-07T06:42:00.9386170Z * [new branch] gh/ydwu4/304/orig -> origin/gh/ydwu4/304/orig 2025-09-07T06:42:00.9386380Z * [new branch] gh/ydwu4/305/base -> origin/gh/ydwu4/305/base 2025-09-07T06:42:00.9386526Z * [new branch] gh/ydwu4/305/head -> origin/gh/ydwu4/305/head 2025-09-07T06:42:00.9386659Z * [new branch] gh/ydwu4/305/orig -> origin/gh/ydwu4/305/orig 2025-09-07T06:42:00.9386799Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-09-07T06:42:00.9386936Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-09-07T06:42:00.9387084Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-09-07T06:42:00.9390708Z * [new branch] gh/ydwu4/307/base -> origin/gh/ydwu4/307/base 2025-09-07T06:42:00.9395111Z * [new branch] gh/ydwu4/307/head -> origin/gh/ydwu4/307/head 2025-09-07T06:42:00.9399411Z * [new branch] gh/ydwu4/307/orig -> origin/gh/ydwu4/307/orig 2025-09-07T06:42:00.9401402Z * [new branch] gh/ydwu4/308/base -> origin/gh/ydwu4/308/base 2025-09-07T06:42:00.9401561Z * [new branch] gh/ydwu4/308/head -> origin/gh/ydwu4/308/head 2025-09-07T06:42:00.9401978Z * [new branch] gh/ydwu4/308/orig -> origin/gh/ydwu4/308/orig 2025-09-07T06:42:00.9402125Z * [new branch] gh/ydwu4/309/base -> origin/gh/ydwu4/309/base 2025-09-07T06:42:00.9402309Z * [new branch] gh/ydwu4/309/head -> origin/gh/ydwu4/309/head 2025-09-07T06:42:00.9402461Z * [new branch] gh/ydwu4/309/orig -> origin/gh/ydwu4/309/orig 2025-09-07T06:42:00.9402610Z * [new branch] gh/ydwu4/310/base -> origin/gh/ydwu4/310/base 2025-09-07T06:42:00.9402748Z * [new branch] gh/ydwu4/310/head -> origin/gh/ydwu4/310/head 2025-09-07T06:42:00.9402896Z * [new branch] gh/ydwu4/310/orig -> origin/gh/ydwu4/310/orig 2025-09-07T06:42:00.9403055Z * [new branch] gh/ydwu4/311/base -> origin/gh/ydwu4/311/base 2025-09-07T06:42:00.9403186Z * [new branch] gh/ydwu4/311/head -> origin/gh/ydwu4/311/head 2025-09-07T06:42:00.9403338Z * [new branch] gh/ydwu4/311/orig -> origin/gh/ydwu4/311/orig 2025-09-07T06:42:00.9403483Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-09-07T06:42:00.9403719Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-09-07T06:42:00.9403867Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-09-07T06:42:00.9404015Z * [new branch] gh/ydwu4/313/base -> origin/gh/ydwu4/313/base 2025-09-07T06:42:00.9404160Z * [new branch] gh/ydwu4/313/head -> origin/gh/ydwu4/313/head 2025-09-07T06:42:00.9404304Z * [new branch] gh/ydwu4/313/orig -> origin/gh/ydwu4/313/orig 2025-09-07T06:42:00.9404460Z * [new branch] gh/ydwu4/314/base -> origin/gh/ydwu4/314/base 2025-09-07T06:42:00.9406106Z * [new branch] gh/ydwu4/314/head -> origin/gh/ydwu4/314/head 2025-09-07T06:42:00.9406524Z * [new branch] gh/ydwu4/314/orig -> origin/gh/ydwu4/314/orig 2025-09-07T06:42:00.9406694Z * [new branch] gh/ydwu4/315/base -> origin/gh/ydwu4/315/base 2025-09-07T06:42:00.9406843Z * [new branch] gh/ydwu4/315/head -> origin/gh/ydwu4/315/head 2025-09-07T06:42:00.9407087Z * [new branch] gh/ydwu4/315/orig -> origin/gh/ydwu4/315/orig 2025-09-07T06:42:00.9407234Z * [new branch] gh/ydwu4/316/base -> origin/gh/ydwu4/316/base 2025-09-07T06:42:00.9411730Z * [new branch] gh/ydwu4/316/head -> origin/gh/ydwu4/316/head 2025-09-07T06:42:00.9412026Z * [new branch] gh/ydwu4/316/orig -> origin/gh/ydwu4/316/orig 2025-09-07T06:42:00.9412296Z * [new branch] gh/ydwu4/317/base -> origin/gh/ydwu4/317/base 2025-09-07T06:42:00.9412445Z * [new branch] gh/ydwu4/317/head -> origin/gh/ydwu4/317/head 2025-09-07T06:42:00.9412667Z * [new branch] gh/ydwu4/317/orig -> origin/gh/ydwu4/317/orig 2025-09-07T06:42:00.9416813Z * [new branch] gh/ydwu4/318/base -> origin/gh/ydwu4/318/base 2025-09-07T06:42:00.9417011Z * [new branch] gh/ydwu4/318/head -> origin/gh/ydwu4/318/head 2025-09-07T06:42:00.9417151Z * [new branch] gh/ydwu4/318/orig -> origin/gh/ydwu4/318/orig 2025-09-07T06:42:00.9417280Z * [new branch] gh/ydwu4/319/base -> origin/gh/ydwu4/319/base 2025-09-07T06:42:00.9417406Z * [new branch] gh/ydwu4/319/head -> origin/gh/ydwu4/319/head 2025-09-07T06:42:00.9417542Z * [new branch] gh/ydwu4/319/orig -> origin/gh/ydwu4/319/orig 2025-09-07T06:42:00.9417691Z * [new branch] gh/ydwu4/320/base -> origin/gh/ydwu4/320/base 2025-09-07T06:42:00.9417823Z * [new branch] gh/ydwu4/320/head -> origin/gh/ydwu4/320/head 2025-09-07T06:42:00.9421138Z * [new branch] gh/ydwu4/320/orig -> origin/gh/ydwu4/320/orig 2025-09-07T06:42:00.9421280Z * [new branch] gh/ydwu4/321/base -> origin/gh/ydwu4/321/base 2025-09-07T06:42:00.9421419Z * [new branch] gh/ydwu4/321/head -> origin/gh/ydwu4/321/head 2025-09-07T06:42:00.9421548Z * [new branch] gh/ydwu4/321/orig -> origin/gh/ydwu4/321/orig 2025-09-07T06:42:00.9423429Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-09-07T06:42:00.9424336Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-09-07T06:42:00.9424802Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-09-07T06:42:00.9430135Z * [new branch] gh/ydwu4/323/base -> origin/gh/ydwu4/323/base 2025-09-07T06:42:00.9435634Z * [new branch] gh/ydwu4/323/head -> origin/gh/ydwu4/323/head 2025-09-07T06:42:00.9440652Z * [new branch] gh/ydwu4/323/orig -> origin/gh/ydwu4/323/orig 2025-09-07T06:42:00.9446113Z * [new branch] gh/ydwu4/324/base -> origin/gh/ydwu4/324/base 2025-09-07T06:42:00.9450901Z * [new branch] gh/ydwu4/324/head -> origin/gh/ydwu4/324/head 2025-09-07T06:42:00.9455920Z * [new branch] gh/ydwu4/324/orig -> origin/gh/ydwu4/324/orig 2025-09-07T06:42:00.9460338Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-09-07T06:42:00.9462182Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-09-07T06:42:00.9462334Z * [new branch] gh/yf225/171/base -> origin/gh/yf225/171/base 2025-09-07T06:42:00.9462485Z * [new branch] gh/yf225/171/head -> origin/gh/yf225/171/head 2025-09-07T06:42:00.9462639Z * [new branch] gh/yf225/171/orig -> origin/gh/yf225/171/orig 2025-09-07T06:42:00.9462770Z * [new branch] gh/yf225/172/base -> origin/gh/yf225/172/base 2025-09-07T06:42:00.9462940Z * [new branch] gh/yf225/172/head -> origin/gh/yf225/172/head 2025-09-07T06:42:00.9463076Z * [new branch] gh/yf225/172/orig -> origin/gh/yf225/172/orig 2025-09-07T06:42:00.9463217Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-09-07T06:42:00.9463361Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-09-07T06:42:00.9463520Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-09-07T06:42:00.9463678Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-09-07T06:42:00.9464031Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-09-07T06:42:00.9464187Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-09-07T06:42:00.9464330Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-09-07T06:42:00.9464475Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-09-07T06:42:00.9464638Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-09-07T06:42:00.9464782Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-09-07T06:42:00.9464931Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-09-07T06:42:00.9465071Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-09-07T06:42:00.9465227Z * [new branch] gh/ysiraichi/79/base -> origin/gh/ysiraichi/79/base 2025-09-07T06:42:00.9465376Z * [new branch] gh/ysiraichi/79/head -> origin/gh/ysiraichi/79/head 2025-09-07T06:42:00.9465518Z * [new branch] gh/ysiraichi/79/orig -> origin/gh/ysiraichi/79/orig 2025-09-07T06:42:00.9465756Z * [new branch] gh/ysiraichi/88/base -> origin/gh/ysiraichi/88/base 2025-09-07T06:42:00.9465913Z * [new branch] gh/ysiraichi/88/head -> origin/gh/ysiraichi/88/head 2025-09-07T06:42:00.9466069Z * [new branch] gh/ysiraichi/88/orig -> origin/gh/ysiraichi/88/orig 2025-09-07T06:42:00.9466212Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2025-09-07T06:42:00.9466355Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2025-09-07T06:42:00.9466502Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2025-09-07T06:42:00.9466644Z * [new branch] gh/zhxchen17/31/base -> origin/gh/zhxchen17/31/base 2025-09-07T06:42:00.9466799Z * [new branch] gh/zhxchen17/31/head -> origin/gh/zhxchen17/31/head 2025-09-07T06:42:00.9466942Z * [new branch] gh/zhxchen17/31/orig -> origin/gh/zhxchen17/31/orig 2025-09-07T06:42:00.9467094Z * [new branch] gh/zhxchen17/34/base -> origin/gh/zhxchen17/34/base 2025-09-07T06:42:00.9467238Z * [new branch] gh/zhxchen17/34/head -> origin/gh/zhxchen17/34/head 2025-09-07T06:42:00.9467434Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2025-09-07T06:42:00.9467586Z * [new branch] gh/zhxchen17/35/head -> origin/gh/zhxchen17/35/head 2025-09-07T06:42:00.9467727Z * [new branch] gh/zhxchen17/37/base -> origin/gh/zhxchen17/37/base 2025-09-07T06:42:00.9467878Z * [new branch] gh/zhxchen17/37/head -> origin/gh/zhxchen17/37/head 2025-09-07T06:42:00.9468023Z * [new branch] gh/zhxchen17/37/orig -> origin/gh/zhxchen17/37/orig 2025-09-07T06:42:00.9468168Z * [new branch] gh/zhxchen17/38/base -> origin/gh/zhxchen17/38/base 2025-09-07T06:42:00.9468329Z * [new branch] gh/zhxchen17/38/head -> origin/gh/zhxchen17/38/head 2025-09-07T06:42:00.9468471Z * [new branch] gh/zhxchen17/38/orig -> origin/gh/zhxchen17/38/orig 2025-09-07T06:42:00.9468625Z * [new branch] gh/zhxchen17/39/base -> origin/gh/zhxchen17/39/base 2025-09-07T06:42:00.9468768Z * [new branch] gh/zhxchen17/39/head -> origin/gh/zhxchen17/39/head 2025-09-07T06:42:00.9468922Z * [new branch] gh/zhxchen17/39/orig -> origin/gh/zhxchen17/39/orig 2025-09-07T06:42:00.9471407Z * [new branch] gh/zhxchen17/40/base -> origin/gh/zhxchen17/40/base 2025-09-07T06:42:00.9471652Z * [new branch] gh/zhxchen17/40/head -> origin/gh/zhxchen17/40/head 2025-09-07T06:42:00.9472052Z * [new branch] gh/zhxchen17/40/orig -> origin/gh/zhxchen17/40/orig 2025-09-07T06:42:00.9472344Z * [new branch] gh/zhxchen17/41/base -> origin/gh/zhxchen17/41/base 2025-09-07T06:42:00.9472597Z * [new branch] gh/zhxchen17/41/head -> origin/gh/zhxchen17/41/head 2025-09-07T06:42:00.9472764Z * [new branch] gh/zhxchen17/41/orig -> origin/gh/zhxchen17/41/orig 2025-09-07T06:42:00.9473433Z * [new branch] gh/zhxchen17/42/base -> origin/gh/zhxchen17/42/base 2025-09-07T06:42:00.9478699Z * [new branch] gh/zhxchen17/42/head -> origin/gh/zhxchen17/42/head 2025-09-07T06:42:00.9483780Z * [new branch] gh/zhxchen17/42/orig -> origin/gh/zhxchen17/42/orig 2025-09-07T06:42:00.9489339Z * [new branch] gh/zhxchen17/43/base -> origin/gh/zhxchen17/43/base 2025-09-07T06:42:00.9491598Z * [new branch] gh/zhxchen17/43/head -> origin/gh/zhxchen17/43/head 2025-09-07T06:42:00.9497539Z * [new branch] gh/zhxchen17/43/orig -> origin/gh/zhxchen17/43/orig 2025-09-07T06:42:00.9499904Z * [new branch] gh/zhxchen17/44/base -> origin/gh/zhxchen17/44/base 2025-09-07T06:42:00.9500102Z * [new branch] gh/zhxchen17/44/head -> origin/gh/zhxchen17/44/head 2025-09-07T06:42:00.9500259Z * [new branch] gh/zhxchen17/44/orig -> origin/gh/zhxchen17/44/orig 2025-09-07T06:42:00.9500426Z * [new branch] gh/zhxchen17/45/base -> origin/gh/zhxchen17/45/base 2025-09-07T06:42:00.9500582Z * [new branch] gh/zhxchen17/45/head -> origin/gh/zhxchen17/45/head 2025-09-07T06:42:00.9500724Z * [new branch] gh/zhxchen17/45/orig -> origin/gh/zhxchen17/45/orig 2025-09-07T06:42:00.9500871Z * [new branch] gh/zklaus/10/base -> origin/gh/zklaus/10/base 2025-09-07T06:42:00.9501027Z * [new branch] gh/zklaus/10/head -> origin/gh/zklaus/10/head 2025-09-07T06:42:00.9501172Z * [new branch] gh/zklaus/10/orig -> origin/gh/zklaus/10/orig 2025-09-07T06:42:00.9501313Z * [new branch] gh/zklaus/11/base -> origin/gh/zklaus/11/base 2025-09-07T06:42:00.9501444Z * [new branch] gh/zklaus/11/head -> origin/gh/zklaus/11/head 2025-09-07T06:42:00.9501577Z * [new branch] gh/zklaus/11/orig -> origin/gh/zklaus/11/orig 2025-09-07T06:42:00.9501878Z * [new branch] gh/zklaus/12/base -> origin/gh/zklaus/12/base 2025-09-07T06:42:00.9502013Z * [new branch] gh/zklaus/12/head -> origin/gh/zklaus/12/head 2025-09-07T06:42:00.9502155Z * [new branch] gh/zklaus/12/orig -> origin/gh/zklaus/12/orig 2025-09-07T06:42:00.9502294Z * [new branch] gh/zklaus/14/base -> origin/gh/zklaus/14/base 2025-09-07T06:42:00.9502438Z * [new branch] gh/zklaus/14/head -> origin/gh/zklaus/14/head 2025-09-07T06:42:00.9502577Z * [new branch] gh/zklaus/14/orig -> origin/gh/zklaus/14/orig 2025-09-07T06:42:00.9502713Z * [new branch] gh/zklaus/15/base -> origin/gh/zklaus/15/base 2025-09-07T06:42:00.9502853Z * [new branch] gh/zklaus/15/head -> origin/gh/zklaus/15/head 2025-09-07T06:42:00.9502984Z * [new branch] gh/zklaus/15/orig -> origin/gh/zklaus/15/orig 2025-09-07T06:42:00.9503128Z * [new branch] gh/zklaus/16/base -> origin/gh/zklaus/16/base 2025-09-07T06:42:00.9503261Z * [new branch] gh/zklaus/16/head -> origin/gh/zklaus/16/head 2025-09-07T06:42:00.9503400Z * [new branch] gh/zklaus/16/orig -> origin/gh/zklaus/16/orig 2025-09-07T06:42:00.9503533Z * [new branch] gh/zklaus/17/base -> origin/gh/zklaus/17/base 2025-09-07T06:42:00.9503706Z * [new branch] gh/zklaus/17/head -> origin/gh/zklaus/17/head 2025-09-07T06:42:00.9503913Z * [new branch] gh/zklaus/17/orig -> origin/gh/zklaus/17/orig 2025-09-07T06:42:00.9504053Z * [new branch] gh/zklaus/18/base -> origin/gh/zklaus/18/base 2025-09-07T06:42:00.9504195Z * [new branch] gh/zklaus/18/head -> origin/gh/zklaus/18/head 2025-09-07T06:42:00.9504329Z * [new branch] gh/zklaus/18/orig -> origin/gh/zklaus/18/orig 2025-09-07T06:42:00.9504467Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-09-07T06:42:00.9504610Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-09-07T06:42:00.9504746Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-09-07T06:42:00.9504897Z * [new branch] gh/zklaus/20/base -> origin/gh/zklaus/20/base 2025-09-07T06:42:00.9505033Z * [new branch] gh/zklaus/20/head -> origin/gh/zklaus/20/head 2025-09-07T06:42:00.9505179Z * [new branch] gh/zklaus/20/orig -> origin/gh/zklaus/20/orig 2025-09-07T06:42:00.9505329Z * [new branch] gh/zklaus/7/base -> origin/gh/zklaus/7/base 2025-09-07T06:42:00.9505470Z * [new branch] gh/zklaus/7/head -> origin/gh/zklaus/7/head 2025-09-07T06:42:00.9505615Z * [new branch] gh/zklaus/7/orig -> origin/gh/zklaus/7/orig 2025-09-07T06:42:00.9506415Z * [new branch] gh/zklaus/9/base -> origin/gh/zklaus/9/base 2025-09-07T06:42:00.9506679Z * [new branch] gh/zklaus/9/head -> origin/gh/zklaus/9/head 2025-09-07T06:42:00.9514793Z * [new branch] gh/zklaus/9/orig -> origin/gh/zklaus/9/orig 2025-09-07T06:42:00.9519276Z * [new branch] gh/zou3519/1175/base -> origin/gh/zou3519/1175/base 2025-09-07T06:42:00.9519468Z * [new branch] gh/zou3519/1175/head -> origin/gh/zou3519/1175/head 2025-09-07T06:42:00.9520083Z * [new branch] gh/zou3519/1175/orig -> origin/gh/zou3519/1175/orig 2025-09-07T06:42:00.9520258Z * [new branch] gh/zou3519/1177/base -> origin/gh/zou3519/1177/base 2025-09-07T06:42:00.9520402Z * [new branch] gh/zou3519/1177/head -> origin/gh/zou3519/1177/head 2025-09-07T06:42:00.9520538Z * [new branch] gh/zou3519/1177/orig -> origin/gh/zou3519/1177/orig 2025-09-07T06:42:00.9520897Z * [new branch] gh/zou3519/1191/base -> origin/gh/zou3519/1191/base 2025-09-07T06:42:00.9521054Z * [new branch] gh/zou3519/1191/head -> origin/gh/zou3519/1191/head 2025-09-07T06:42:00.9521202Z * [new branch] gh/zou3519/1191/orig -> origin/gh/zou3519/1191/orig 2025-09-07T06:42:00.9521367Z * [new branch] gh/zou3519/1192/base -> origin/gh/zou3519/1192/base 2025-09-07T06:42:00.9521521Z * [new branch] gh/zou3519/1192/head -> origin/gh/zou3519/1192/head 2025-09-07T06:42:00.9521672Z * [new branch] gh/zou3519/1192/orig -> origin/gh/zou3519/1192/orig 2025-09-07T06:42:00.9521826Z * [new branch] gh/zou3519/1193/base -> origin/gh/zou3519/1193/base 2025-09-07T06:42:00.9521981Z * [new branch] gh/zou3519/1193/head -> origin/gh/zou3519/1193/head 2025-09-07T06:42:00.9522124Z * [new branch] gh/zou3519/1193/orig -> origin/gh/zou3519/1193/orig 2025-09-07T06:42:00.9524886Z * [new branch] gh/zou3519/1194/base -> origin/gh/zou3519/1194/base 2025-09-07T06:42:00.9525045Z * [new branch] gh/zou3519/1194/head -> origin/gh/zou3519/1194/head 2025-09-07T06:42:00.9525192Z * [new branch] gh/zou3519/1194/orig -> origin/gh/zou3519/1194/orig 2025-09-07T06:42:00.9525431Z * [new branch] gh/zou3519/1195/base -> origin/gh/zou3519/1195/base 2025-09-07T06:42:00.9530605Z * [new branch] gh/zou3519/1195/head -> origin/gh/zou3519/1195/head 2025-09-07T06:42:00.9535288Z * [new branch] gh/zou3519/1195/orig -> origin/gh/zou3519/1195/orig 2025-09-07T06:42:00.9540152Z * [new branch] gh/zou3519/1196/base -> origin/gh/zou3519/1196/base 2025-09-07T06:42:00.9542141Z * [new branch] gh/zou3519/1196/head -> origin/gh/zou3519/1196/head 2025-09-07T06:42:00.9542323Z * [new branch] gh/zou3519/1196/orig -> origin/gh/zou3519/1196/orig 2025-09-07T06:42:00.9542496Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-09-07T06:42:00.9542644Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-09-07T06:42:00.9542792Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-09-07T06:42:00.9542956Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-09-07T06:42:00.9543101Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-09-07T06:42:00.9543256Z * [new branch] gh/zpcore/10/base -> origin/gh/zpcore/10/base 2025-09-07T06:42:00.9543394Z * [new branch] gh/zpcore/10/head -> origin/gh/zpcore/10/head 2025-09-07T06:42:00.9543537Z * [new branch] gh/zpcore/10/orig -> origin/gh/zpcore/10/orig 2025-09-07T06:42:00.9543680Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-09-07T06:42:00.9543818Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-09-07T06:42:00.9543965Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-09-07T06:42:00.9544115Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-09-07T06:42:00.9544265Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-09-07T06:42:00.9544425Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-09-07T06:42:00.9544564Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-09-07T06:42:00.9544710Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-09-07T06:42:00.9544856Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-09-07T06:42:00.9545139Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-09-07T06:42:00.9545276Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-09-07T06:42:00.9545424Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-09-07T06:42:00.9545571Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-09-07T06:42:00.9545795Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-09-07T06:42:00.9545955Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-09-07T06:42:00.9546093Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-09-07T06:42:00.9551332Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-09-07T06:42:00.9554460Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-09-07T06:42:00.9559787Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-09-07T06:42:00.9563661Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-09-07T06:42:00.9566748Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-09-07T06:42:00.9570531Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-09-07T06:42:00.9570688Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-09-07T06:42:00.9570923Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-09-07T06:42:00.9571053Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-09-07T06:42:00.9571203Z * [new branch] google-main -> origin/google-main 2025-09-07T06:42:00.9571375Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-09-07T06:42:00.9571537Z * [new branch] guangyey/host_alloc -> origin/guangyey/host_alloc 2025-09-07T06:42:00.9571680Z * [new branch] guangyey/reimport -> origin/guangyey/reimport 2025-09-07T06:42:00.9571829Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-09-07T06:42:00.9572080Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-09-07T06:42:00.9572247Z * [new branch] haozhe/bf16-dynamic-shape -> origin/haozhe/bf16-dynamic-shape 2025-09-07T06:42:00.9572383Z * [new branch] hc_baseline -> origin/hc_baseline 2025-09-07T06:42:00.9572502Z * [new branch] hf_update -> origin/hf_update 2025-09-07T06:42:00.9572637Z * [new branch] hhh_decomp_mul -> origin/hhh_decomp_mul 2025-09-07T06:42:00.9572751Z * [new branch] hhh_rand -> origin/hhh_rand 2025-09-07T06:42:00.9572883Z * [new branch] hoy/mmsplitk -> origin/hoy/mmsplitk 2025-09-07T06:42:00.9573027Z * [new branch] hoy/triton-PR3973 -> origin/hoy/triton-PR3973 2025-09-07T06:42:00.9573257Z * [new branch] hoy/triton-coalescing-baseline -> origin/hoy/triton-coalescing-baseline 2025-09-07T06:42:00.9573427Z * [new branch] hoy/triton-coalescing-new -> origin/hoy/triton-coalescing-new 2025-09-07T06:42:00.9573611Z * [new branch] hoy/triton-coalescing-vec -> origin/hoy/triton-coalescing-vec 2025-09-07T06:42:00.9574017Z * [new branch] inductordecompfix -> origin/inductordecompfix 2025-09-07T06:42:00.9574621Z * [new branch] inline -> origin/inline 2025-09-07T06:42:00.9574779Z * [new branch] inlining -> origin/inlining 2025-09-07T06:42:00.9575099Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-09-07T06:42:00.9575327Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-09-07T06:42:00.9575451Z * [new branch] int8_sdpa -> origin/int8_sdpa 2025-09-07T06:42:00.9575588Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-09-07T06:42:00.9575865Z * [new branch] issue#58739 -> origin/issue#58739 2025-09-07T06:42:00.9576109Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-09-07T06:42:00.9576324Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-09-07T06:42:00.9576532Z * [new branch] jeanschmidt/disable_rocm_build_tests -> origin/jeanschmidt/disable_rocm_build_tests 2025-09-07T06:42:00.9576708Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-09-07T06:42:00.9576873Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-09-07T06:42:00.9577036Z * [new branch] justinchu/attention-tests -> origin/justinchu/attention-tests 2025-09-07T06:42:00.9579758Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-09-07T06:42:00.9579949Z * [new branch] justinchu/ort-122 -> origin/justinchu/ort-122 2025-09-07T06:42:00.9580141Z * [new branch] justinchuby/dynamo-true -> origin/justinchuby/dynamo-true 2025-09-07T06:42:00.9580457Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-09-07T06:42:00.9580621Z * [new branch] kainan_test -> origin/kainan_test 2025-09-07T06:42:00.9580771Z * [new branch] learnablebias -> origin/learnablebias 2025-09-07T06:42:00.9581256Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-09-07T06:42:00.9582573Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-09-07T06:42:00.9582941Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-09-07T06:42:00.9583861Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-09-07T06:42:00.9584147Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-09-07T06:42:00.9585267Z * [new branch] lintbuilddocker -> origin/lintbuilddocker 2025-09-07T06:42:00.9585580Z * [new branch] llama4-stable -> origin/llama4-stable 2025-09-07T06:42:00.9590431Z * [new branch] logdetfix -> origin/logdetfix 2025-09-07T06:42:00.9590629Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-09-07T06:42:00.9590801Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-09-07T06:42:00.9590998Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-09-07T06:42:00.9591203Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-09-07T06:42:00.9591418Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-09-07T06:42:00.9591811Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-09-07T06:42:00.9592261Z * [new branch] lucaskabela/issue_120648 -> origin/lucaskabela/issue_120648 2025-09-07T06:42:00.9593741Z * [new branch] lucaskabela/misc_typing_dynamo -> origin/lucaskabela/misc_typing_dynamo 2025-09-07T06:42:00.9593979Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-09-07T06:42:00.9594912Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-09-07T06:42:00.9595442Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-09-07T06:42:00.9596227Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-09-07T06:42:00.9596725Z * [new branch] lucaskabela/typing_symbolic_convert -> origin/lucaskabela/typing_symbolic_convert 2025-09-07T06:42:00.9598946Z * [new branch] lucaskabela/typing_utils_improvements -> origin/lucaskabela/typing_utils_improvements 2025-09-07T06:42:00.9599275Z * [new branch] main -> origin/main 2025-09-07T06:42:00.9599536Z * [new branch] main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests 2025-09-07T06:42:00.9599833Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-09-07T06:42:00.9601950Z * [new branch] malfet-patch-12 -> origin/malfet-patch-12 2025-09-07T06:42:00.9602271Z * [new branch] malfet-patch-14 -> origin/malfet-patch-14 2025-09-07T06:42:00.9602453Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-09-07T06:42:00.9603760Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-09-07T06:42:00.9604310Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-09-07T06:42:00.9607041Z * [new branch] malfet/delete-upsteam-cuda -> origin/malfet/delete-upsteam-cuda 2025-09-07T06:42:00.9607426Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-09-07T06:42:00.9607759Z * [new branch] manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps 2025-09-07T06:42:00.9608039Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-09-07T06:42:00.9608690Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-09-07T06:42:00.9608872Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-09-07T06:42:00.9610617Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-09-07T06:42:00.9610992Z * [new branch] mlazos/backup-test-branch -> origin/mlazos/backup-test-branch 2025-09-07T06:42:00.9611250Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-09-07T06:42:00.9611452Z * [new branch] mlazos/baseline -> origin/mlazos/baseline 2025-09-07T06:42:00.9614250Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-09-07T06:42:00.9614593Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-09-07T06:42:00.9614836Z * [new branch] mlazos/better-msg -> origin/mlazos/better-msg 2025-09-07T06:42:00.9615083Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-09-07T06:42:00.9615339Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-09-07T06:42:00.9616011Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-09-07T06:42:00.9619002Z * [new branch] mlazos/ck2 -> origin/mlazos/ck2 2025-09-07T06:42:00.9619334Z * [new branch] mlazos/combokernels -> origin/mlazos/combokernels 2025-09-07T06:42:00.9619527Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-09-07T06:42:00.9619941Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-09-07T06:42:00.9620136Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-09-07T06:42:00.9620750Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-09-07T06:42:00.9621432Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-09-07T06:42:00.9624153Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-09-07T06:42:00.9624351Z * [new branch] mlazos/data-gather -> origin/mlazos/data-gather 2025-09-07T06:42:00.9624520Z * [new branch] mlazos/data-ptrs2 -> origin/mlazos/data-ptrs2 2025-09-07T06:42:00.9624682Z * [new branch] mlazos/data-ptrs3 -> origin/mlazos/data-ptrs3 2025-09-07T06:42:00.9626663Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-09-07T06:42:00.9627039Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-09-07T06:42:00.9627205Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-09-07T06:42:00.9627374Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-09-07T06:42:00.9632443Z * [new branch] mlazos/disable-closures -> origin/mlazos/disable-closures 2025-09-07T06:42:00.9632632Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-09-07T06:42:00.9632967Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-09-07T06:42:00.9633107Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-09-07T06:42:00.9633443Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-09-07T06:42:00.9633607Z * [new branch] mlazos/exp_disable -> origin/mlazos/exp_disable 2025-09-07T06:42:00.9633767Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-09-07T06:42:00.9633932Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-09-07T06:42:00.9634075Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-09-07T06:42:00.9638524Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-09-07T06:42:00.9639122Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-09-07T06:42:00.9639277Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-09-07T06:42:00.9639430Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-09-07T06:42:00.9639580Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-09-07T06:42:00.9639726Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-09-07T06:42:00.9639865Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-09-07T06:42:00.9640008Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-09-07T06:42:00.9640334Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-09-07T06:42:00.9645312Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-09-07T06:42:00.9645503Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-09-07T06:42:00.9645642Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-09-07T06:42:00.9645785Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-09-07T06:42:00.9645918Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-09-07T06:42:00.9646067Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-09-07T06:42:00.9646198Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-09-07T06:42:00.9646325Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-09-07T06:42:00.9646662Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-09-07T06:42:00.9647343Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-09-07T06:42:00.9648715Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-09-07T06:42:00.9648863Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-09-07T06:42:00.9649688Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-09-07T06:42:00.9650107Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-09-07T06:42:00.9653930Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-09-07T06:42:00.9654086Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-09-07T06:42:00.9654218Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-09-07T06:42:00.9654354Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-09-07T06:42:00.9654507Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-09-07T06:42:00.9654682Z * [new branch] mlazos/init-per-param -> origin/mlazos/init-per-param 2025-09-07T06:42:00.9654870Z * [new branch] mlazos/init_per_param -> origin/mlazos/init_per_param 2025-09-07T06:42:00.9655511Z * [new branch] mlazos/less-guards -> origin/mlazos/less-guards 2025-09-07T06:42:00.9656203Z * [new branch] mlazos/lr-composibility -> origin/mlazos/lr-composibility 2025-09-07T06:42:00.9656850Z * [new branch] mlazos/main -> origin/mlazos/main 2025-09-07T06:42:00.9657421Z * [new branch] mlazos/main-test-enablement -> origin/mlazos/main-test-enablement 2025-09-07T06:42:00.9658274Z * [new branch] mlazos/main2 -> origin/mlazos/main2 2025-09-07T06:42:00.9658795Z * [new branch] mlazos/mark-static-update -> origin/mlazos/mark-static-update 2025-09-07T06:42:00.9661680Z * [new branch] mlazos/mcg -> origin/mlazos/mcg 2025-09-07T06:42:00.9662446Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-09-07T06:42:00.9662827Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-09-07T06:42:00.9662987Z * [new branch] mlazos/mlazos/ck2 -> origin/mlazos/mlazos/ck2 2025-09-07T06:42:00.9663343Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-09-07T06:42:00.9663578Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-09-07T06:42:00.9664467Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-09-07T06:42:00.9664946Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-09-07T06:42:00.9666203Z * [new branch] mlazos/more-tests -> origin/mlazos/more-tests 2025-09-07T06:42:00.9668790Z * [new branch] mlazos/no-cpp -> origin/mlazos/no-cpp 2025-09-07T06:42:00.9669328Z * [new branch] mlazos/no-init-group-handling -> origin/mlazos/no-init-group-handling 2025-09-07T06:42:00.9669506Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-09-07T06:42:00.9669690Z * [new branch] mlazos/opt-bench-exp2 -> origin/mlazos/opt-bench-exp2 2025-09-07T06:42:00.9669839Z * [new branch] mlazos/opt-incr -> origin/mlazos/opt-incr 2025-09-07T06:42:00.9670035Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-09-07T06:42:00.9673898Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-09-07T06:42:00.9674076Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-09-07T06:42:00.9674233Z * [new branch] mlazos/revert-inline -> origin/mlazos/revert-inline 2025-09-07T06:42:00.9674514Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-09-07T06:42:00.9674665Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-09-07T06:42:00.9674795Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-09-07T06:42:00.9674929Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-09-07T06:42:00.9675104Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-09-07T06:42:00.9679592Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-09-07T06:42:00.9680007Z * [new branch] mlazos/sub-param-fix -> origin/mlazos/sub-param-fix 2025-09-07T06:42:00.9680168Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-09-07T06:42:00.9680359Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-09-07T06:42:00.9680505Z * [new branch] mlazos/test -> origin/mlazos/test 2025-09-07T06:42:00.9680677Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-09-07T06:42:00.9680872Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-09-07T06:42:00.9681467Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-09-07T06:42:00.9681852Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-09-07T06:42:00.9682871Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-09-07T06:42:00.9683162Z * [new branch] mlazos/topo-fix -> origin/mlazos/topo-fix 2025-09-07T06:42:00.9685887Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-09-07T06:42:00.9686084Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-09-07T06:42:00.9686254Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-09-07T06:42:00.9686409Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-09-07T06:42:00.9686596Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-09-07T06:42:00.9687610Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-09-07T06:42:00.9687946Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-09-07T06:42:00.9693637Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-09-07T06:42:00.9697993Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-09-07T06:42:00.9700507Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-09-07T06:42:00.9701088Z * [new branch] modify-setupvllm -> origin/modify-setupvllm 2025-09-07T06:42:00.9701595Z * [new branch] module-shim -> origin/module-shim 2025-09-07T06:42:00.9701786Z * [new branch] move-theme-out-docker -> origin/move-theme-out-docker 2025-09-07T06:42:00.9701932Z * [new branch] msaroufim/be1 -> origin/msaroufim/be1 2025-09-07T06:42:00.9702077Z * [new branch] msaroufim/cn_path -> origin/msaroufim/cn_path 2025-09-07T06:42:00.9702284Z * [new branch] msaroufim/dtensorfusedadam -> origin/msaroufim/dtensorfusedadam 2025-09-07T06:42:00.9702430Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-09-07T06:42:00.9702580Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-09-07T06:42:00.9702715Z * [new branch] muon_dev -> origin/muon_dev 2025-09-07T06:42:00.9702838Z * [new branch] muon_dev_1 -> origin/muon_dev_1 2025-09-07T06:42:00.9703141Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-09-07T06:42:00.9703292Z * [new branch] nativert_numoutputs -> origin/nativert_numoutputs 2025-09-07T06:42:00.9703465Z * [new branch] new-modifiy-setupvllm -> origin/new-modifiy-setupvllm 2025-09-07T06:42:00.9703599Z * [new branch] new-setupvllm -> origin/new-setupvllm 2025-09-07T06:42:00.9703735Z * [new branch] new_zeros_dtype -> origin/new_zeros_dtype 2025-09-07T06:42:00.9705018Z * [new branch] newtest-base -> origin/newtest-base 2025-09-07T06:42:00.9705164Z * [new branch] ngimel/cat_perf1 -> origin/ngimel/cat_perf1 2025-09-07T06:42:00.9705596Z * [new branch] ngimel/einsum_fix -> origin/ngimel/einsum_fix 2025-09-07T06:42:00.9705880Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-09-07T06:42:00.9707241Z * [new branch] ngimel/fabric_check -> origin/ngimel/fabric_check 2025-09-07T06:42:00.9707402Z * [new branch] ngimel/fabric_fix -> origin/ngimel/fabric_fix 2025-09-07T06:42:00.9707592Z * [new branch] ngimel/fix_driver_init_error -> origin/ngimel/fix_driver_init_error 2025-09-07T06:42:00.9709656Z * [new branch] ngimel/fix_nccl_segment_seg -> origin/ngimel/fix_nccl_segment_seg 2025-09-07T06:42:00.9714371Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-09-07T06:42:00.9718391Z * [new branch] ngimel/modeguard -> origin/ngimel/modeguard 2025-09-07T06:42:00.9723336Z * [new branch] ngimel/multicast_fix -> origin/ngimel/multicast_fix 2025-09-07T06:42:00.9723551Z * [new branch] ngimel/rocm_handle_type -> origin/ngimel/rocm_handle_type 2025-09-07T06:42:00.9723754Z * [new branch] ngimel/symm_handle_fabric -> origin/ngimel/symm_handle_fabric 2025-09-07T06:42:00.9723931Z * [new branch] ngimel/unbind_multimem -> origin/ngimel/unbind_multimem 2025-09-07T06:42:00.9724058Z * [new branch] nightly -> origin/nightly 2025-09-07T06:42:00.9724235Z * [new branch] nmacchioni-patch-10 -> origin/nmacchioni-patch-10 2025-09-07T06:42:00.9724391Z * [new branch] nmacchioni-patch-7 -> origin/nmacchioni-patch-7 2025-09-07T06:42:00.9724577Z * [new branch] nmacchioni-patch-8 -> origin/nmacchioni-patch-8 2025-09-07T06:42:00.9724727Z * [new branch] nmacchioni-patch-9 -> origin/nmacchioni-patch-9 2025-09-07T06:42:00.9724875Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-09-07T06:42:00.9725038Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-09-07T06:42:00.9725164Z * [new branch] one-off -> origin/one-off 2025-09-07T06:42:00.9725307Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-09-07T06:42:00.9725461Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-09-07T06:42:00.9725605Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-09-07T06:42:00.9725907Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-09-07T06:42:00.9726142Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-09-07T06:42:00.9726287Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-09-07T06:42:00.9726425Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-09-07T06:42:00.9726778Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-09-07T06:42:00.9727443Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-09-07T06:42:00.9730576Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-09-07T06:42:00.9730919Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-09-07T06:42:00.9731085Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-09-07T06:42:00.9731230Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-09-07T06:42:00.9731542Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-09-07T06:42:00.9731762Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-09-07T06:42:00.9736852Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-09-07T06:42:00.9737191Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-09-07T06:42:00.9737447Z * [new branch] oulgen/fx_graph -> origin/oulgen/fx_graph 2025-09-07T06:42:00.9737698Z * [new branch] padded-tensor -> origin/padded-tensor 2025-09-07T06:42:00.9737930Z * [new branch] pca2 -> origin/pca2 2025-09-07T06:42:00.9738098Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-09-07T06:42:00.9738407Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-09-07T06:42:00.9738948Z * [new branch] pianpwk/invalidate_fake_memo -> origin/pianpwk/invalidate_fake_memo 2025-09-07T06:42:00.9739378Z * [new branch] pianpwk/max_1_strides -> origin/pianpwk/max_1_strides 2025-09-07T06:42:00.9740456Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-09-07T06:42:00.9741166Z * [new branch] pianpwk/nonzero_memo -> origin/pianpwk/nonzero_memo 2025-09-07T06:42:00.9741784Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-09-07T06:42:00.9742888Z * [new branch] pianpwk/oblivious_slice_forward -> origin/pianpwk/oblivious_slice_forward 2025-09-07T06:42:00.9743200Z * [new branch] pianpwk/oblivious_where -> origin/pianpwk/oblivious_where 2025-09-07T06:42:00.9744149Z * [new branch] pianpwk/param_static_pgo -> origin/pianpwk/param_static_pgo 2025-09-07T06:42:00.9744482Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-09-07T06:42:00.9746437Z * [new branch] pianpwk/remove_guard_fail_break -> origin/pianpwk/remove_guard_fail_break 2025-09-07T06:42:00.9746651Z * [new branch] pianpwk/slice_fresh_symbols -> origin/pianpwk/slice_fresh_symbols 2025-09-07T06:42:00.9746846Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-09-07T06:42:00.9753874Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-09-07T06:42:00.9754231Z * [new branch] pianpwk/test_slice_fake_impl -> origin/pianpwk/test_slice_fake_impl 2025-09-07T06:42:00.9754443Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-09-07T06:42:00.9754622Z * [new branch] pianpwk/unbacked_channels_last -> origin/pianpwk/unbacked_channels_last 2025-09-07T06:42:00.9754936Z * [new branch] pianpwk/unbacked_safe_conv1d -> origin/pianpwk/unbacked_safe_conv1d 2025-09-07T06:42:00.9755103Z * [new branch] pianpwk/unbacked_sdpa_flash -> origin/pianpwk/unbacked_sdpa_flash 2025-09-07T06:42:00.9755275Z * [new branch] pianpwk/unbacked_should_swap -> origin/pianpwk/unbacked_should_swap 2025-09-07T06:42:00.9755826Z * [new branch] pianpwk/unbacked_should_swap_2 -> origin/pianpwk/unbacked_should_swap_2 2025-09-07T06:42:00.9756229Z * [new branch] pianpwk/unbacked_slice_binding -> origin/pianpwk/unbacked_slice_binding 2025-09-07T06:42:00.9756427Z * [new branch] pianpwk/unbacked_slice_forward -> origin/pianpwk/unbacked_slice_forward 2025-09-07T06:42:00.9756593Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-09-07T06:42:00.9756759Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-09-07T06:42:00.9756955Z * [new branch] pianpwk/whitelist_optimizer -> origin/pianpwk/whitelist_optimizer 2025-09-07T06:42:00.9757116Z * [new branch] pin-torchao -> origin/pin-torchao 2025-09-07T06:42:00.9761725Z * [new branch] piz/fall_back_missing_0716 -> origin/piz/fall_back_missing_0716 2025-09-07T06:42:00.9762059Z * [new branch] piz/improve_scatter_0808 -> origin/piz/improve_scatter_0808 2025-09-07T06:42:00.9762228Z * [new branch] pool-separate -> origin/pool-separate 2025-09-07T06:42:00.9762457Z * [new branch] pr-156087 -> origin/pr-156087 2025-09-07T06:42:00.9767465Z * [new branch] pr/131860 -> origin/pr/131860 2025-09-07T06:42:00.9767807Z * [new branch] predispatch_to -> origin/predispatch_to 2025-09-07T06:42:00.9767984Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-09-07T06:42:00.9768126Z * [new branch] pyobjectslot -> origin/pyobjectslot 2025-09-07T06:42:00.9768456Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-09-07T06:42:00.9768792Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-09-07T06:42:00.9769405Z * [new branch] quint-bits -> origin/quint-bits 2025-09-07T06:42:00.9769691Z * [new branch] release/1.10 -> origin/release/1.10 2025-09-07T06:42:00.9769880Z * [new branch] release/1.11 -> origin/release/1.11 2025-09-07T06:42:00.9770005Z * [new branch] release/1.12 -> origin/release/1.12 2025-09-07T06:42:00.9770128Z * [new branch] release/1.13 -> origin/release/1.13 2025-09-07T06:42:00.9770265Z * [new branch] release/1.4 -> origin/release/1.4 2025-09-07T06:42:00.9775401Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-09-07T06:42:00.9775739Z * [new branch] release/1.5 -> origin/release/1.5 2025-09-07T06:42:00.9775891Z * [new branch] release/1.6 -> origin/release/1.6 2025-09-07T06:42:00.9776021Z * [new branch] release/1.7 -> origin/release/1.7 2025-09-07T06:42:00.9776140Z * [new branch] release/1.8 -> origin/release/1.8 2025-09-07T06:42:00.9776263Z * [new branch] release/1.9 -> origin/release/1.9 2025-09-07T06:42:00.9776516Z * [new branch] release/2.0 -> origin/release/2.0 2025-09-07T06:42:00.9777003Z * [new branch] release/2.1 -> origin/release/2.1 2025-09-07T06:42:00.9777161Z * [new branch] release/2.2 -> origin/release/2.2 2025-09-07T06:42:00.9777296Z * [new branch] release/2.3 -> origin/release/2.3 2025-09-07T06:42:00.9777438Z * [new branch] release/2.4 -> origin/release/2.4 2025-09-07T06:42:00.9781749Z * [new branch] release/2.5 -> origin/release/2.5 2025-09-07T06:42:00.9781922Z * [new branch] release/2.6 -> origin/release/2.6 2025-09-07T06:42:00.9782159Z * [new branch] release/2.7 -> origin/release/2.7 2025-09-07T06:42:00.9782294Z * [new branch] release/2.8 -> origin/release/2.8 2025-09-07T06:42:00.9782605Z * [new branch] release_notes -> origin/release_notes 2025-09-07T06:42:00.9785112Z * [new branch] remove-actionable-label -> origin/remove-actionable-label 2025-09-07T06:42:00.9785466Z * [new branch] remove-ao -> origin/remove-ao 2025-09-07T06:42:00.9785851Z * [new branch] removedeprecatedvllmtest -> origin/removedeprecatedvllmtest 2025-09-07T06:42:00.9786118Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-09-07T06:42:00.9786355Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-09-07T06:42:00.9786596Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-09-07T06:42:00.9786825Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-09-07T06:42:00.9787063Z * [new branch] replace-pytorch-labs-20250812-204125 -> origin/replace-pytorch-labs-20250812-204125 2025-09-07T06:42:00.9787303Z * [new branch] replace-pytorch-labs-20250812-205624 -> origin/replace-pytorch-labs-20250812-205624 2025-09-07T06:42:00.9794583Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-09-07T06:42:00.9794995Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-09-07T06:42:00.9795454Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-09-07T06:42:00.9795921Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-09-07T06:42:00.9796195Z * [new branch] rocm-monitoring -> origin/rocm-monitoring 2025-09-07T06:42:00.9796389Z * [new branch] ruisi/relax_memory -> origin/ruisi/relax_memory 2025-09-07T06:42:00.9797096Z * [new branch] run-torchbench-smoke-test-h100 -> origin/run-torchbench-smoke-test-h100 2025-09-07T06:42:00.9797393Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-09-07T06:42:00.9797594Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-09-07T06:42:00.9797773Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-09-07T06:42:00.9797912Z * [new branch] rzou/njt -> origin/rzou/njt 2025-09-07T06:42:00.9798033Z * [new branch] rzou/pca -> origin/rzou/pca 2025-09-07T06:42:00.9803908Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-09-07T06:42:00.9804267Z * [new branch] rzou/setup_context -> origin/rzou/setup_context 2025-09-07T06:42:00.9804658Z * [new branch] sanchitintel/refactor_aten_int8_woq_gemm -> origin/sanchitintel/refactor_aten_int8_woq_gemm 2025-09-07T06:42:00.9805115Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-09-07T06:42:00.9805330Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-09-07T06:42:00.9805466Z * [new branch] save -> origin/save 2025-09-07T06:42:00.9805605Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-09-07T06:42:00.9805766Z * [new branch] seemethere-patch-1 -> origin/seemethere-patch-1 2025-09-07T06:42:00.9805902Z * [new branch] setupvllm -> origin/setupvllm 2025-09-07T06:42:00.9806044Z * [new branch] share_and_pin_fork -> origin/share_and_pin_fork 2025-09-07T06:42:00.9811102Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-09-07T06:42:00.9811433Z * [new branch] shikaili_fp8_allgather -> origin/shikaili_fp8_allgather 2025-09-07T06:42:00.9811642Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-09-07T06:42:00.9811796Z * [new branch] shoumikhin-patch-12 -> origin/shoumikhin-patch-12 2025-09-07T06:42:00.9812095Z * [new branch] simplify-fq-per-channel -> origin/simplify-fq-per-channel 2025-09-07T06:42:00.9812341Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-09-07T06:42:00.9813031Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-09-07T06:42:00.9813204Z * [new branch] sqzhang/flight4 -> origin/sqzhang/flight4 2025-09-07T06:42:00.9813370Z * [new branch] sqzhang/flight4plus -> origin/sqzhang/flight4plus 2025-09-07T06:42:00.9813562Z * [new branch] sraikund/record_funct_test -> origin/sraikund/record_funct_test 2025-09-07T06:42:00.9813703Z * [new branch] sraikund16/test -> origin/sraikund16/test 2025-09-07T06:42:00.9813895Z * [new branch] stablize-compilation-time -> origin/stablize-compilation-time 2025-09-07T06:42:00.9819870Z * [new branch] standalone-templates -> origin/standalone-templates 2025-09-07T06:42:00.9820550Z * [new branch] standalone_package_weights -> origin/standalone_package_weights 2025-09-07T06:42:00.9820723Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-09-07T06:42:00.9820875Z * [new branch] subgraph_fuse -> origin/subgraph_fuse 2025-09-07T06:42:00.9821072Z * [new branch] support-uv-in-collect_env -> origin/support-uv-in-collect_env 2025-09-07T06:42:00.9821209Z * [new branch] sve-poc -> origin/sve-poc 2025-09-07T06:42:00.9821371Z * [new branch] svekars-patch-1 -> origin/svekars-patch-1 2025-09-07T06:42:00.9821500Z * [new branch] switch-bn -> origin/switch-bn 2025-09-07T06:42:00.9821678Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-09-07T06:42:00.9825285Z * [new branch] tenpercent/ck_rocm_ci_v3 -> origin/tenpercent/ck_rocm_ci_v3 2025-09-07T06:42:00.9825501Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-09-07T06:42:00.9831859Z * [new branch] test-7054 -> origin/test-7054 2025-09-07T06:42:00.9837316Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-09-07T06:42:00.9842870Z * [new branch] test-myst-markdown-docstring -> origin/test-myst-markdown-docstring 2025-09-07T06:42:00.9843083Z * [new branch] test-old -> origin/test-old 2025-09-07T06:42:00.9843303Z * [new branch] test-vec-migration-internally -> origin/test-vec-migration-internally 2025-09-07T06:42:00.9843455Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-09-07T06:42:00.9843583Z * [new branch] test/inductor -> origin/test/inductor 2025-09-07T06:42:00.9843751Z * [new branch] tianren/flex_paged_attn_fix -> origin/tianren/flex_paged_attn_fix 2025-09-07T06:42:00.9843952Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-09-07T06:42:00.9844090Z * [new branch] tianren/test -> origin/tianren/test 2025-09-07T06:42:00.9844241Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-09-07T06:42:00.9844364Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-09-07T06:42:00.9844725Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-09-07T06:42:00.9844887Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-09-07T06:42:00.9845024Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-09-07T06:42:00.9845150Z * [new branch] tree_vec_base -> origin/tree_vec_base 2025-09-07T06:42:00.9845280Z * [new branch] triton-update -> origin/triton-update 2025-09-07T06:42:00.9845423Z * [new branch] triton_kernel -> origin/triton_kernel 2025-09-07T06:42:00.9845563Z * [new branch] triton_kernel_perf -> origin/triton_kernel_perf 2025-09-07T06:42:00.9845689Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-09-07T06:42:00.9845888Z * [new branch] tweak-transformer-dependabot -> origin/tweak-transformer-dependabot 2025-09-07T06:42:00.9846011Z * [new branch] type_dec -> origin/type_dec 2025-09-07T06:42:00.9846188Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-09-07T06:42:00.9846435Z * [new branch] update-audio-commit-hash/16818882925-1712-1 -> origin/update-audio-commit-hash/16818882925-1712-1 2025-09-07T06:42:00.9846717Z * [new branch] update-audio-commit-hash/16895560422-1720-1 -> origin/update-audio-commit-hash/16895560422-1720-1 2025-09-07T06:42:00.9847543Z * [new branch] update-audio-commit-hash/16924174496-1738-1 -> origin/update-audio-commit-hash/16924174496-1738-1 2025-09-07T06:42:00.9847833Z * [new branch] update-audio-commit-hash/17002010821-1749-1 -> origin/update-audio-commit-hash/17002010821-1749-1 2025-09-07T06:42:00.9848096Z * [new branch] update-audio-commit-hash/17056004427-1766-1 -> origin/update-audio-commit-hash/17056004427-1766-1 2025-09-07T06:42:00.9848355Z * [new branch] update-audio-commit-hash/17085054029-1767-1 -> origin/update-audio-commit-hash/17085054029-1767-1 2025-09-07T06:42:00.9855307Z * [new branch] update-audio-commit-hash/17142507405-1771-1 -> origin/update-audio-commit-hash/17142507405-1771-1 2025-09-07T06:42:00.9855586Z * [new branch] update-audio-commit-hash/17168762740-1773-1 -> origin/update-audio-commit-hash/17168762740-1773-1 2025-09-07T06:42:00.9855846Z * [new branch] update-audio-commit-hash/17311174639-1780-1 -> origin/update-audio-commit-hash/17311174639-1780-1 2025-09-07T06:42:00.9856098Z * [new branch] update-audio-commit-hash/17336898740-1781-1 -> origin/update-audio-commit-hash/17336898740-1781-1 2025-09-07T06:42:00.9856350Z * [new branch] update-audio-commit-hash/17389727684-1786-1 -> origin/update-audio-commit-hash/17389727684-1786-1 2025-09-07T06:42:00.9856580Z * [new branch] update-audio-commit-hash/17449538142-1790-1 -> origin/update-audio-commit-hash/17449538142-1790-1 2025-09-07T06:42:00.9856893Z * [new branch] update-audio-commit-hash/17507351808-1794-1 -> origin/update-audio-commit-hash/17507351808-1794-1 2025-09-07T06:42:00.9857088Z * [new branch] update-dynamic-shapes-doc -> origin/update-dynamic-shapes-doc 2025-09-07T06:42:00.9857363Z * [new branch] update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1 2025-09-07T06:42:00.9857611Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-09-07T06:42:00.9857844Z * [new branch] update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1 2025-09-07T06:42:00.9858073Z * [new branch] update-vllm-commit-hash/16737365217-1704-1 -> origin/update-vllm-commit-hash/16737365217-1704-1 2025-09-07T06:42:00.9858458Z * [new branch] update-vllm-commit-hash/16843157111-1713-1 -> origin/update-vllm-commit-hash/16843157111-1713-1 2025-09-07T06:42:00.9858695Z * [new branch] update-vllm-commit-hash/16855312394-1714-1 -> origin/update-vllm-commit-hash/16855312394-1714-1 2025-09-07T06:42:00.9859795Z * [new branch] update-vllm-commit-hash/16924174496-1738-1 -> origin/update-vllm-commit-hash/16924174496-1738-1 2025-09-07T06:42:00.9860051Z * [new branch] update-vllm-commit-hash/16952608705-1745-1 -> origin/update-vllm-commit-hash/16952608705-1745-1 2025-09-07T06:42:00.9860281Z * [new branch] update-vllm-commit-hash/16979836546-1748-1 -> origin/update-vllm-commit-hash/16979836546-1748-1 2025-09-07T06:42:00.9860638Z * [new branch] update-vllm-commit-hash/17014576881-1756-1 -> origin/update-vllm-commit-hash/17014576881-1756-1 2025-09-07T06:42:00.9860886Z * [new branch] update-vllm-commit-hash/17027830869-1761-1 -> origin/update-vllm-commit-hash/17027830869-1761-1 2025-09-07T06:42:00.9861261Z * [new branch] update-vllm-commit-hash/17056004427-1766-1 -> origin/update-vllm-commit-hash/17056004427-1766-1 2025-09-07T06:42:00.9861514Z * [new branch] update-vllm-commit-hash/17085054029-1767-1 -> origin/update-vllm-commit-hash/17085054029-1767-1 2025-09-07T06:42:00.9861897Z * [new branch] update-vllm-commit-hash/17113610216-1768-1 -> origin/update-vllm-commit-hash/17113610216-1768-1 2025-09-07T06:42:00.9863294Z * [new branch] update-vllm-commit-hash/17142507405-1771-1 -> origin/update-vllm-commit-hash/17142507405-1771-1 2025-09-07T06:42:00.9863758Z * [new branch] update-vllm-commit-hash/17181878974-1774-1 -> origin/update-vllm-commit-hash/17181878974-1774-1 2025-09-07T06:42:00.9864301Z * [new branch] update-vllm-commit-hash/17311174639-1780-1 -> origin/update-vllm-commit-hash/17311174639-1780-1 2025-09-07T06:42:00.9866443Z * [new branch] update-vllm-commit-hash/17336898740-1781-1 -> origin/update-vllm-commit-hash/17336898740-1781-1 2025-09-07T06:42:00.9866879Z * [new branch] update-vllm-commit-hash/17364352302-1785-1 -> origin/update-vllm-commit-hash/17364352302-1785-1 2025-09-07T06:42:00.9867261Z * [new branch] update-vllm-commit-hash/17389727684-1786-1 -> origin/update-vllm-commit-hash/17389727684-1786-1 2025-09-07T06:42:00.9867636Z * [new branch] update-vllm-commit-hash/17449538142-1790-1 -> origin/update-vllm-commit-hash/17449538142-1790-1 2025-09-07T06:42:00.9867966Z * [new branch] update-vllm-commit-hash/17480069797-1791-1 -> origin/update-vllm-commit-hash/17480069797-1791-1 2025-09-07T06:42:00.9868494Z * [new branch] update-vllm-commit-hash/17507351808-1794-1 -> origin/update-vllm-commit-hash/17507351808-1794-1 2025-09-07T06:42:00.9870137Z * [new branch] update-xla-commit-hash/16873912760-198-1 -> origin/update-xla-commit-hash/16873912760-198-1 2025-09-07T06:42:00.9870545Z * [new branch] update-xla-commit-hash/17034266655-199-1 -> origin/update-xla-commit-hash/17034266655-199-1 2025-09-07T06:42:00.9870852Z * [new branch] update-xla-commit-hash/17202464405-200-1 -> origin/update-xla-commit-hash/17202464405-200-1 2025-09-07T06:42:00.9873691Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-09-07T06:42:00.9874067Z * [new branch] update_executorch_pin -> origin/update_executorch_pin 2025-09-07T06:42:00.9874380Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-09-07T06:42:00.9874629Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-09-07T06:42:00.9875192Z * [new branch] update_slow_tests_1752478971 -> origin/update_slow_tests_1752478971 2025-09-07T06:42:00.9875390Z * [new branch] update_slow_tests_1755502951 -> origin/update_slow_tests_1755502951 2025-09-07T06:42:00.9876313Z * [new branch] update_slow_tests_1756107664 -> origin/update_slow_tests_1756107664 2025-09-07T06:42:00.9876545Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-09-07T06:42:00.9879024Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-09-07T06:42:00.9879370Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-09-07T06:42:00.9879523Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-09-07T06:42:00.9879752Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-09-07T06:42:00.9881116Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-09-07T06:42:00.9883945Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-09-07T06:42:00.9884258Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-09-07T06:42:00.9884484Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-09-07T06:42:00.9884654Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-09-07T06:42:00.9884796Z * [new branch] validate_fn -> origin/validate_fn 2025-09-07T06:42:00.9886367Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-09-07T06:42:00.9886561Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-09-07T06:42:00.9889622Z * [new branch] viable/strict -> origin/viable/strict 2025-09-07T06:42:00.9889964Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-09-07T06:42:00.9890155Z * [new branch] vllmpin -> origin/vllmpin 2025-09-07T06:42:00.9890456Z * [new branch] wdvr/conda_devcontainer -> origin/wdvr/conda_devcontainer 2025-09-07T06:42:00.9892129Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-09-07T06:42:00.9892496Z * [new branch] weight_sharing_cpp -> origin/weight_sharing_cpp 2025-09-07T06:42:00.9895900Z * [new branch] whc/flight4 -> origin/whc/flight4 2025-09-07T06:42:00.9896065Z * [new branch] whc/flight51 -> origin/whc/flight51 2025-09-07T06:42:00.9896485Z * [new branch] whc/flight53 -> origin/whc/flight53 2025-09-07T06:42:00.9896625Z * [new branch] whc/stage2 -> origin/whc/stage2 2025-09-07T06:42:00.9896918Z * [new branch] whc/uneven -> origin/whc/uneven 2025-09-07T06:42:00.9897784Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-09-07T06:42:00.9898283Z * [new branch] win_warnings -> origin/win_warnings 2025-09-07T06:42:00.9899351Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-09-07T06:42:00.9899664Z * [new branch] workonoldcommit -> origin/workonoldcommit 2025-09-07T06:42:00.9900997Z * [new branch] wychi-autotune-prune-configs-by-shared-mem -> origin/wychi-autotune-prune-configs-by-shared-mem 2025-09-07T06:42:00.9901737Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-09-07T06:42:00.9902158Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-09-07T06:42:00.9903240Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-09-07T06:42:00.9903534Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-09-07T06:42:00.9905896Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-09-07T06:42:00.9906086Z * [new branch] xmfan/ca_api -> origin/xmfan/ca_api 2025-09-07T06:42:00.9906407Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-09-07T06:42:00.9906589Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-09-07T06:42:00.9910865Z * [new branch] xmfan/ca_cudagraphs -> origin/xmfan/ca_cudagraphs 2025-09-07T06:42:00.9911054Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-09-07T06:42:00.9911198Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-09-07T06:42:00.9911397Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-09-07T06:42:00.9911552Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-09-07T06:42:00.9911700Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-09-07T06:42:00.9912181Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-09-07T06:42:00.9913167Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-09-07T06:42:00.9913349Z * [new branch] xmfan/ca_mem_base -> origin/xmfan/ca_mem_base 2025-09-07T06:42:00.9916987Z * [new branch] xmfan/ca_mem_fix -> origin/xmfan/ca_mem_fix 2025-09-07T06:42:00.9917175Z * [new branch] xmfan/ca_memory_fix -> origin/xmfan/ca_memory_fix 2025-09-07T06:42:00.9917361Z * [new branch] xmfan/ca_memory_fix_rebased -> origin/xmfan/ca_memory_fix_rebased 2025-09-07T06:42:00.9917698Z * [new branch] xmfan/ca_memory_fix_rebased2 -> origin/xmfan/ca_memory_fix_rebased2 2025-09-07T06:42:00.9917856Z * [new branch] xmfan/ca_move_to_cuda -> origin/xmfan/ca_move_to_cuda 2025-09-07T06:42:00.9918174Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-09-07T06:42:00.9921230Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-09-07T06:42:00.9927521Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-09-07T06:42:00.9932576Z * [new branch] xmfan/ca_scalar -> origin/xmfan/ca_scalar 2025-09-07T06:42:00.9932779Z * [new branch] xmfan/ca_subclass_mem_fix -> origin/xmfan/ca_subclass_mem_fix 2025-09-07T06:42:00.9932952Z * [new branch] xmfan/ca_warm_mem -> origin/xmfan/ca_warm_mem 2025-09-07T06:42:00.9933110Z * [new branch] xmfan/ca_warm_mem_base -> origin/xmfan/ca_warm_mem_base 2025-09-07T06:42:00.9933287Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-09-07T06:42:00.9933426Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-09-07T06:42:00.9933554Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-09-07T06:42:00.9933684Z * [new branch] xmfan/cacu_may27 -> origin/xmfan/cacu_may27 2025-09-07T06:42:00.9933864Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-09-07T06:42:00.9934046Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-09-07T06:42:00.9934188Z * [new branch] xmfan/issue_123374 -> origin/xmfan/issue_123374 2025-09-07T06:42:00.9934461Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-09-07T06:42:00.9934729Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-09-07T06:42:00.9934876Z * [new branch] xmfan/segfault_test -> origin/xmfan/segfault_test 2025-09-07T06:42:00.9935023Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-09-07T06:42:00.9935365Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-09-07T06:42:00.9935495Z * [new branch] xmfan/test -> origin/xmfan/test 2025-09-07T06:42:00.9935676Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-09-07T06:42:00.9935828Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-09-07T06:42:00.9936019Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-09-07T06:42:00.9936546Z * [new branch] yihan_quantization -> origin/yihan_quantization 2025-09-07T06:42:00.9936829Z * [new branch] yiming/add_jit_trace_benchmark -> origin/yiming/add_jit_trace_benchmark 2025-09-07T06:42:00.9937087Z * [new branch] yiming/add_nativert_benchmark -> origin/yiming/add_nativert_benchmark 2025-09-07T06:42:00.9937321Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-09-07T06:42:00.9937546Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-09-07T06:42:00.9937717Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-09-07T06:42:00.9937865Z * [new branch] zainr/git-push-v2 -> origin/zainr/git-push-v2 2025-09-07T06:42:00.9938140Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-09-07T06:42:00.9938286Z * [new branch] zainr/test -> origin/zainr/test 2025-09-07T06:42:00.9938481Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-09-07T06:42:00.9942149Z * [new branch] zainr/unstable -> origin/zainr/unstable 2025-09-07T06:42:00.9942495Z * [new branch] zainr/unstable-xla -> origin/zainr/unstable-xla 2025-09-07T06:42:00.9942695Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-09-07T06:42:00.9942971Z * [new branch] zb2p -> origin/zb2p 2025-09-07T06:42:00.9943227Z * [new branch] zero_grad_optimization -> origin/zero_grad_optimization 2025-09-07T06:42:00.9943406Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-09-07T06:42:00.9943723Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-09-07T06:42:00.9946178Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-09-07T06:42:00.9946521Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-09-07T06:42:00.9946914Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-09-07T06:42:00.9947499Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-09-07T06:42:00.9948400Z * [new tag] ciflow/binaries/156049 -> ciflow/binaries/156049 2025-09-07T06:42:00.9948920Z * [new tag] ciflow/binaries/156712 -> ciflow/binaries/156712 2025-09-07T06:42:00.9949141Z * [new tag] ciflow/binaries/157432 -> ciflow/binaries/157432 2025-09-07T06:42:00.9949491Z * [new tag] ciflow/binaries/157685 -> ciflow/binaries/157685 2025-09-07T06:42:00.9951416Z * [new tag] ciflow/binaries/157689 -> ciflow/binaries/157689 2025-09-07T06:42:00.9951585Z * [new tag] ciflow/binaries/158104 -> ciflow/binaries/158104 2025-09-07T06:42:00.9951749Z * [new tag] ciflow/binaries/160229 -> ciflow/binaries/160229 2025-09-07T06:42:00.9955533Z * [new tag] ciflow/binaries/160720 -> ciflow/binaries/160720 2025-09-07T06:42:00.9955699Z * [new tag] ciflow/binaries/162080 -> ciflow/binaries/162080 2025-09-07T06:42:00.9955825Z * [new tag] ciflow/binaries/162329 -> ciflow/binaries/162329 2025-09-07T06:42:00.9956168Z * [new tag] ciflow/binaries_libtorch/156049 -> ciflow/binaries_libtorch/156049 2025-09-07T06:42:00.9956324Z * [new tag] ciflow/binaries_libtorch/156711 -> ciflow/binaries_libtorch/156711 2025-09-07T06:42:00.9956480Z * [new tag] ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432 2025-09-07T06:42:00.9956622Z * [new tag] ciflow/binaries_wheel/156049 -> ciflow/binaries_wheel/156049 2025-09-07T06:42:00.9956770Z * [new tag] ciflow/binaries_wheel/156711 -> ciflow/binaries_wheel/156711 2025-09-07T06:42:00.9956908Z * [new tag] ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432 2025-09-07T06:42:00.9957044Z * [new tag] ciflow/binaries_wheel/162136 -> ciflow/binaries_wheel/162136 2025-09-07T06:42:00.9959046Z * [new tag] ciflow/binaries_wheel/162252 -> ciflow/binaries_wheel/162252 2025-09-07T06:42:00.9959235Z * [new tag] ciflow/binaries_wheel/162325 -> ciflow/binaries_wheel/162325 2025-09-07T06:42:00.9959424Z * [new tag] ciflow/h100-distributed/156703 -> ciflow/h100-distributed/156703 2025-09-07T06:42:00.9959692Z * [new tag] ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635 2025-09-07T06:42:00.9959898Z * [new tag] ciflow/h100-symm-mem/161984 -> ciflow/h100-symm-mem/161984 2025-09-07T06:42:00.9960028Z * [new tag] ciflow/h100-symm-mem/162003 -> ciflow/h100-symm-mem/162003 2025-09-07T06:42:00.9960383Z * [new tag] ciflow/h100-symm-mem/162011 -> ciflow/h100-symm-mem/162011 2025-09-07T06:42:00.9960601Z * [new tag] ciflow/h100-symm-mem/162026 -> ciflow/h100-symm-mem/162026 2025-09-07T06:42:00.9961180Z * [new tag] ciflow/h100-symm-mem/162033 -> ciflow/h100-symm-mem/162033 2025-09-07T06:42:00.9961364Z * [new tag] ciflow/h100-symm-mem/162040 -> ciflow/h100-symm-mem/162040 2025-09-07T06:42:00.9966364Z * [new tag] ciflow/h100-symm-mem/162041 -> ciflow/h100-symm-mem/162041 2025-09-07T06:42:00.9966529Z * [new tag] ciflow/h100-symm-mem/162142 -> ciflow/h100-symm-mem/162142 2025-09-07T06:42:00.9966680Z * [new tag] ciflow/h100-symm-mem/162150 -> ciflow/h100-symm-mem/162150 2025-09-07T06:42:00.9966812Z * [new tag] ciflow/h100-symm-mem/162243 -> ciflow/h100-symm-mem/162243 2025-09-07T06:42:00.9966953Z * [new tag] ciflow/h100-symm-mem/162320 -> ciflow/h100-symm-mem/162320 2025-09-07T06:42:00.9967110Z * [new tag] ciflow/h100/159158 -> ciflow/h100/159158 2025-09-07T06:42:00.9967232Z * [new tag] ciflow/h100/160480 -> ciflow/h100/160480 2025-09-07T06:42:00.9967342Z * [new tag] ciflow/h100/161749 -> ciflow/h100/161749 2025-09-07T06:42:00.9967454Z * [new tag] ciflow/h100/162022 -> ciflow/h100/162022 2025-09-07T06:42:00.9967583Z * [new tag] ciflow/h100/162278 -> ciflow/h100/162278 2025-09-07T06:42:00.9967851Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/156592 -> ciflow/inductor-perf-test-nightly-rocm/156592 2025-09-07T06:42:00.9968570Z * [new tag] ciflow/inductor-perf-test-nightly/156592 -> ciflow/inductor-perf-test-nightly/156592 2025-09-07T06:42:00.9969202Z * [new tag] ciflow/inductor-periodic/162063 -> ciflow/inductor-periodic/162063 2025-09-07T06:42:00.9969443Z * [new tag] ciflow/inductor-periodic/162227 -> ciflow/inductor-periodic/162227 2025-09-07T06:42:00.9969618Z * [new tag] ciflow/inductor-periodic/162323 -> ciflow/inductor-periodic/162323 2025-09-07T06:42:00.9969778Z * [new tag] ciflow/inductor-rocm/154170 -> ciflow/inductor-rocm/154170 2025-09-07T06:42:00.9969923Z * [new tag] ciflow/inductor-rocm/159146 -> ciflow/inductor-rocm/159146 2025-09-07T06:42:00.9970233Z * [new tag] ciflow/inductor-rocm/159158 -> ciflow/inductor-rocm/159158 2025-09-07T06:42:00.9970380Z * [new tag] ciflow/inductor-rocm/161715 -> ciflow/inductor-rocm/161715 2025-09-07T06:42:00.9970523Z * [new tag] ciflow/inductor-rocm/162053 -> ciflow/inductor-rocm/162053 2025-09-07T06:42:00.9973835Z * [new tag] ciflow/inductor-rocm/162056 -> ciflow/inductor-rocm/162056 2025-09-07T06:42:00.9974021Z * [new tag] ciflow/inductor/137400 -> ciflow/inductor/137400 2025-09-07T06:42:00.9974282Z * [new tag] ciflow/inductor/148180 -> ciflow/inductor/148180 2025-09-07T06:42:00.9974421Z * [new tag] ciflow/inductor/148328 -> ciflow/inductor/148328 2025-09-07T06:42:00.9974579Z * [new tag] ciflow/inductor/148484 -> ciflow/inductor/148484 2025-09-07T06:42:00.9974714Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-09-07T06:42:00.9974869Z * [new tag] ciflow/inductor/152624 -> ciflow/inductor/152624 2025-09-07T06:42:00.9975043Z * [new tag] ciflow/inductor/154694 -> ciflow/inductor/154694 2025-09-07T06:42:00.9975177Z * [new tag] ciflow/inductor/156049 -> ciflow/inductor/156049 2025-09-07T06:42:00.9975312Z * [new tag] ciflow/inductor/156592 -> ciflow/inductor/156592 2025-09-07T06:42:00.9975445Z * [new tag] ciflow/inductor/157635 -> ciflow/inductor/157635 2025-09-07T06:42:00.9981676Z * [new tag] ciflow/inductor/157685 -> ciflow/inductor/157685 2025-09-07T06:42:00.9981875Z * [new tag] ciflow/inductor/157686 -> ciflow/inductor/157686 2025-09-07T06:42:00.9982013Z * [new tag] ciflow/inductor/157689 -> ciflow/inductor/157689 2025-09-07T06:42:00.9982154Z * [new tag] ciflow/inductor/157699 -> ciflow/inductor/157699 2025-09-07T06:42:00.9982295Z * [new tag] ciflow/inductor/157743 -> ciflow/inductor/157743 2025-09-07T06:42:00.9982425Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-09-07T06:42:00.9982563Z * [new tag] ciflow/inductor/158091 -> ciflow/inductor/158091 2025-09-07T06:42:00.9982694Z * [new tag] ciflow/inductor/158104 -> ciflow/inductor/158104 2025-09-07T06:42:00.9982830Z * [new tag] ciflow/inductor/158404 -> ciflow/inductor/158404 2025-09-07T06:42:00.9982960Z * [new tag] ciflow/inductor/158647 -> ciflow/inductor/158647 2025-09-07T06:42:00.9983103Z * [new tag] ciflow/inductor/158932 -> ciflow/inductor/158932 2025-09-07T06:42:00.9983242Z * [new tag] ciflow/inductor/159146 -> ciflow/inductor/159146 2025-09-07T06:42:00.9983378Z * [new tag] ciflow/inductor/159158 -> ciflow/inductor/159158 2025-09-07T06:42:00.9983525Z * [new tag] ciflow/inductor/159274 -> ciflow/inductor/159274 2025-09-07T06:42:00.9983665Z * [new tag] ciflow/inductor/159664 -> ciflow/inductor/159664 2025-09-07T06:42:00.9983808Z * [new tag] ciflow/inductor/159778 -> ciflow/inductor/159778 2025-09-07T06:42:00.9983947Z * [new tag] ciflow/inductor/159835 -> ciflow/inductor/159835 2025-09-07T06:42:00.9984081Z * [new tag] ciflow/inductor/159944 -> ciflow/inductor/159944 2025-09-07T06:42:00.9984236Z * [new tag] ciflow/inductor/160161 -> ciflow/inductor/160161 2025-09-07T06:42:00.9984366Z * [new tag] ciflow/inductor/160174 -> ciflow/inductor/160174 2025-09-07T06:42:00.9984503Z * [new tag] ciflow/inductor/160323 -> ciflow/inductor/160323 2025-09-07T06:42:00.9984683Z * [new tag] ciflow/inductor/160324 -> ciflow/inductor/160324 2025-09-07T06:42:00.9985846Z * [new tag] ciflow/inductor/160325 -> ciflow/inductor/160325 2025-09-07T06:42:00.9986335Z * [new tag] ciflow/inductor/160326 -> ciflow/inductor/160326 2025-09-07T06:42:00.9991503Z * [new tag] ciflow/inductor/160327 -> ciflow/inductor/160327 2025-09-07T06:42:00.9996985Z * [new tag] ciflow/inductor/160328 -> ciflow/inductor/160328 2025-09-07T06:42:01.0002071Z * [new tag] ciflow/inductor/160329 -> ciflow/inductor/160329 2025-09-07T06:42:01.0002404Z * [new tag] ciflow/inductor/160480 -> ciflow/inductor/160480 2025-09-07T06:42:01.0002555Z * [new tag] ciflow/inductor/160532 -> ciflow/inductor/160532 2025-09-07T06:42:01.0002784Z * [new tag] ciflow/inductor/160539 -> ciflow/inductor/160539 2025-09-07T06:42:01.0003033Z * [new tag] ciflow/inductor/160580 -> ciflow/inductor/160580 2025-09-07T06:42:01.0003253Z * [new tag] ciflow/inductor/160685 -> ciflow/inductor/160685 2025-09-07T06:42:01.0003437Z * [new tag] ciflow/inductor/160686 -> ciflow/inductor/160686 2025-09-07T06:42:01.0003568Z * [new tag] ciflow/inductor/160687 -> ciflow/inductor/160687 2025-09-07T06:42:01.0003691Z * [new tag] ciflow/inductor/160688 -> ciflow/inductor/160688 2025-09-07T06:42:01.0003934Z * [new tag] ciflow/inductor/160690 -> ciflow/inductor/160690 2025-09-07T06:42:01.0008288Z * [new tag] ciflow/inductor/160706 -> ciflow/inductor/160706 2025-09-07T06:42:01.0011070Z * [new tag] ciflow/inductor/160729 -> ciflow/inductor/160729 2025-09-07T06:42:01.0011246Z * [new tag] ciflow/inductor/160798 -> ciflow/inductor/160798 2025-09-07T06:42:01.0011416Z * [new tag] ciflow/inductor/160836 -> ciflow/inductor/160836 2025-09-07T06:42:01.0011648Z * [new tag] ciflow/inductor/160843 -> ciflow/inductor/160843 2025-09-07T06:42:01.0011814Z * [new tag] ciflow/inductor/160869 -> ciflow/inductor/160869 2025-09-07T06:42:01.0011966Z * [new tag] ciflow/inductor/160920 -> ciflow/inductor/160920 2025-09-07T06:42:01.0012117Z * [new tag] ciflow/inductor/160943 -> ciflow/inductor/160943 2025-09-07T06:42:01.0012256Z * [new tag] ciflow/inductor/161092 -> ciflow/inductor/161092 2025-09-07T06:42:01.0012411Z * [new tag] ciflow/inductor/161093 -> ciflow/inductor/161093 2025-09-07T06:42:01.0012563Z * [new tag] ciflow/inductor/161109 -> ciflow/inductor/161109 2025-09-07T06:42:01.0012717Z * [new tag] ciflow/inductor/161118 -> ciflow/inductor/161118 2025-09-07T06:42:01.0012845Z * [new tag] ciflow/inductor/161178 -> ciflow/inductor/161178 2025-09-07T06:42:01.0013000Z * [new tag] ciflow/inductor/161246 -> ciflow/inductor/161246 2025-09-07T06:42:01.0013150Z * [new tag] ciflow/inductor/161349 -> ciflow/inductor/161349 2025-09-07T06:42:01.0013298Z * [new tag] ciflow/inductor/161350 -> ciflow/inductor/161350 2025-09-07T06:42:01.0013457Z * [new tag] ciflow/inductor/161351 -> ciflow/inductor/161351 2025-09-07T06:42:01.0013606Z * [new tag] ciflow/inductor/161397 -> ciflow/inductor/161397 2025-09-07T06:42:01.0013759Z * [new tag] ciflow/inductor/161404 -> ciflow/inductor/161404 2025-09-07T06:42:01.0013909Z * [new tag] ciflow/inductor/161405 -> ciflow/inductor/161405 2025-09-07T06:42:01.0014072Z * [new tag] ciflow/inductor/161406 -> ciflow/inductor/161406 2025-09-07T06:42:01.0014229Z * [new tag] ciflow/inductor/161410 -> ciflow/inductor/161410 2025-09-07T06:42:01.0014375Z * [new tag] ciflow/inductor/161414 -> ciflow/inductor/161414 2025-09-07T06:42:01.0014647Z * [new tag] ciflow/inductor/161442 -> ciflow/inductor/161442 2025-09-07T06:42:01.0014806Z * [new tag] ciflow/inductor/161458 -> ciflow/inductor/161458 2025-09-07T06:42:01.0014958Z * [new tag] ciflow/inductor/161468 -> ciflow/inductor/161468 2025-09-07T06:42:01.0015104Z * [new tag] ciflow/inductor/161469 -> ciflow/inductor/161469 2025-09-07T06:42:01.0015249Z * [new tag] ciflow/inductor/161485 -> ciflow/inductor/161485 2025-09-07T06:42:01.0015424Z * [new tag] ciflow/inductor/161499 -> ciflow/inductor/161499 2025-09-07T06:42:01.0015577Z * [new tag] ciflow/inductor/161534 -> ciflow/inductor/161534 2025-09-07T06:42:01.0015734Z * [new tag] ciflow/inductor/161595 -> ciflow/inductor/161595 2025-09-07T06:42:01.0015883Z * [new tag] ciflow/inductor/161596 -> ciflow/inductor/161596 2025-09-07T06:42:01.0016032Z * [new tag] ciflow/inductor/161630 -> ciflow/inductor/161630 2025-09-07T06:42:01.0016189Z * [new tag] ciflow/inductor/161667 -> ciflow/inductor/161667 2025-09-07T06:42:01.0016336Z * [new tag] ciflow/inductor/161670 -> ciflow/inductor/161670 2025-09-07T06:42:01.0016488Z * [new tag] ciflow/inductor/161673 -> ciflow/inductor/161673 2025-09-07T06:42:01.0016643Z * [new tag] ciflow/inductor/161674 -> ciflow/inductor/161674 2025-09-07T06:42:01.0016848Z * [new tag] ciflow/inductor/161675 -> ciflow/inductor/161675 2025-09-07T06:42:01.0017001Z * [new tag] ciflow/inductor/161693 -> ciflow/inductor/161693 2025-09-07T06:42:01.0017147Z * [new tag] ciflow/inductor/161695 -> ciflow/inductor/161695 2025-09-07T06:42:01.0017301Z * [new tag] ciflow/inductor/161715 -> ciflow/inductor/161715 2025-09-07T06:42:01.0017451Z * [new tag] ciflow/inductor/161730 -> ciflow/inductor/161730 2025-09-07T06:42:01.0017605Z * [new tag] ciflow/inductor/161732 -> ciflow/inductor/161732 2025-09-07T06:42:01.0019214Z * [new tag] ciflow/inductor/161744 -> ciflow/inductor/161744 2025-09-07T06:42:01.0019788Z * [new tag] ciflow/inductor/161746 -> ciflow/inductor/161746 2025-09-07T06:42:01.0019960Z * [new tag] ciflow/inductor/161747 -> ciflow/inductor/161747 2025-09-07T06:42:01.0020108Z * [new tag] ciflow/inductor/161819 -> ciflow/inductor/161819 2025-09-07T06:42:01.0020246Z * [new tag] ciflow/inductor/161821 -> ciflow/inductor/161821 2025-09-07T06:42:01.0020375Z * [new tag] ciflow/inductor/161828 -> ciflow/inductor/161828 2025-09-07T06:42:01.0020516Z * [new tag] ciflow/inductor/161879 -> ciflow/inductor/161879 2025-09-07T06:42:01.0020674Z * [new tag] ciflow/inductor/161880 -> ciflow/inductor/161880 2025-09-07T06:42:01.0020808Z * [new tag] ciflow/inductor/161881 -> ciflow/inductor/161881 2025-09-07T06:42:01.0023713Z * [new tag] ciflow/inductor/161907 -> ciflow/inductor/161907 2025-09-07T06:42:01.0023991Z * [new tag] ciflow/inductor/161914 -> ciflow/inductor/161914 2025-09-07T06:42:01.0024269Z * [new tag] ciflow/inductor/161924 -> ciflow/inductor/161924 2025-09-07T06:42:01.0024524Z * [new tag] ciflow/inductor/161936 -> ciflow/inductor/161936 2025-09-07T06:42:01.0024686Z * [new tag] ciflow/inductor/161938 -> ciflow/inductor/161938 2025-09-07T06:42:01.0024825Z * [new tag] ciflow/inductor/161939 -> ciflow/inductor/161939 2025-09-07T06:42:01.0024967Z * [new tag] ciflow/inductor/161940 -> ciflow/inductor/161940 2025-09-07T06:42:01.0025115Z * [new tag] ciflow/inductor/161955 -> ciflow/inductor/161955 2025-09-07T06:42:01.0025392Z * [new tag] ciflow/inductor/161957 -> ciflow/inductor/161957 2025-09-07T06:42:01.0030781Z * [new tag] ciflow/inductor/161975 -> ciflow/inductor/161975 2025-09-07T06:42:01.0034084Z * [new tag] ciflow/inductor/161977 -> ciflow/inductor/161977 2025-09-07T06:42:01.0038192Z * [new tag] ciflow/inductor/161978 -> ciflow/inductor/161978 2025-09-07T06:42:01.0042784Z * [new tag] ciflow/inductor/161979 -> ciflow/inductor/161979 2025-09-07T06:42:01.0047826Z * [new tag] ciflow/inductor/161980 -> ciflow/inductor/161980 2025-09-07T06:42:01.0047994Z * [new tag] ciflow/inductor/161988 -> ciflow/inductor/161988 2025-09-07T06:42:01.0048163Z * [new tag] ciflow/inductor/161994 -> ciflow/inductor/161994 2025-09-07T06:42:01.0048291Z * [new tag] ciflow/inductor/162013 -> ciflow/inductor/162013 2025-09-07T06:42:01.0048434Z * [new tag] ciflow/inductor/162014 -> ciflow/inductor/162014 2025-09-07T06:42:01.0048564Z * [new tag] ciflow/inductor/162017 -> ciflow/inductor/162017 2025-09-07T06:42:01.0048700Z * [new tag] ciflow/inductor/162021 -> ciflow/inductor/162021 2025-09-07T06:42:01.0048826Z * [new tag] ciflow/inductor/162023 -> ciflow/inductor/162023 2025-09-07T06:42:01.0048953Z * [new tag] ciflow/inductor/162027 -> ciflow/inductor/162027 2025-09-07T06:42:01.0049288Z * [new tag] ciflow/inductor/162029 -> ciflow/inductor/162029 2025-09-07T06:42:01.0049422Z * [new tag] ciflow/inductor/162030 -> ciflow/inductor/162030 2025-09-07T06:42:01.0049553Z * [new tag] ciflow/inductor/162031 -> ciflow/inductor/162031 2025-09-07T06:42:01.0049681Z * [new tag] ciflow/inductor/162033 -> ciflow/inductor/162033 2025-09-07T06:42:01.0049819Z * [new tag] ciflow/inductor/162052 -> ciflow/inductor/162052 2025-09-07T06:42:01.0049943Z * [new tag] ciflow/inductor/162053 -> ciflow/inductor/162053 2025-09-07T06:42:01.0050067Z * [new tag] ciflow/inductor/162056 -> ciflow/inductor/162056 2025-09-07T06:42:01.0050197Z * [new tag] ciflow/inductor/162063 -> ciflow/inductor/162063 2025-09-07T06:42:01.0050321Z * [new tag] ciflow/inductor/162066 -> ciflow/inductor/162066 2025-09-07T06:42:01.0050453Z * [new tag] ciflow/inductor/162068 -> ciflow/inductor/162068 2025-09-07T06:42:01.0050577Z * [new tag] ciflow/inductor/162081 -> ciflow/inductor/162081 2025-09-07T06:42:01.0050700Z * [new tag] ciflow/inductor/162088 -> ciflow/inductor/162088 2025-09-07T06:42:01.0050829Z * [new tag] ciflow/inductor/162089 -> ciflow/inductor/162089 2025-09-07T06:42:01.0050957Z * [new tag] ciflow/inductor/162094 -> ciflow/inductor/162094 2025-09-07T06:42:01.0051086Z * [new tag] ciflow/inductor/162098 -> ciflow/inductor/162098 2025-09-07T06:42:01.0051206Z * [new tag] ciflow/inductor/162101 -> ciflow/inductor/162101 2025-09-07T06:42:01.0051335Z * [new tag] ciflow/inductor/162102 -> ciflow/inductor/162102 2025-09-07T06:42:01.0051459Z * [new tag] ciflow/inductor/162104 -> ciflow/inductor/162104 2025-09-07T06:42:01.0051585Z * [new tag] ciflow/inductor/162106 -> ciflow/inductor/162106 2025-09-07T06:42:01.0051715Z * [new tag] ciflow/inductor/162108 -> ciflow/inductor/162108 2025-09-07T06:42:01.0051838Z * [new tag] ciflow/inductor/162126 -> ciflow/inductor/162126 2025-09-07T06:42:01.0051967Z * [new tag] ciflow/inductor/162149 -> ciflow/inductor/162149 2025-09-07T06:42:01.0052142Z * [new tag] ciflow/inductor/162164 -> ciflow/inductor/162164 2025-09-07T06:42:01.0052265Z * [new tag] ciflow/inductor/162166 -> ciflow/inductor/162166 2025-09-07T06:42:01.0052393Z * [new tag] ciflow/inductor/162169 -> ciflow/inductor/162169 2025-09-07T06:42:01.0052515Z * [new tag] ciflow/inductor/162170 -> ciflow/inductor/162170 2025-09-07T06:42:01.0052642Z * [new tag] ciflow/inductor/162171 -> ciflow/inductor/162171 2025-09-07T06:42:01.0052768Z * [new tag] ciflow/inductor/162183 -> ciflow/inductor/162183 2025-09-07T06:42:01.0052910Z * [new tag] ciflow/inductor/162189 -> ciflow/inductor/162189 2025-09-07T06:42:01.0053032Z * [new tag] ciflow/inductor/162190 -> ciflow/inductor/162190 2025-09-07T06:42:01.0053153Z * [new tag] ciflow/inductor/162191 -> ciflow/inductor/162191 2025-09-07T06:42:01.0053290Z * [new tag] ciflow/inductor/162194 -> ciflow/inductor/162194 2025-09-07T06:42:01.0053412Z * [new tag] ciflow/inductor/162200 -> ciflow/inductor/162200 2025-09-07T06:42:01.0053542Z * [new tag] ciflow/inductor/162201 -> ciflow/inductor/162201 2025-09-07T06:42:01.0053666Z * [new tag] ciflow/inductor/162208 -> ciflow/inductor/162208 2025-09-07T06:42:01.0053797Z * [new tag] ciflow/inductor/162211 -> ciflow/inductor/162211 2025-09-07T06:42:01.0054046Z * [new tag] ciflow/inductor/162216 -> ciflow/inductor/162216 2025-09-07T06:42:01.0054171Z * [new tag] ciflow/inductor/162220 -> ciflow/inductor/162220 2025-09-07T06:42:01.0054304Z * [new tag] ciflow/inductor/162222 -> ciflow/inductor/162222 2025-09-07T06:42:01.0054426Z * [new tag] ciflow/inductor/162227 -> ciflow/inductor/162227 2025-09-07T06:42:01.0054556Z * [new tag] ciflow/inductor/162238 -> ciflow/inductor/162238 2025-09-07T06:42:01.0054678Z * [new tag] ciflow/inductor/162239 -> ciflow/inductor/162239 2025-09-07T06:42:01.0055308Z * [new tag] ciflow/inductor/162240 -> ciflow/inductor/162240 2025-09-07T06:42:01.0055679Z * [new tag] ciflow/inductor/162244 -> ciflow/inductor/162244 2025-09-07T06:42:01.0055991Z * [new tag] ciflow/inductor/162245 -> ciflow/inductor/162245 2025-09-07T06:42:01.0056147Z * [new tag] ciflow/inductor/162262 -> ciflow/inductor/162262 2025-09-07T06:42:01.0056286Z * [new tag] ciflow/inductor/162275 -> ciflow/inductor/162275 2025-09-07T06:42:01.0056416Z * [new tag] ciflow/inductor/162278 -> ciflow/inductor/162278 2025-09-07T06:42:01.0056557Z * [new tag] ciflow/inductor/162284 -> ciflow/inductor/162284 2025-09-07T06:42:01.0056705Z * [new tag] ciflow/inductor/162286 -> ciflow/inductor/162286 2025-09-07T06:42:01.0057462Z * [new tag] ciflow/inductor/162288 -> ciflow/inductor/162288 2025-09-07T06:42:01.0057630Z * [new tag] ciflow/inductor/162293 -> ciflow/inductor/162293 2025-09-07T06:42:01.0060341Z * [new tag] ciflow/inductor/162294 -> ciflow/inductor/162294 2025-09-07T06:42:01.0060663Z * [new tag] ciflow/inductor/162295 -> ciflow/inductor/162295 2025-09-07T06:42:01.0060981Z * [new tag] ciflow/inductor/162296 -> ciflow/inductor/162296 2025-09-07T06:42:01.0061245Z * [new tag] ciflow/inductor/162298 -> ciflow/inductor/162298 2025-09-07T06:42:01.0061456Z * [new tag] ciflow/inductor/162307 -> ciflow/inductor/162307 2025-09-07T06:42:01.0061598Z * [new tag] ciflow/inductor/162309 -> ciflow/inductor/162309 2025-09-07T06:42:01.0061748Z * [new tag] ciflow/inductor/162311 -> ciflow/inductor/162311 2025-09-07T06:42:01.0062016Z * [new tag] ciflow/inductor/162312 -> ciflow/inductor/162312 2025-09-07T06:42:01.0062158Z * [new tag] ciflow/inductor/162315 -> ciflow/inductor/162315 2025-09-07T06:42:01.0062557Z * [new tag] ciflow/inductor/162316 -> ciflow/inductor/162316 2025-09-07T06:42:01.0063918Z * [new tag] ciflow/inductor/162318 -> ciflow/inductor/162318 2025-09-07T06:42:01.0064087Z * [new tag] ciflow/inductor/162323 -> ciflow/inductor/162323 2025-09-07T06:42:01.0064239Z * [new tag] ciflow/inductor/162341 -> ciflow/inductor/162341 2025-09-07T06:42:01.0066278Z * [new tag] ciflow/inductor/162345 -> ciflow/inductor/162345 2025-09-07T06:42:01.0066470Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-09-07T06:42:01.0066618Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-09-07T06:42:01.0066996Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-09-07T06:42:01.0069489Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-09-07T06:42:01.0069832Z * [new tag] ciflow/linux-aarch64/159737 -> ciflow/linux-aarch64/159737 2025-09-07T06:42:01.0070119Z * [new tag] ciflow/linux-aarch64/160078 -> ciflow/linux-aarch64/160078 2025-09-07T06:42:01.0070374Z * [new tag] ciflow/mps/157553 -> ciflow/mps/157553 2025-09-07T06:42:01.0070659Z * [new tag] ciflow/mps/157635 -> ciflow/mps/157635 2025-09-07T06:42:01.0070795Z * [new tag] ciflow/mps/161988 -> ciflow/mps/161988 2025-09-07T06:42:01.0070914Z * [new tag] ciflow/mps/162108 -> ciflow/mps/162108 2025-09-07T06:42:01.0073188Z * [new tag] ciflow/mps/162153 -> ciflow/mps/162153 2025-09-07T06:42:01.0073473Z * [new tag] ciflow/mps/162281 -> ciflow/mps/162281 2025-09-07T06:42:01.0073645Z * [new tag] ciflow/nightly/156049 -> ciflow/nightly/156049 2025-09-07T06:42:01.0073806Z * [new tag] ciflow/nightly/158104 -> ciflow/nightly/158104 2025-09-07T06:42:01.0073984Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-09-07T06:42:01.0074255Z * [new tag] ciflow/periodic-rocm-mi300/161529 -> ciflow/periodic-rocm-mi300/161529 2025-09-07T06:42:01.0074467Z * [new tag] ciflow/periodic-rocm-mi300/161715 -> ciflow/periodic-rocm-mi300/161715 2025-09-07T06:42:01.0075866Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-09-07T06:42:01.0076480Z * [new tag] ciflow/periodic/156703 -> ciflow/periodic/156703 2025-09-07T06:42:01.0076784Z * [new tag] ciflow/periodic/161715 -> ciflow/periodic/161715 2025-09-07T06:42:01.0077047Z * [new tag] ciflow/periodic/162021 -> ciflow/periodic/162021 2025-09-07T06:42:01.0077178Z * [new tag] ciflow/periodic/162323 -> ciflow/periodic/162323 2025-09-07T06:42:01.0078722Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-09-07T06:42:01.0083143Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-09-07T06:42:01.0083447Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-09-07T06:42:01.0083936Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-09-07T06:42:01.0084132Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-09-07T06:42:01.0084298Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-09-07T06:42:01.0084454Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-09-07T06:42:01.0084763Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-09-07T06:42:01.0084914Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-09-07T06:42:01.0085085Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-09-07T06:42:01.0085769Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-09-07T06:42:01.0086083Z * [new tag] ciflow/rocm-mi300/154170 -> ciflow/rocm-mi300/154170 2025-09-07T06:42:01.0086415Z * [new tag] ciflow/rocm-mi300/158747 -> ciflow/rocm-mi300/158747 2025-09-07T06:42:01.0086577Z * [new tag] ciflow/rocm-mi300/159146 -> ciflow/rocm-mi300/159146 2025-09-07T06:42:01.0090766Z * [new tag] ciflow/rocm-mi300/159158 -> ciflow/rocm-mi300/159158 2025-09-07T06:42:01.0091089Z * [new tag] ciflow/rocm-mi300/161715 -> ciflow/rocm-mi300/161715 2025-09-07T06:42:01.0091324Z * [new tag] ciflow/rocm-mi300/161957 -> ciflow/rocm-mi300/161957 2025-09-07T06:42:01.0091542Z * [new tag] ciflow/rocm-mi300/162053 -> ciflow/rocm-mi300/162053 2025-09-07T06:42:01.0091766Z * [new tag] ciflow/rocm-mi300/162056 -> ciflow/rocm-mi300/162056 2025-09-07T06:42:01.0091920Z * [new tag] ciflow/rocm-mi300/162112 -> ciflow/rocm-mi300/162112 2025-09-07T06:42:01.0092056Z * [new tag] ciflow/rocm-mi300/162245 -> ciflow/rocm-mi300/162245 2025-09-07T06:42:01.0092324Z * [new tag] ciflow/rocm-mi300/162278 -> ciflow/rocm-mi300/162278 2025-09-07T06:42:01.0092466Z * [new tag] ciflow/rocm-mi300/162288 -> ciflow/rocm-mi300/162288 2025-09-07T06:42:01.0092606Z * [new tag] ciflow/rocm-mi355/162053 -> ciflow/rocm-mi355/162053 2025-09-07T06:42:01.0092740Z * [new tag] ciflow/rocm-mi355/162056 -> ciflow/rocm-mi355/162056 2025-09-07T06:42:01.0092869Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-09-07T06:42:01.0092989Z * [new tag] ciflow/rocm/154170 -> ciflow/rocm/154170 2025-09-07T06:42:01.0093644Z * [new tag] ciflow/rocm/156491 -> ciflow/rocm/156491 2025-09-07T06:42:01.0093802Z * [new tag] ciflow/rocm/156592 -> ciflow/rocm/156592 2025-09-07T06:42:01.0093921Z * [new tag] ciflow/rocm/158747 -> ciflow/rocm/158747 2025-09-07T06:42:01.0094302Z * [new tag] ciflow/rocm/159146 -> ciflow/rocm/159146 2025-09-07T06:42:01.0098544Z * [new tag] ciflow/rocm/159158 -> ciflow/rocm/159158 2025-09-07T06:42:01.0098716Z * [new tag] ciflow/rocm/161715 -> ciflow/rocm/161715 2025-09-07T06:42:01.0098854Z * [new tag] ciflow/rocm/161972 -> ciflow/rocm/161972 2025-09-07T06:42:01.0099001Z * [new tag] ciflow/rocm/162052 -> ciflow/rocm/162052 2025-09-07T06:42:01.0099118Z * [new tag] ciflow/rocm/162053 -> ciflow/rocm/162053 2025-09-07T06:42:01.0099242Z * [new tag] ciflow/rocm/162056 -> ciflow/rocm/162056 2025-09-07T06:42:01.0099356Z * [new tag] ciflow/rocm/162112 -> ciflow/rocm/162112 2025-09-07T06:42:01.0099469Z * [new tag] ciflow/rocm/162278 -> ciflow/rocm/162278 2025-09-07T06:42:01.0099622Z * [new tag] ciflow/rocm/162288 -> ciflow/rocm/162288 2025-09-07T06:42:01.0099740Z * [new tag] ciflow/rocm/162305 -> ciflow/rocm/162305 2025-09-07T06:42:01.0101854Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-09-07T06:42:01.0102017Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-09-07T06:42:01.0102393Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-09-07T06:42:01.0102710Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-09-07T06:42:01.0102846Z * [new tag] ciflow/slow/161395 -> ciflow/slow/161395 2025-09-07T06:42:01.0103221Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-09-07T06:42:01.0104304Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-09-07T06:42:01.0104620Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-09-07T06:42:01.0105613Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-09-07T06:42:01.0106088Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-09-07T06:42:01.0109167Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-09-07T06:42:01.0109389Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-09-07T06:42:01.0109516Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-09-07T06:42:01.0109839Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-09-07T06:42:01.0109974Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-09-07T06:42:01.0110372Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-09-07T06:42:01.0110651Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-09-07T06:42:01.0111536Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-09-07T06:42:01.0112058Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-09-07T06:42:01.0114474Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-09-07T06:42:01.0114692Z * [new tag] ciflow/triton_binaries/162329 -> ciflow/triton_binaries/162329 2025-09-07T06:42:01.0114819Z * [new tag] ciflow/trunk/113258 -> ciflow/trunk/113258 2025-09-07T06:42:01.0114943Z * [new tag] ciflow/trunk/137400 -> ciflow/trunk/137400 2025-09-07T06:42:01.0115088Z * [new tag] ciflow/trunk/148180 -> ciflow/trunk/148180 2025-09-07T06:42:01.0115568Z * [new tag] ciflow/trunk/148328 -> ciflow/trunk/148328 2025-09-07T06:42:01.0115872Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-09-07T06:42:01.0119834Z * [new tag] ciflow/trunk/148919 -> ciflow/trunk/148919 2025-09-07T06:42:01.0120169Z * [new tag] ciflow/trunk/152624 -> ciflow/trunk/152624 2025-09-07T06:42:01.0120330Z * [new tag] ciflow/trunk/154170 -> ciflow/trunk/154170 2025-09-07T06:42:01.0120551Z * [new tag] ciflow/trunk/154694 -> ciflow/trunk/154694 2025-09-07T06:42:01.0120704Z * [new tag] ciflow/trunk/156049 -> ciflow/trunk/156049 2025-09-07T06:42:01.0120845Z * [new tag] ciflow/trunk/156703 -> ciflow/trunk/156703 2025-09-07T06:42:01.0120963Z * [new tag] ciflow/trunk/156711 -> ciflow/trunk/156711 2025-09-07T06:42:01.0121211Z * [new tag] ciflow/trunk/157432 -> ciflow/trunk/157432 2025-09-07T06:42:01.0121874Z * [new tag] ciflow/trunk/157685 -> ciflow/trunk/157685 2025-09-07T06:42:01.0122044Z * [new tag] ciflow/trunk/157689 -> ciflow/trunk/157689 2025-09-07T06:42:01.0122175Z * [new tag] ciflow/trunk/157699 -> ciflow/trunk/157699 2025-09-07T06:42:01.0122402Z * [new tag] ciflow/trunk/157813 -> ciflow/trunk/157813 2025-09-07T06:42:01.0122839Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-09-07T06:42:01.0123235Z * [new tag] ciflow/trunk/158091 -> ciflow/trunk/158091 2025-09-07T06:42:01.0127982Z * [new tag] ciflow/trunk/158104 -> ciflow/trunk/158104 2025-09-07T06:42:01.0128155Z * [new tag] ciflow/trunk/158404 -> ciflow/trunk/158404 2025-09-07T06:42:01.0128290Z * [new tag] ciflow/trunk/158647 -> ciflow/trunk/158647 2025-09-07T06:42:01.0128429Z * [new tag] ciflow/trunk/158846 -> ciflow/trunk/158846 2025-09-07T06:42:01.0128565Z * [new tag] ciflow/trunk/159158 -> ciflow/trunk/159158 2025-09-07T06:42:01.0128687Z * [new tag] ciflow/trunk/159682 -> ciflow/trunk/159682 2025-09-07T06:42:01.0128807Z * [new tag] ciflow/trunk/159835 -> ciflow/trunk/159835 2025-09-07T06:42:01.0128933Z * [new tag] ciflow/trunk/160161 -> ciflow/trunk/160161 2025-09-07T06:42:01.0129065Z * [new tag] ciflow/trunk/160236 -> ciflow/trunk/160236 2025-09-07T06:42:01.0129633Z * [new tag] ciflow/trunk/160329 -> ciflow/trunk/160329 2025-09-07T06:42:01.0132921Z * [new tag] ciflow/trunk/160480 -> ciflow/trunk/160480 2025-09-07T06:42:01.0133313Z * [new tag] ciflow/trunk/160532 -> ciflow/trunk/160532 2025-09-07T06:42:01.0133446Z * [new tag] ciflow/trunk/160836 -> ciflow/trunk/160836 2025-09-07T06:42:01.0133764Z * [new tag] ciflow/trunk/160843 -> ciflow/trunk/160843 2025-09-07T06:42:01.0133892Z * [new tag] ciflow/trunk/160869 -> ciflow/trunk/160869 2025-09-07T06:42:01.0134008Z * [new tag] ciflow/trunk/160940 -> ciflow/trunk/160940 2025-09-07T06:42:01.0134131Z * [new tag] ciflow/trunk/160943 -> ciflow/trunk/160943 2025-09-07T06:42:01.0134254Z * [new tag] ciflow/trunk/160953 -> ciflow/trunk/160953 2025-09-07T06:42:01.0134377Z * [new tag] ciflow/trunk/161035 -> ciflow/trunk/161035 2025-09-07T06:42:01.0134489Z * [new tag] ciflow/trunk/161178 -> ciflow/trunk/161178 2025-09-07T06:42:01.0134611Z * [new tag] ciflow/trunk/161349 -> ciflow/trunk/161349 2025-09-07T06:42:01.0134731Z * [new tag] ciflow/trunk/161350 -> ciflow/trunk/161350 2025-09-07T06:42:01.0134876Z * [new tag] ciflow/trunk/161351 -> ciflow/trunk/161351 2025-09-07T06:42:01.0135286Z * [new tag] ciflow/trunk/161395 -> ciflow/trunk/161395 2025-09-07T06:42:01.0136249Z * [new tag] ciflow/trunk/161405 -> ciflow/trunk/161405 2025-09-07T06:42:01.0136692Z * [new tag] ciflow/trunk/161406 -> ciflow/trunk/161406 2025-09-07T06:42:01.0136846Z * [new tag] ciflow/trunk/161410 -> ciflow/trunk/161410 2025-09-07T06:42:01.0137433Z * [new tag] ciflow/trunk/161468 -> ciflow/trunk/161468 2025-09-07T06:42:01.0137828Z * [new tag] ciflow/trunk/161499 -> ciflow/trunk/161499 2025-09-07T06:42:01.0138850Z * [new tag] ciflow/trunk/161527 -> ciflow/trunk/161527 2025-09-07T06:42:01.0139324Z * [new tag] ciflow/trunk/161534 -> ciflow/trunk/161534 2025-09-07T06:42:01.0139482Z * [new tag] ciflow/trunk/161591 -> ciflow/trunk/161591 2025-09-07T06:42:01.0139914Z * [new tag] ciflow/trunk/161595 -> ciflow/trunk/161595 2025-09-07T06:42:01.0141194Z * [new tag] ciflow/trunk/161596 -> ciflow/trunk/161596 2025-09-07T06:42:01.0141342Z * [new tag] ciflow/trunk/161633 -> ciflow/trunk/161633 2025-09-07T06:42:01.0141482Z * [new tag] ciflow/trunk/161634 -> ciflow/trunk/161634 2025-09-07T06:42:01.0142500Z * [new tag] ciflow/trunk/161635 -> ciflow/trunk/161635 2025-09-07T06:42:01.0142647Z * [new tag] ciflow/trunk/161667 -> ciflow/trunk/161667 2025-09-07T06:42:01.0143003Z * [new tag] ciflow/trunk/161670 -> ciflow/trunk/161670 2025-09-07T06:42:01.0143390Z * [new tag] ciflow/trunk/161692 -> ciflow/trunk/161692 2025-09-07T06:42:01.0144164Z * [new tag] ciflow/trunk/161693 -> ciflow/trunk/161693 2025-09-07T06:42:01.0144373Z * [new tag] ciflow/trunk/161695 -> ciflow/trunk/161695 2025-09-07T06:42:01.0145038Z * [new tag] ciflow/trunk/161730 -> ciflow/trunk/161730 2025-09-07T06:42:01.0145338Z * [new tag] ciflow/trunk/161744 -> ciflow/trunk/161744 2025-09-07T06:42:01.0146283Z * [new tag] ciflow/trunk/161749 -> ciflow/trunk/161749 2025-09-07T06:42:01.0146578Z * [new tag] ciflow/trunk/161881 -> ciflow/trunk/161881 2025-09-07T06:42:01.0146772Z * [new tag] ciflow/trunk/161924 -> ciflow/trunk/161924 2025-09-07T06:42:01.0150051Z * [new tag] ciflow/trunk/161926 -> ciflow/trunk/161926 2025-09-07T06:42:01.0150231Z * [new tag] ciflow/trunk/161936 -> ciflow/trunk/161936 2025-09-07T06:42:01.0150362Z * [new tag] ciflow/trunk/161952 -> ciflow/trunk/161952 2025-09-07T06:42:01.0150652Z * [new tag] ciflow/trunk/161955 -> ciflow/trunk/161955 2025-09-07T06:42:01.0150790Z * [new tag] ciflow/trunk/161957 -> ciflow/trunk/161957 2025-09-07T06:42:01.0150918Z * [new tag] ciflow/trunk/161959 -> ciflow/trunk/161959 2025-09-07T06:42:01.0151229Z * [new tag] ciflow/trunk/161977 -> ciflow/trunk/161977 2025-09-07T06:42:01.0151395Z * [new tag] ciflow/trunk/161988 -> ciflow/trunk/161988 2025-09-07T06:42:01.0151524Z * [new tag] ciflow/trunk/161994 -> ciflow/trunk/161994 2025-09-07T06:42:01.0153969Z * [new tag] ciflow/trunk/162007 -> ciflow/trunk/162007 2025-09-07T06:42:01.0154301Z * [new tag] ciflow/trunk/162013 -> ciflow/trunk/162013 2025-09-07T06:42:01.0154541Z * [new tag] ciflow/trunk/162017 -> ciflow/trunk/162017 2025-09-07T06:42:01.0154770Z * [new tag] ciflow/trunk/162021 -> ciflow/trunk/162021 2025-09-07T06:42:01.0154945Z * [new tag] ciflow/trunk/162022 -> ciflow/trunk/162022 2025-09-07T06:42:01.0155165Z * [new tag] ciflow/trunk/162040 -> ciflow/trunk/162040 2025-09-07T06:42:01.0155545Z * [new tag] ciflow/trunk/162041 -> ciflow/trunk/162041 2025-09-07T06:42:01.0156150Z * [new tag] ciflow/trunk/162062 -> ciflow/trunk/162062 2025-09-07T06:42:01.0162512Z * [new tag] ciflow/trunk/162066 -> ciflow/trunk/162066 2025-09-07T06:42:01.0164787Z * [new tag] ciflow/trunk/162089 -> ciflow/trunk/162089 2025-09-07T06:42:01.0169967Z * [new tag] ciflow/trunk/162099 -> ciflow/trunk/162099 2025-09-07T06:42:01.0172078Z * [new tag] ciflow/trunk/162104 -> ciflow/trunk/162104 2025-09-07T06:42:01.0172339Z * [new tag] ciflow/trunk/162106 -> ciflow/trunk/162106 2025-09-07T06:42:01.0178101Z * [new tag] ciflow/trunk/162112 -> ciflow/trunk/162112 2025-09-07T06:42:01.0183053Z * [new tag] ciflow/trunk/162119 -> ciflow/trunk/162119 2025-09-07T06:42:01.0183216Z * [new tag] ciflow/trunk/162142 -> ciflow/trunk/162142 2025-09-07T06:42:01.0183587Z * [new tag] ciflow/trunk/162169 -> ciflow/trunk/162169 2025-09-07T06:42:01.0183926Z * [new tag] ciflow/trunk/162183 -> ciflow/trunk/162183 2025-09-07T06:42:01.0184056Z * [new tag] ciflow/trunk/162190 -> ciflow/trunk/162190 2025-09-07T06:42:01.0184172Z * [new tag] ciflow/trunk/162194 -> ciflow/trunk/162194 2025-09-07T06:42:01.0184295Z * [new tag] ciflow/trunk/162200 -> ciflow/trunk/162200 2025-09-07T06:42:01.0184411Z * [new tag] ciflow/trunk/162206 -> ciflow/trunk/162206 2025-09-07T06:42:01.0184531Z * [new tag] ciflow/trunk/162208 -> ciflow/trunk/162208 2025-09-07T06:42:01.0184653Z * [new tag] ciflow/trunk/162222 -> ciflow/trunk/162222 2025-09-07T06:42:01.0184768Z * [new tag] ciflow/trunk/162238 -> ciflow/trunk/162238 2025-09-07T06:42:01.0184890Z * [new tag] ciflow/trunk/162244 -> ciflow/trunk/162244 2025-09-07T06:42:01.0185005Z * [new tag] ciflow/trunk/162267 -> ciflow/trunk/162267 2025-09-07T06:42:01.0185131Z * [new tag] ciflow/trunk/162269 -> ciflow/trunk/162269 2025-09-07T06:42:01.0185245Z * [new tag] ciflow/trunk/162278 -> ciflow/trunk/162278 2025-09-07T06:42:01.0185361Z * [new tag] ciflow/trunk/162286 -> ciflow/trunk/162286 2025-09-07T06:42:01.0185486Z * [new tag] ciflow/trunk/162288 -> ciflow/trunk/162288 2025-09-07T06:42:01.0185603Z * [new tag] ciflow/trunk/162293 -> ciflow/trunk/162293 2025-09-07T06:42:01.0185984Z * [new tag] ciflow/trunk/162310 -> ciflow/trunk/162310 2025-09-07T06:42:01.0186111Z * [new tag] ciflow/trunk/162311 -> ciflow/trunk/162311 2025-09-07T06:42:01.0186229Z * [new tag] ciflow/trunk/162315 -> ciflow/trunk/162315 2025-09-07T06:42:01.0186356Z * [new tag] ciflow/trunk/162325 -> ciflow/trunk/162325 2025-09-07T06:42:01.0186477Z * [new tag] ciflow/trunk/162328 -> ciflow/trunk/162328 2025-09-07T06:42:01.0186602Z * [new tag] ciflow/trunk/162329 -> ciflow/trunk/162329 2025-09-07T06:42:01.0186723Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-09-07T06:42:01.0186853Z * [new tag] ciflow/vllm/162292 -> ciflow/vllm/162292 2025-09-07T06:42:01.0187005Z * [new tag] ciflow/win-arm64/156049 -> ciflow/win-arm64/156049 2025-09-07T06:42:01.0187137Z * [new tag] ciflow/win-arm64/158104 -> ciflow/win-arm64/158104 2025-09-07T06:42:01.0187266Z * [new tag] ciflow/xpu/157699 -> ciflow/xpu/157699 2025-09-07T06:42:01.0187378Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-09-07T06:42:01.0187492Z * [new tag] ciflow/xpu/159459 -> ciflow/xpu/159459 2025-09-07T06:42:01.0187605Z * [new tag] ciflow/xpu/159718 -> ciflow/xpu/159718 2025-09-07T06:42:01.0187714Z * [new tag] ciflow/xpu/159944 -> ciflow/xpu/159944 2025-09-07T06:42:01.0187829Z * [new tag] ciflow/xpu/160867 -> ciflow/xpu/160867 2025-09-07T06:42:01.0187939Z * [new tag] ciflow/xpu/160938 -> ciflow/xpu/160938 2025-09-07T06:42:01.0188054Z * [new tag] ciflow/xpu/160940 -> ciflow/xpu/160940 2025-09-07T06:42:01.0188166Z * [new tag] ciflow/xpu/160953 -> ciflow/xpu/160953 2025-09-07T06:42:01.0188284Z * [new tag] ciflow/xpu/161045 -> ciflow/xpu/161045 2025-09-07T06:42:01.0188394Z * [new tag] ciflow/xpu/161058 -> ciflow/xpu/161058 2025-09-07T06:42:01.0188503Z * [new tag] ciflow/xpu/161246 -> ciflow/xpu/161246 2025-09-07T06:42:01.0188620Z * [new tag] ciflow/xpu/161397 -> ciflow/xpu/161397 2025-09-07T06:42:01.0188773Z * [new tag] ciflow/xpu/161485 -> ciflow/xpu/161485 2025-09-07T06:42:01.0188895Z * [new tag] ciflow/xpu/161988 -> ciflow/xpu/161988 2025-09-07T06:42:01.0189011Z * [new tag] ciflow/xpu/162062 -> ciflow/xpu/162062 2025-09-07T06:42:01.0189132Z * [new tag] cslpull75 -> cslpull75 2025-09-07T06:42:01.0189252Z * [new tag] cslpull76 -> cslpull76 2025-09-07T06:42:01.0189359Z * [new tag] cslpull77 -> cslpull77 2025-09-07T06:42:01.0189472Z * [new tag] cslpull78 -> cslpull78 2025-09-07T06:42:01.0189576Z * [new tag] cslpull79 -> cslpull79 2025-09-07T06:42:01.0189679Z * [new tag] cslpull80 -> cslpull80 2025-09-07T06:42:01.0189791Z * [new tag] cslpull81 -> cslpull81 2025-09-07T06:42:01.0189896Z * [new tag] cslpull82 -> cslpull82 2025-09-07T06:42:01.0190005Z * [new tag] cslpull83 -> cslpull83 2025-09-07T06:42:01.0190118Z * [new tag] cslpull84 -> cslpull84 2025-09-07T06:42:01.0190228Z * [new tag] cslpull85 -> cslpull85 2025-09-07T06:42:01.0190328Z * [new tag] cslpull86 -> cslpull86 2025-09-07T06:42:01.0190428Z * [new tag] cslpull87 -> cslpull87 2025-09-07T06:42:01.0190572Z * [new tag] cslpull88 -> cslpull88 2025-09-07T06:42:01.0190678Z * [new tag] cslpull89 -> cslpull89 2025-09-07T06:42:01.0190779Z * [new tag] cslpull90 -> cslpull90 2025-09-07T06:42:01.0190873Z * [new tag] cslpull91 -> cslpull91 2025-09-07T06:42:01.0190968Z * [new tag] cslpull92 -> cslpull92 2025-09-07T06:42:01.0191077Z * [new tag] flight_5 -> flight_5 2025-09-07T06:42:01.0191342Z * [new tag] flight_5.1 -> flight_5.1 2025-09-07T06:42:01.0191473Z * [new tag] flight_5.2 -> flight_5.2 2025-09-07T06:42:01.0192881Z * [new tag] flight_5.3 -> flight_5.3 2025-09-07T06:42:01.0193040Z * [new tag] forpull1 -> forpull1 2025-09-07T06:42:01.0193202Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-09-07T06:42:01.0193645Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-09-07T06:42:01.0197649Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-09-07T06:42:01.0197974Z * [new tag] nightly-binary -> nightly-binary 2025-09-07T06:42:01.0198217Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-09-07T06:42:01.0198369Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-09-07T06:42:01.0198622Z * [new tag] trunk/00636e0171e7e733628c408084805442270cf608 -> trunk/00636e0171e7e733628c408084805442270cf608 2025-09-07T06:42:01.0199013Z * [new tag] trunk/019fed39aa6b2dd8c69347378d53423e5efae8d4 -> trunk/019fed39aa6b2dd8c69347378d53423e5efae8d4 2025-09-07T06:42:01.0199743Z * [new tag] trunk/01ab325cc2e0dc221af4d710974e1b9175066544 -> trunk/01ab325cc2e0dc221af4d710974e1b9175066544 2025-09-07T06:42:01.0200063Z * [new tag] trunk/01edcd4df8bf0c7b4cc2d3ec868bd2059eeea83b -> trunk/01edcd4df8bf0c7b4cc2d3ec868bd2059eeea83b 2025-09-07T06:42:01.0200321Z * [new tag] trunk/040d00af048967dde7938d358d7f5988cbd18388 -> trunk/040d00af048967dde7938d358d7f5988cbd18388 2025-09-07T06:42:01.0200976Z * [new tag] trunk/0447f2d99b4351b2ff129dce6eebb371024f73e5 -> trunk/0447f2d99b4351b2ff129dce6eebb371024f73e5 2025-09-07T06:42:01.0201371Z * [new tag] trunk/047603d35bdc70046216384838d6340feab79bf4 -> trunk/047603d35bdc70046216384838d6340feab79bf4 2025-09-07T06:42:01.0201642Z * [new tag] trunk/06da7c0730b3764f178ec3a90dedf4ffa4202d81 -> trunk/06da7c0730b3764f178ec3a90dedf4ffa4202d81 2025-09-07T06:42:01.0202088Z * [new tag] trunk/081cab045472ce045634548cc6c14a4870641e23 -> trunk/081cab045472ce045634548cc6c14a4870641e23 2025-09-07T06:42:01.0204185Z * [new tag] trunk/09587daf8c9f21f5340f73921ce5f23d1a4a4572 -> trunk/09587daf8c9f21f5340f73921ce5f23d1a4a4572 2025-09-07T06:42:01.0204635Z * [new tag] trunk/09be1890d72cc34fc946965dc4a27736bf0ca8c6 -> trunk/09be1890d72cc34fc946965dc4a27736bf0ca8c6 2025-09-07T06:42:01.0204999Z * [new tag] trunk/09d2f1b6315d6d416fbf452793d65795863ebc66 -> trunk/09d2f1b6315d6d416fbf452793d65795863ebc66 2025-09-07T06:42:01.0205334Z * [new tag] trunk/0af70e2353e1dcda83175fd4834ecb7b63e009e0 -> trunk/0af70e2353e1dcda83175fd4834ecb7b63e009e0 2025-09-07T06:42:01.0205648Z * [new tag] trunk/0c0e056a9e20c17271a6144dd32c0c7e3ba26736 -> trunk/0c0e056a9e20c17271a6144dd32c0c7e3ba26736 2025-09-07T06:42:01.0206095Z * [new tag] trunk/0cd6c56bdfa9178ff61be82ce3b178926ddb64a9 -> trunk/0cd6c56bdfa9178ff61be82ce3b178926ddb64a9 2025-09-07T06:42:01.0206972Z * [new tag] trunk/0d421ace32c1605ee8e452ee1eeb03bd243dd96c -> trunk/0d421ace32c1605ee8e452ee1eeb03bd243dd96c 2025-09-07T06:42:01.0207489Z * [new tag] trunk/0d71a9dd5b4b6d1dde58d91c9b71d96bc6a6a171 -> trunk/0d71a9dd5b4b6d1dde58d91c9b71d96bc6a6a171 2025-09-07T06:42:01.0207943Z * [new tag] trunk/0d84ff3b78f55492d3d4708458c92d776274939e -> trunk/0d84ff3b78f55492d3d4708458c92d776274939e 2025-09-07T06:42:01.0214199Z * [new tag] trunk/0f45aaf4414048b17d720d0915ce221a8de8ec63 -> trunk/0f45aaf4414048b17d720d0915ce221a8de8ec63 2025-09-07T06:42:01.0214686Z * [new tag] trunk/0ff8eabf1387de5acd6712a03bda61f1a3dfa27f -> trunk/0ff8eabf1387de5acd6712a03bda61f1a3dfa27f 2025-09-07T06:42:01.0215093Z * [new tag] trunk/104f2680e03d13a4765ca69f905d8f16fc0c822f -> trunk/104f2680e03d13a4765ca69f905d8f16fc0c822f 2025-09-07T06:42:01.0215492Z * [new tag] trunk/12814701555d3e41dfcdf8f9273af5821e322df0 -> trunk/12814701555d3e41dfcdf8f9273af5821e322df0 2025-09-07T06:42:01.0215762Z * [new tag] trunk/13b65196db422bdb394cb482e208c61ed448898c -> trunk/13b65196db422bdb394cb482e208c61ed448898c 2025-09-07T06:42:01.0216043Z * [new tag] trunk/13d66e2a66eceed14b8a8f5a971087df4f688a46 -> trunk/13d66e2a66eceed14b8a8f5a971087df4f688a46 2025-09-07T06:42:01.0216313Z * [new tag] trunk/145a3a7bda15e3963a33eb1b54bba5d4a270b225 -> trunk/145a3a7bda15e3963a33eb1b54bba5d4a270b225 2025-09-07T06:42:01.0216569Z * [new tag] trunk/146371483318e17929daefd37c8e459d9d6d47bb -> trunk/146371483318e17929daefd37c8e459d9d6d47bb 2025-09-07T06:42:01.0216821Z * [new tag] trunk/15c77a8cfd341e74fd124b077492ef2bfa51b339 -> trunk/15c77a8cfd341e74fd124b077492ef2bfa51b339 2025-09-07T06:42:01.0217076Z * [new tag] trunk/17fa8eec4a1e32939ab4d364ee6e75487a79b654 -> trunk/17fa8eec4a1e32939ab4d364ee6e75487a79b654 2025-09-07T06:42:01.0217327Z * [new tag] trunk/190c391a28845a14df26abb228d26aa813efb20c -> trunk/190c391a28845a14df26abb228d26aa813efb20c 2025-09-07T06:42:01.0217598Z * [new tag] trunk/1a588ace4667bde1331fbd8ed957157dca5cee68 -> trunk/1a588ace4667bde1331fbd8ed957157dca5cee68 2025-09-07T06:42:01.0217847Z * [new tag] trunk/1aa7476885e8f6e7b0ec3a5b6383aad9d3f343e7 -> trunk/1aa7476885e8f6e7b0ec3a5b6383aad9d3f343e7 2025-09-07T06:42:01.0218094Z * [new tag] trunk/1aeb421c342c9e9607842f4c87cb46e8e816ee53 -> trunk/1aeb421c342c9e9607842f4c87cb46e8e816ee53 2025-09-07T06:42:01.0218762Z * [new tag] trunk/1c1b28d5b6a942fafe23b2f09302d93c25226d4a -> trunk/1c1b28d5b6a942fafe23b2f09302d93c25226d4a 2025-09-07T06:42:01.0219023Z * [new tag] trunk/1ebd70d0c0d562d3be9abdee2a21906584af7d99 -> trunk/1ebd70d0c0d562d3be9abdee2a21906584af7d99 2025-09-07T06:42:01.0219294Z * [new tag] trunk/1ec2c15914da4ef7bd926ed9aebc8671c75fe965 -> trunk/1ec2c15914da4ef7bd926ed9aebc8671c75fe965 2025-09-07T06:42:01.0219542Z * [new tag] trunk/1f51056bd64e73d1aa81321bc3c098575b1bc78a -> trunk/1f51056bd64e73d1aa81321bc3c098575b1bc78a 2025-09-07T06:42:01.0219944Z * [new tag] trunk/1f820de639c75a1562d3fb03f160439f853ae07b -> trunk/1f820de639c75a1562d3fb03f160439f853ae07b 2025-09-07T06:42:01.0220190Z * [new tag] trunk/204697f0e695d82894c5010fbec664c4391f90cc -> trunk/204697f0e695d82894c5010fbec664c4391f90cc 2025-09-07T06:42:01.0220448Z * [new tag] trunk/20629b1619fe636227d01fc85ba221daa7185a05 -> trunk/20629b1619fe636227d01fc85ba221daa7185a05 2025-09-07T06:42:01.0220702Z * [new tag] trunk/20b47acef845e9c4f71da9429a396d293f50ebe7 -> trunk/20b47acef845e9c4f71da9429a396d293f50ebe7 2025-09-07T06:42:01.0226837Z * [new tag] trunk/20bfb2539d7c5250379648eda35f80b8a7d642dd -> trunk/20bfb2539d7c5250379648eda35f80b8a7d642dd 2025-09-07T06:42:01.0227126Z * [new tag] trunk/21fae99c180d17def562797ea0fb154d8fdf88e3 -> trunk/21fae99c180d17def562797ea0fb154d8fdf88e3 2025-09-07T06:42:01.0227588Z * [new tag] trunk/248355faf53f9f7ba2fd0a367d59600c6d991e7f -> trunk/248355faf53f9f7ba2fd0a367d59600c6d991e7f 2025-09-07T06:42:01.0227858Z * [new tag] trunk/25f4aaed9ec26f39c13862323ff8582006473d23 -> trunk/25f4aaed9ec26f39c13862323ff8582006473d23 2025-09-07T06:42:01.0233079Z * [new tag] trunk/261a84a1764412f8e659c956e3f81997ec3de9d5 -> trunk/261a84a1764412f8e659c956e3f81997ec3de9d5 2025-09-07T06:42:01.0233535Z * [new tag] trunk/28f4ab0737937858730f29f5c4e601e109cf9d5f -> trunk/28f4ab0737937858730f29f5c4e601e109cf9d5f 2025-09-07T06:42:01.0233920Z * [new tag] trunk/291cd11f2d5df6f48d348cce0e4e762f274f4dc4 -> trunk/291cd11f2d5df6f48d348cce0e4e762f274f4dc4 2025-09-07T06:42:01.0234265Z * [new tag] trunk/29280864d941e6108ab57f7298f520c0cf9696e9 -> trunk/29280864d941e6108ab57f7298f520c0cf9696e9 2025-09-07T06:42:01.0234968Z * [new tag] trunk/2a45837e98c63cae9d1a2e2133a727b829e549d5 -> trunk/2a45837e98c63cae9d1a2e2133a727b829e549d5 2025-09-07T06:42:01.0235326Z * [new tag] trunk/2a5c0785e2f975697fd7bdf1411de6e03dcaa1ef -> trunk/2a5c0785e2f975697fd7bdf1411de6e03dcaa1ef 2025-09-07T06:42:01.0235599Z * [new tag] trunk/2b8a83901c58a0858ea9e4ce00055f48e6ed164c -> trunk/2b8a83901c58a0858ea9e4ce00055f48e6ed164c 2025-09-07T06:42:01.0235872Z * [new tag] trunk/2ba65472dd54488a86a50326ea990195fc6732d6 -> trunk/2ba65472dd54488a86a50326ea990195fc6732d6 2025-09-07T06:42:01.0236185Z * [new tag] trunk/2c03f0acc53ed13fe8ebfe809129f25996e009a0 -> trunk/2c03f0acc53ed13fe8ebfe809129f25996e009a0 2025-09-07T06:42:01.0236472Z * [new tag] trunk/2dd529df0092799f68ee7afcf52338276906706a -> trunk/2dd529df0092799f68ee7afcf52338276906706a 2025-09-07T06:42:01.0236753Z * [new tag] trunk/2f6b4b1ad3f82bb3bd984f6e65744ea339ffb8b5 -> trunk/2f6b4b1ad3f82bb3bd984f6e65744ea339ffb8b5 2025-09-07T06:42:01.0237015Z * [new tag] trunk/2fa0520a64ed8aa734a56c4d124958f0b5711ca8 -> trunk/2fa0520a64ed8aa734a56c4d124958f0b5711ca8 2025-09-07T06:42:01.0237267Z * [new tag] trunk/302df2ac5dc4222294c09d48804a2dddb8f4bad8 -> trunk/302df2ac5dc4222294c09d48804a2dddb8f4bad8 2025-09-07T06:42:01.0237514Z * [new tag] trunk/33028597bfa2e0178e28c8cce33cb9b3800cac43 -> trunk/33028597bfa2e0178e28c8cce33cb9b3800cac43 2025-09-07T06:42:01.0237754Z * [new tag] trunk/34aa78274d6770086025a967fa63a86830e08176 -> trunk/34aa78274d6770086025a967fa63a86830e08176 2025-09-07T06:42:01.0238267Z * [new tag] trunk/3559c354ce6a14d11fe29fb12fa2747a2f2af449 -> trunk/3559c354ce6a14d11fe29fb12fa2747a2f2af449 2025-09-07T06:42:01.0238530Z * [new tag] trunk/36d207fcaaede0d1e58a5168084c307b32b6fd8b -> trunk/36d207fcaaede0d1e58a5168084c307b32b6fd8b 2025-09-07T06:42:01.0238958Z * [new tag] trunk/377033757ae5ca524ea842f1b0a5f446ed3d8fe0 -> trunk/377033757ae5ca524ea842f1b0a5f446ed3d8fe0 2025-09-07T06:42:01.0239340Z * [new tag] trunk/3771380f83fcac154a7c89ad679311d8c4818287 -> trunk/3771380f83fcac154a7c89ad679311d8c4818287 2025-09-07T06:42:01.0239715Z * [new tag] trunk/3a207816cc569f78863d86c01f2a3d265350e39f -> trunk/3a207816cc569f78863d86c01f2a3d265350e39f 2025-09-07T06:42:01.0240117Z * [new tag] trunk/3a20a20e7065ec927fdd216d4da3b04f879b3c67 -> trunk/3a20a20e7065ec927fdd216d4da3b04f879b3c67 2025-09-07T06:42:01.0240515Z * [new tag] trunk/3bbc2e3e4f025523eaa5dbff220b3e96bca608d0 -> trunk/3bbc2e3e4f025523eaa5dbff220b3e96bca608d0 2025-09-07T06:42:01.0240918Z * [new tag] trunk/3c0ff1b569c45cfa6935ad8031a9d4cf1551aa3f -> trunk/3c0ff1b569c45cfa6935ad8031a9d4cf1551aa3f 2025-09-07T06:42:01.0241724Z * [new tag] trunk/3c45af079afc92a03b03ddf4f9198902ffcf30cf -> trunk/3c45af079afc92a03b03ddf4f9198902ffcf30cf 2025-09-07T06:42:01.0242177Z * [new tag] trunk/3dde5d7f9bf80dd6623a712bc429e9e4302464b5 -> trunk/3dde5d7f9bf80dd6623a712bc429e9e4302464b5 2025-09-07T06:42:01.0243657Z * [new tag] trunk/403a3a393cda7e60f503f3b04b8805a845dcf45d -> trunk/403a3a393cda7e60f503f3b04b8805a845dcf45d 2025-09-07T06:42:01.0244043Z * [new tag] trunk/420c52ecf36f86d32da0853bfbe074b682b070aa -> trunk/420c52ecf36f86d32da0853bfbe074b682b070aa 2025-09-07T06:42:01.0244409Z * [new tag] trunk/43b7c86a2c0f91320f5c5f4827b111edff06fdb6 -> trunk/43b7c86a2c0f91320f5c5f4827b111edff06fdb6 2025-09-07T06:42:01.0244782Z * [new tag] trunk/451ed931562ec8b46d1f7e6c266a68132a119336 -> trunk/451ed931562ec8b46d1f7e6c266a68132a119336 2025-09-07T06:42:01.0245529Z * [new tag] trunk/480c7391126656154318fabf1d57ebc01e196e63 -> trunk/480c7391126656154318fabf1d57ebc01e196e63 2025-09-07T06:42:01.0245837Z * [new tag] trunk/48bedd753da22634aa94fbafeb731e82025404f3 -> trunk/48bedd753da22634aa94fbafeb731e82025404f3 2025-09-07T06:42:01.0246117Z * [new tag] trunk/494878a11b79071ada0b98f34042d47155be6d1c -> trunk/494878a11b79071ada0b98f34042d47155be6d1c 2025-09-07T06:42:01.0246388Z * [new tag] trunk/4ae57d448c0a7d37e4cfd5c27d977fad2cef4051 -> trunk/4ae57d448c0a7d37e4cfd5c27d977fad2cef4051 2025-09-07T06:42:01.0246649Z * [new tag] trunk/4cdaf8265d86f984254b62052da8c26ef61ef1cf -> trunk/4cdaf8265d86f984254b62052da8c26ef61ef1cf 2025-09-07T06:42:01.0246909Z * [new tag] trunk/4d4abec80f03cd8fdefe1d9cb3a60d3690cd777e -> trunk/4d4abec80f03cd8fdefe1d9cb3a60d3690cd777e 2025-09-07T06:42:01.0247178Z * [new tag] trunk/4e42aa8ffc44b8340eb0eeaf80a2cafc4763a186 -> trunk/4e42aa8ffc44b8340eb0eeaf80a2cafc4763a186 2025-09-07T06:42:01.0247426Z * [new tag] trunk/4f72d932feee0749397fec876dcd43994f50b215 -> trunk/4f72d932feee0749397fec876dcd43994f50b215 2025-09-07T06:42:01.0247675Z * [new tag] trunk/50fc22dedf3c4a27be61fa05551c4f320281b42d -> trunk/50fc22dedf3c4a27be61fa05551c4f320281b42d 2025-09-07T06:42:01.0247934Z * [new tag] trunk/5211f1f908907ffc064b56e43cf8659f7fc22aa9 -> trunk/5211f1f908907ffc064b56e43cf8659f7fc22aa9 2025-09-07T06:42:01.0248200Z * [new tag] trunk/524b78d4f67045b83bb69edc56ab16efe282971c -> trunk/524b78d4f67045b83bb69edc56ab16efe282971c 2025-09-07T06:42:01.0248458Z * [new tag] trunk/54e275e0d81fe1e1ccfa4fb5f2a5a9aaca00ca15 -> trunk/54e275e0d81fe1e1ccfa4fb5f2a5a9aaca00ca15 2025-09-07T06:42:01.0248849Z * [new tag] trunk/5561e45758d59c94605873d5db48ed459c004c3b -> trunk/5561e45758d59c94605873d5db48ed459c004c3b 2025-09-07T06:42:01.0249258Z * [new tag] trunk/57278d45f046d4f89f45d373b1af4dd56934ff24 -> trunk/57278d45f046d4f89f45d373b1af4dd56934ff24 2025-09-07T06:42:01.0249635Z * [new tag] trunk/5927a70934ccf7b70182d364c23245a7dd685503 -> trunk/5927a70934ccf7b70182d364c23245a7dd685503 2025-09-07T06:42:01.0250421Z * [new tag] trunk/5985e28912aeb40b103ebfcf2fd0665eb4a50599 -> trunk/5985e28912aeb40b103ebfcf2fd0665eb4a50599 2025-09-07T06:42:01.0250722Z * [new tag] trunk/5a2da090ed6db88bb657c4e51ec0b310cd08bff6 -> trunk/5a2da090ed6db88bb657c4e51ec0b310cd08bff6 2025-09-07T06:42:01.0250978Z * [new tag] trunk/5c473e9f5ee0ef0fc38e6cf34a95b547f8cdc8d5 -> trunk/5c473e9f5ee0ef0fc38e6cf34a95b547f8cdc8d5 2025-09-07T06:42:01.0251423Z * [new tag] trunk/5c67426d6847667a7c55a2dd01f470fa37238c18 -> trunk/5c67426d6847667a7c55a2dd01f470fa37238c18 2025-09-07T06:42:01.0251950Z * [new tag] trunk/5da573c42c332bc68d4b7946c69f690a876d951a -> trunk/5da573c42c332bc68d4b7946c69f690a876d951a 2025-09-07T06:42:01.0255371Z * [new tag] trunk/5e5870e858f60ff4bf87d03f3592097e934a9580 -> trunk/5e5870e858f60ff4bf87d03f3592097e934a9580 2025-09-07T06:42:01.0255849Z * [new tag] trunk/5f3cbc9442aa55b5afb29f4ac8ca9be569003e84 -> trunk/5f3cbc9442aa55b5afb29f4ac8ca9be569003e84 2025-09-07T06:42:01.0256381Z * [new tag] trunk/600c25e9a17fe56e3dee872be8854db08916ba0c -> trunk/600c25e9a17fe56e3dee872be8854db08916ba0c 2025-09-07T06:42:01.0257160Z * [new tag] trunk/601ae8e4831fc8123fffcfb8fd2e6b6381b42e14 -> trunk/601ae8e4831fc8123fffcfb8fd2e6b6381b42e14 2025-09-07T06:42:01.0257441Z * [new tag] trunk/6087ef41e54c2494b117ffd923faf20f515a6806 -> trunk/6087ef41e54c2494b117ffd923faf20f515a6806 2025-09-07T06:42:01.0257714Z * [new tag] trunk/626cb7df8161dd4ecb4fe43b60f37ce9076f56b1 -> trunk/626cb7df8161dd4ecb4fe43b60f37ce9076f56b1 2025-09-07T06:42:01.0257957Z * [new tag] trunk/62c3f9a97fd3dea7132a93066d32d893ffe101e6 -> trunk/62c3f9a97fd3dea7132a93066d32d893ffe101e6 2025-09-07T06:42:01.0258203Z * [new tag] trunk/63a9c23fe99eacfd09610c36dfe8f01b053c1a35 -> trunk/63a9c23fe99eacfd09610c36dfe8f01b053c1a35 2025-09-07T06:42:01.0258451Z * [new tag] trunk/65985937d97505f648b6ed852c3129f2dd08b251 -> trunk/65985937d97505f648b6ed852c3129f2dd08b251 2025-09-07T06:42:01.0258995Z * [new tag] trunk/66f3b4a682a6153517dd23369fdc3289b6494b07 -> trunk/66f3b4a682a6153517dd23369fdc3289b6494b07 2025-09-07T06:42:01.0259405Z * [new tag] trunk/6737e2c996990024187ba620d2764f3b6f6add2c -> trunk/6737e2c996990024187ba620d2764f3b6f6add2c 2025-09-07T06:42:01.0259975Z * [new tag] trunk/67c31dcd364f10072a55f4a30ffd1151c686283a -> trunk/67c31dcd364f10072a55f4a30ffd1151c686283a 2025-09-07T06:42:01.0260552Z * [new tag] trunk/68738beff73e9c3512e18b4edea811a897ce42db -> trunk/68738beff73e9c3512e18b4edea811a897ce42db 2025-09-07T06:42:01.0261202Z * [new tag] trunk/69a25f68884a168550695fdb1a7c310c54d29536 -> trunk/69a25f68884a168550695fdb1a7c310c54d29536 2025-09-07T06:42:01.0261759Z * [new tag] trunk/6b1900c22f1a07b9519346898d4c71d8a2b0f12f -> trunk/6b1900c22f1a07b9519346898d4c71d8a2b0f12f 2025-09-07T06:42:01.0262384Z * [new tag] trunk/6b8b3ac4403f771bd4a8f9a45d93347304148774 -> trunk/6b8b3ac4403f771bd4a8f9a45d93347304148774 2025-09-07T06:42:01.0262960Z * [new tag] trunk/6f7608d603834d6068b2e7a5d59bec3973b6bb1b -> trunk/6f7608d603834d6068b2e7a5d59bec3973b6bb1b 2025-09-07T06:42:01.0263584Z * [new tag] trunk/70d36e047dfb3488fd6335016711a784d810ebda -> trunk/70d36e047dfb3488fd6335016711a784d810ebda 2025-09-07T06:42:01.0264064Z * [new tag] trunk/71992dd805ff9d6763f77214dfe8b0465e88c87b -> trunk/71992dd805ff9d6763f77214dfe8b0465e88c87b 2025-09-07T06:42:01.0264702Z * [new tag] trunk/734ce8eba9c69381f187359bf0fef1d71d84cd20 -> trunk/734ce8eba9c69381f187359bf0fef1d71d84cd20 2025-09-07T06:42:01.0265339Z * [new tag] trunk/73eb4511fb863a37944342b7e92aae706de603c8 -> trunk/73eb4511fb863a37944342b7e92aae706de603c8 2025-09-07T06:42:01.0266236Z * [new tag] trunk/75bc23cfc345bd4c05e7f97c416c4b3d2d1fa64b -> trunk/75bc23cfc345bd4c05e7f97c416c4b3d2d1fa64b 2025-09-07T06:42:01.0266494Z * [new tag] trunk/771f369448321a387f2018535bc8b8b6e5f12fab -> trunk/771f369448321a387f2018535bc8b8b6e5f12fab 2025-09-07T06:42:01.0272146Z * [new tag] trunk/789d4942127143f2adcb53612c058ce4c9a2cf20 -> trunk/789d4942127143f2adcb53612c058ce4c9a2cf20 2025-09-07T06:42:01.0272441Z * [new tag] trunk/791eff96c85678c950888f9da24650083ee673fe -> trunk/791eff96c85678c950888f9da24650083ee673fe 2025-09-07T06:42:01.0272730Z * [new tag] trunk/793fc12aff1f69fbbf9f4278182fb52bbe350fc9 -> trunk/793fc12aff1f69fbbf9f4278182fb52bbe350fc9 2025-09-07T06:42:01.0272988Z * [new tag] trunk/79fcd5247a9a129eee526a14df30bfc6a22b3f01 -> trunk/79fcd5247a9a129eee526a14df30bfc6a22b3f01 2025-09-07T06:42:01.0273246Z * [new tag] trunk/7f4ff79210eb06924f223ae3a1941ee0e2635348 -> trunk/7f4ff79210eb06924f223ae3a1941ee0e2635348 2025-09-07T06:42:01.0273636Z * [new tag] trunk/8076a185c85112be62be292eb47409c88a585b1c -> trunk/8076a185c85112be62be292eb47409c88a585b1c 2025-09-07T06:42:01.0273882Z * [new tag] trunk/80dd397f1979371a5583fa3d5c7352029522a78d -> trunk/80dd397f1979371a5583fa3d5c7352029522a78d 2025-09-07T06:42:01.0274123Z * [new tag] trunk/8171d6052ec12628eb67e0040839314056014429 -> trunk/8171d6052ec12628eb67e0040839314056014429 2025-09-07T06:42:01.0274376Z * [new tag] trunk/81aeefa657b7ccc26b275c50a9f33b2f056e8071 -> trunk/81aeefa657b7ccc26b275c50a9f33b2f056e8071 2025-09-07T06:42:01.0274643Z * [new tag] trunk/81b7b16618bda250ce55982894a83dc0805eb64c -> trunk/81b7b16618bda250ce55982894a83dc0805eb64c 2025-09-07T06:42:01.0274887Z * [new tag] trunk/827f0d405448de31f79d1089f7d7fceab2f87895 -> trunk/827f0d405448de31f79d1089f7d7fceab2f87895 2025-09-07T06:42:01.0275141Z * [new tag] trunk/82f63c8f6de63c30132a8ac299b6e8c2fd0d3fe8 -> trunk/82f63c8f6de63c30132a8ac299b6e8c2fd0d3fe8 2025-09-07T06:42:01.0275397Z * [new tag] trunk/850e1382a9c56bfde18af09d3e72352d775e9435 -> trunk/850e1382a9c56bfde18af09d3e72352d775e9435 2025-09-07T06:42:01.0275650Z * [new tag] trunk/8678d831c48e616b717bff50f2d03141d2e9f965 -> trunk/8678d831c48e616b717bff50f2d03141d2e9f965 2025-09-07T06:42:01.0275895Z * [new tag] trunk/869cbcc16e489a4f5a14a93d5779b0ea86061c60 -> trunk/869cbcc16e489a4f5a14a93d5779b0ea86061c60 2025-09-07T06:42:01.0280465Z * [new tag] trunk/8703debf669bc2238211bfd039f4ecdd8228b7f7 -> trunk/8703debf669bc2238211bfd039f4ecdd8228b7f7 2025-09-07T06:42:01.0285665Z * [new tag] trunk/874069fbe46e82da5cfa405e6c0deb12e89ff608 -> trunk/874069fbe46e82da5cfa405e6c0deb12e89ff608 2025-09-07T06:42:01.0290678Z * [new tag] trunk/8875d6e394da2fffd04f31b28bf258c94d4776a3 -> trunk/8875d6e394da2fffd04f31b28bf258c94d4776a3 2025-09-07T06:42:01.0291125Z * [new tag] trunk/88d94d17e8c5155451393afa6eb3bab48ab61c16 -> trunk/88d94d17e8c5155451393afa6eb3bab48ab61c16 2025-09-07T06:42:01.0291534Z * [new tag] trunk/890626632def7e0ef95a2d01e87a0e4627824a9f -> trunk/890626632def7e0ef95a2d01e87a0e4627824a9f 2025-09-07T06:42:01.0291883Z * [new tag] trunk/8975cda2520b7b1b5bc3b4d8213edf261fa82570 -> trunk/8975cda2520b7b1b5bc3b4d8213edf261fa82570 2025-09-07T06:42:01.0292156Z * [new tag] trunk/89d41d3f61d04f14730ec26f008a59bef6624610 -> trunk/89d41d3f61d04f14730ec26f008a59bef6624610 2025-09-07T06:42:01.0292599Z * [new tag] trunk/8bb213b6d599ef1273fe52f9b1f6d476056c3a41 -> trunk/8bb213b6d599ef1273fe52f9b1f6d476056c3a41 2025-09-07T06:42:01.0293172Z * [new tag] trunk/8e23a1227b5fb2e39afaa7d57c075a75b640a5af -> trunk/8e23a1227b5fb2e39afaa7d57c075a75b640a5af 2025-09-07T06:42:01.0293453Z * [new tag] trunk/8ec551bb354ab2b85fbbba9d461740a20366d248 -> trunk/8ec551bb354ab2b85fbbba9d461740a20366d248 2025-09-07T06:42:01.0293727Z * [new tag] trunk/8fd3c9ce919c8d5c645fd348bba517e948cbc29d -> trunk/8fd3c9ce919c8d5c645fd348bba517e948cbc29d 2025-09-07T06:42:01.0293968Z * [new tag] trunk/90f50f7e68e120d9574e6e3189e37b4280010ad9 -> trunk/90f50f7e68e120d9574e6e3189e37b4280010ad9 2025-09-07T06:42:01.0294218Z * [new tag] trunk/91f0bcf43fc0bc743350d491ac63b77e92054ac9 -> trunk/91f0bcf43fc0bc743350d491ac63b77e92054ac9 2025-09-07T06:42:01.0294458Z * [new tag] trunk/92576a594b8121f6b0b1b5a3ea16d08792fc68ab -> trunk/92576a594b8121f6b0b1b5a3ea16d08792fc68ab 2025-09-07T06:42:01.0294703Z * [new tag] trunk/92a43025e0baa1f2ce345f28d22913b518a1ab9d -> trunk/92a43025e0baa1f2ce345f28d22913b518a1ab9d 2025-09-07T06:42:01.0294939Z * [new tag] trunk/93fb23d6fae7c4e82c4239a1033e522088742634 -> trunk/93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:42:01.0295189Z * [new tag] trunk/9458d1ac3bd70c2af316a8ba95d2c6c9c1199c9c -> trunk/9458d1ac3bd70c2af316a8ba95d2c6c9c1199c9c 2025-09-07T06:42:01.0295559Z * [new tag] trunk/9480cdc0b61488c89a23c2f64f43b2dcedc8728e -> trunk/9480cdc0b61488c89a23c2f64f43b2dcedc8728e 2025-09-07T06:42:01.0295801Z * [new tag] trunk/9491d289b329e4ba4a9f5f5b1be7960671bb7840 -> trunk/9491d289b329e4ba4a9f5f5b1be7960671bb7840 2025-09-07T06:42:01.0296045Z * [new tag] trunk/9499c8761cd2067feb9877414e818f6fd00290f1 -> trunk/9499c8761cd2067feb9877414e818f6fd00290f1 2025-09-07T06:42:01.0296292Z * [new tag] trunk/95ee0bfea99d3d346d6502b91b497d2b35795504 -> trunk/95ee0bfea99d3d346d6502b91b497d2b35795504 2025-09-07T06:42:01.0296540Z * [new tag] trunk/98374612fc2febd686be20761e56bdc2424bc36a -> trunk/98374612fc2febd686be20761e56bdc2424bc36a 2025-09-07T06:42:01.0296779Z * [new tag] trunk/98efc9e93d8fc61eb53cb91378443617cb550500 -> trunk/98efc9e93d8fc61eb53cb91378443617cb550500 2025-09-07T06:42:01.0297041Z * [new tag] trunk/994f2a5dbcbdc915da39bf6f6ce4d1f5e74835c9 -> trunk/994f2a5dbcbdc915da39bf6f6ce4d1f5e74835c9 2025-09-07T06:42:01.0297276Z * [new tag] trunk/99f356fa58c8d726cef022d8710f5491291158f6 -> trunk/99f356fa58c8d726cef022d8710f5491291158f6 2025-09-07T06:42:01.0297518Z * [new tag] trunk/9a1c5c0a078b94d13ac5c1ae0d754d19fb73bf99 -> trunk/9a1c5c0a078b94d13ac5c1ae0d754d19fb73bf99 2025-09-07T06:42:01.0297770Z * [new tag] trunk/9a665ca3c472384e9d722bddba79e5a7680f1abd -> trunk/9a665ca3c472384e9d722bddba79e5a7680f1abd 2025-09-07T06:42:01.0298025Z * [new tag] trunk/9aedb3cd87b52160872173c177f61053d97bed57 -> trunk/9aedb3cd87b52160872173c177f61053d97bed57 2025-09-07T06:42:01.0298266Z * [new tag] trunk/9b81fe281da41f2421506339d26b027a468902f4 -> trunk/9b81fe281da41f2421506339d26b027a468902f4 2025-09-07T06:42:01.0298514Z * [new tag] trunk/9bdcee01f86e2969cff1140cdecfca13cb51816e -> trunk/9bdcee01f86e2969cff1140cdecfca13cb51816e 2025-09-07T06:42:01.0298770Z * [new tag] trunk/9c03d6be87eedc06e524e202e07a7e776551a839 -> trunk/9c03d6be87eedc06e524e202e07a7e776551a839 2025-09-07T06:42:01.0299004Z * [new tag] trunk/9c957723a0fedd9c637e63e023a613019e2cab60 -> trunk/9c957723a0fedd9c637e63e023a613019e2cab60 2025-09-07T06:42:01.0299242Z * [new tag] trunk/9e5247f51d81735e5f1e65e80588985fa93bccc5 -> trunk/9e5247f51d81735e5f1e65e80588985fa93bccc5 2025-09-07T06:42:01.0299540Z * [new tag] trunk/9eadb37cdd699f7e8e8177a5227bfeb16184ef26 -> trunk/9eadb37cdd699f7e8e8177a5227bfeb16184ef26 2025-09-07T06:42:01.0299781Z * [new tag] trunk/a00cdc1e4159db73c9ffb3f25e93e55877709a29 -> trunk/a00cdc1e4159db73c9ffb3f25e93e55877709a29 2025-09-07T06:42:01.0300032Z * [new tag] trunk/a02ee4a816d11380c6f564c1aba64d56af5ba705 -> trunk/a02ee4a816d11380c6f564c1aba64d56af5ba705 2025-09-07T06:42:01.0300262Z * [new tag] trunk/a3c7f77e50f900721817934120d60c2361b3c40d -> trunk/a3c7f77e50f900721817934120d60c2361b3c40d 2025-09-07T06:42:01.0300511Z * [new tag] trunk/a3d72b09ae12126a2b7d4a63a45ac100a882a802 -> trunk/a3d72b09ae12126a2b7d4a63a45ac100a882a802 2025-09-07T06:42:01.0300745Z * [new tag] trunk/a3e5466002791da609fcb069155d8ee347baee92 -> trunk/a3e5466002791da609fcb069155d8ee347baee92 2025-09-07T06:42:01.0300995Z * [new tag] trunk/a714437093ed196eee28f7de454cf4c41badc098 -> trunk/a714437093ed196eee28f7de454cf4c41badc098 2025-09-07T06:42:01.0301281Z * [new tag] trunk/a75e8cd27098f290de0b7439685d05ce02e91356 -> trunk/a75e8cd27098f290de0b7439685d05ce02e91356 2025-09-07T06:42:01.0301530Z * [new tag] trunk/a8d6943d36c1c2a5f90d3573460695bad4b623ae -> trunk/a8d6943d36c1c2a5f90d3573460695bad4b623ae 2025-09-07T06:42:01.0302224Z * [new tag] trunk/a918bbad6ab20649ff82eefb48417ecbe96bcb34 -> trunk/a918bbad6ab20649ff82eefb48417ecbe96bcb34 2025-09-07T06:42:01.0302835Z * [new tag] trunk/a99d8d39bc842d6ebc3e368b178e4884d24b056e -> trunk/a99d8d39bc842d6ebc3e368b178e4884d24b056e 2025-09-07T06:42:01.0303400Z * [new tag] trunk/aac1a50a191b4102d566c9c1ea22f06d6c2e3f02 -> trunk/aac1a50a191b4102d566c9c1ea22f06d6c2e3f02 2025-09-07T06:42:01.0303991Z * [new tag] trunk/aad96a202244c7d0d120c04ba8db593edd8c0f92 -> trunk/aad96a202244c7d0d120c04ba8db593edd8c0f92 2025-09-07T06:42:01.0304591Z * [new tag] trunk/ab643e4dbbaf7b663d4237514cbf01af9b11565c -> trunk/ab643e4dbbaf7b663d4237514cbf01af9b11565c 2025-09-07T06:42:01.0305224Z * [new tag] trunk/abc447174cd2cf8591edbc70a9f836f9a5779f47 -> trunk/abc447174cd2cf8591edbc70a9f836f9a5779f47 2025-09-07T06:42:01.0306141Z * [new tag] trunk/acece97c3a9dceb63194e314da93fdf37cf15a0d -> trunk/acece97c3a9dceb63194e314da93fdf37cf15a0d 2025-09-07T06:42:01.0306664Z * [new tag] trunk/adae7f66aacf3f248c3101b858cf98d5809119fa -> trunk/adae7f66aacf3f248c3101b858cf98d5809119fa 2025-09-07T06:42:01.0307284Z * [new tag] trunk/ae0edc133e61e3b16caf0b2ee0ff3f33ab72af4c -> trunk/ae0edc133e61e3b16caf0b2ee0ff3f33ab72af4c 2025-09-07T06:42:01.0308204Z * [new tag] trunk/aed33a8fcbd60b052d4559d261390c5797129c6d -> trunk/aed33a8fcbd60b052d4559d261390c5797129c6d 2025-09-07T06:42:01.0308445Z * [new tag] trunk/b04e922712080a3652e438d05e8bb74e0cd2d238 -> trunk/b04e922712080a3652e438d05e8bb74e0cd2d238 2025-09-07T06:42:01.0313271Z * [new tag] trunk/b0a3e58dd71c1a039ac0ef51e5bd8f704f632f6f -> trunk/b0a3e58dd71c1a039ac0ef51e5bd8f704f632f6f 2025-09-07T06:42:01.0313562Z * [new tag] trunk/b16d3f4c8c01d461c2f01064e9ca5fa2b33f5cf1 -> trunk/b16d3f4c8c01d461c2f01064e9ca5fa2b33f5cf1 2025-09-07T06:42:01.0313797Z * [new tag] trunk/b18bb6796f210a183e687d9d64984a5a9d13cf09 -> trunk/b18bb6796f210a183e687d9d64984a5a9d13cf09 2025-09-07T06:42:01.0314055Z * [new tag] trunk/b1bb98ddebdd3e41bf7987372409bdce96ae55de -> trunk/b1bb98ddebdd3e41bf7987372409bdce96ae55de 2025-09-07T06:42:01.0314281Z * [new tag] trunk/b2b4add0e754411372060e1d7b4057a66439172b -> trunk/b2b4add0e754411372060e1d7b4057a66439172b 2025-09-07T06:42:01.0314521Z * [new tag] trunk/b2c7b9ad2dc5a7c0b61febd307761bd5bc2f0f05 -> trunk/b2c7b9ad2dc5a7c0b61febd307761bd5bc2f0f05 2025-09-07T06:42:01.0314746Z * [new tag] trunk/b40d9432be44a6b5974ee62e7d19c3c61c5ece37 -> trunk/b40d9432be44a6b5974ee62e7d19c3c61c5ece37 2025-09-07T06:42:01.0315157Z * [new tag] trunk/b4ad38279b178b7bd14355123c1101e2e853e77b -> trunk/b4ad38279b178b7bd14355123c1101e2e853e77b 2025-09-07T06:42:01.0315391Z * [new tag] trunk/b67c41039835bd9b20b83cd6233e86baaa5f5dde -> trunk/b67c41039835bd9b20b83cd6233e86baaa5f5dde 2025-09-07T06:42:01.0315628Z * [new tag] trunk/b6d0a9ea9056ede4f7024dbf3bd6c43be3aff49c -> trunk/b6d0a9ea9056ede4f7024dbf3bd6c43be3aff49c 2025-09-07T06:42:01.0315880Z * [new tag] trunk/b7dad7dd49448c88d0751fa2e29c70afe985f734 -> trunk/b7dad7dd49448c88d0751fa2e29c70afe985f734 2025-09-07T06:42:01.0316147Z * [new tag] trunk/b7e207ca9f046ddd716076965a0cce403ba99052 -> trunk/b7e207ca9f046ddd716076965a0cce403ba99052 2025-09-07T06:42:01.0317059Z * [new tag] trunk/b919560c4a7010e2d89facee25586269a994746e -> trunk/b919560c4a7010e2d89facee25586269a994746e 2025-09-07T06:42:01.0317457Z * [new tag] trunk/b9ba612f7a968f7b27e121ca8f4d0a4d954f5354 -> trunk/b9ba612f7a968f7b27e121ca8f4d0a4d954f5354 2025-09-07T06:42:01.0320362Z * [new tag] trunk/ba7f546ccccb5e0b36d9070dc25f26a9647f89f8 -> trunk/ba7f546ccccb5e0b36d9070dc25f26a9647f89f8 2025-09-07T06:42:01.0320797Z * [new tag] trunk/bb950284c7e72905994bc25dd436c10e48088d85 -> trunk/bb950284c7e72905994bc25dd436c10e48088d85 2025-09-07T06:42:01.0321526Z * [new tag] trunk/bbedc71fd3267c639c38b4ec25eaa22f973d9c4d -> trunk/bbedc71fd3267c639c38b4ec25eaa22f973d9c4d 2025-09-07T06:42:01.0322166Z * [new tag] trunk/bc4db2c27fce6ff1648bdc5af31ec225d2a31f37 -> trunk/bc4db2c27fce6ff1648bdc5af31ec225d2a31f37 2025-09-07T06:42:01.0322558Z * [new tag] trunk/bc505977fb66677a09c31155c987330fbb18a865 -> trunk/bc505977fb66677a09c31155c987330fbb18a865 2025-09-07T06:42:01.0322923Z * [new tag] trunk/bd39e47feea7326afb5bbb67fcb1e69279239527 -> trunk/bd39e47feea7326afb5bbb67fcb1e69279239527 2025-09-07T06:42:01.0323615Z * [new tag] trunk/be5b03dde96638f25ffd732a4fed7e41b4cf40e1 -> trunk/be5b03dde96638f25ffd732a4fed7e41b4cf40e1 2025-09-07T06:42:01.0323926Z * [new tag] trunk/bffc7dd1f374d8408911cd22c6b3d6df39ded9b3 -> trunk/bffc7dd1f374d8408911cd22c6b3d6df39ded9b3 2025-09-07T06:42:01.0324240Z * [new tag] trunk/c024b1f5a18d5c5aee5cc2acdd4c52b24b93ffcf -> trunk/c024b1f5a18d5c5aee5cc2acdd4c52b24b93ffcf 2025-09-07T06:42:01.0324503Z * [new tag] trunk/c0983e6cc0acf71689e1851d12609e00b3f59371 -> trunk/c0983e6cc0acf71689e1851d12609e00b3f59371 2025-09-07T06:42:01.0324905Z * [new tag] trunk/c10195e723eeeedd099ed8b73eda7184ca618fad -> trunk/c10195e723eeeedd099ed8b73eda7184ca618fad 2025-09-07T06:42:01.0325283Z * [new tag] trunk/c157cf6488ade6a7ee2ce2d25b059e1335630a99 -> trunk/c157cf6488ade6a7ee2ce2d25b059e1335630a99 2025-09-07T06:42:01.0325640Z * [new tag] trunk/c2a30246172fd71d56529907ffd3c27b76b1f3a7 -> trunk/c2a30246172fd71d56529907ffd3c27b76b1f3a7 2025-09-07T06:42:01.0325936Z * [new tag] trunk/c32111149921b48bfef909293f1049e21619ed76 -> trunk/c32111149921b48bfef909293f1049e21619ed76 2025-09-07T06:42:01.0326778Z * [new tag] trunk/c37103234afc832dcad307e9016230810957c9d5 -> trunk/c37103234afc832dcad307e9016230810957c9d5 2025-09-07T06:42:01.0327339Z * [new tag] trunk/c3ceca2995cd35e1376c4b0704669bff1a81e836 -> trunk/c3ceca2995cd35e1376c4b0704669bff1a81e836 2025-09-07T06:42:01.0327947Z * [new tag] trunk/c3d54dea9febb1236d48d19e5d4876a63f2e20fd -> trunk/c3d54dea9febb1236d48d19e5d4876a63f2e20fd 2025-09-07T06:42:01.0328311Z * [new tag] trunk/c465b3d52c5687fe910d35a5c75341b77f821741 -> trunk/c465b3d52c5687fe910d35a5c75341b77f821741 2025-09-07T06:42:01.0329021Z * [new tag] trunk/c5b8a10be5e89396da916d1069ffcb7135f0372b -> trunk/c5b8a10be5e89396da916d1069ffcb7135f0372b 2025-09-07T06:42:01.0329471Z * [new tag] trunk/c7e41071a08f4045bc11ab60ec366d7357d56e30 -> trunk/c7e41071a08f4045bc11ab60ec366d7357d56e30 2025-09-07T06:42:01.0330483Z * [new tag] trunk/c98ddaca6d2e19ca37aff00c4ff0cda1e9a6ff65 -> trunk/c98ddaca6d2e19ca37aff00c4ff0cda1e9a6ff65 2025-09-07T06:42:01.0330720Z * [new tag] trunk/cb1e31362c7b53acf4ac95b9f8878064c184f03b -> trunk/cb1e31362c7b53acf4ac95b9f8878064c184f03b 2025-09-07T06:42:01.0332754Z * [new tag] trunk/cbfb005f7cce79974795b148e265f594f59477c8 -> trunk/cbfb005f7cce79974795b148e265f594f59477c8 2025-09-07T06:42:01.0333038Z * [new tag] trunk/cc5bdd12401bda835291d2f3cb297132ebdbf358 -> trunk/cc5bdd12401bda835291d2f3cb297132ebdbf358 2025-09-07T06:42:01.0335515Z * [new tag] trunk/cd529b686d54bbaa443f5b310140de48422d96c7 -> trunk/cd529b686d54bbaa443f5b310140de48422d96c7 2025-09-07T06:42:01.0335759Z * [new tag] trunk/cec0ff122815582af5302360aff03676558c5c87 -> trunk/cec0ff122815582af5302360aff03676558c5c87 2025-09-07T06:42:01.0336019Z * [new tag] trunk/d11720efdb563d02cf4f7d324311fb15a755268e -> trunk/d11720efdb563d02cf4f7d324311fb15a755268e 2025-09-07T06:42:01.0336243Z * [new tag] trunk/d1706d9128ae24d9048167e80d3fe5196d19035e -> trunk/d1706d9128ae24d9048167e80d3fe5196d19035e 2025-09-07T06:42:01.0336488Z * [new tag] trunk/d1a15abfdcaef138f2d9e93a9f46be44f30b766d -> trunk/d1a15abfdcaef138f2d9e93a9f46be44f30b766d 2025-09-07T06:42:01.0336851Z * [new tag] trunk/d232a95d4a79404ca05c1f52d37fde7339dcdf49 -> trunk/d232a95d4a79404ca05c1f52d37fde7339dcdf49 2025-09-07T06:42:01.0339752Z * [new tag] trunk/d2d4c8e9b2371c9aacfb771d9402ac7427b9778e -> trunk/d2d4c8e9b2371c9aacfb771d9402ac7427b9778e 2025-09-07T06:42:01.0340374Z * [new tag] trunk/d33840c542b387ab08ba49aa6c45aa9567fd9be7 -> trunk/d33840c542b387ab08ba49aa6c45aa9567fd9be7 2025-09-07T06:42:01.0341004Z * [new tag] trunk/d5643e8f3a648a99636bfa1f2a41d54bd3c0d0f1 -> trunk/d5643e8f3a648a99636bfa1f2a41d54bd3c0d0f1 2025-09-07T06:42:01.0341553Z * [new tag] trunk/d5b38410b5b6cf75c7a7389972777a6497926ee7 -> trunk/d5b38410b5b6cf75c7a7389972777a6497926ee7 2025-09-07T06:42:01.0342126Z * [new tag] trunk/d5e0f4202ba14632e4d14862ace096609e763462 -> trunk/d5e0f4202ba14632e4d14862ace096609e763462 2025-09-07T06:42:01.0342728Z * [new tag] trunk/d636c181f9140a7b59be10b36eae23039fc2bb72 -> trunk/d636c181f9140a7b59be10b36eae23039fc2bb72 2025-09-07T06:42:01.0343293Z * [new tag] trunk/d64718503728001a1e78168fd7f2d4ff23e57285 -> trunk/d64718503728001a1e78168fd7f2d4ff23e57285 2025-09-07T06:42:01.0343866Z * [new tag] trunk/d67c29ad22670320d676b02e394274af34e8e643 -> trunk/d67c29ad22670320d676b02e394274af34e8e643 2025-09-07T06:42:01.0344458Z * [new tag] trunk/d6b74568e2c98ce58ecc145b72ac66d4caf7ce95 -> trunk/d6b74568e2c98ce58ecc145b72ac66d4caf7ce95 2025-09-07T06:42:01.0345034Z * [new tag] trunk/d711f27845abd45007ccab6076649ebd896c2661 -> trunk/d711f27845abd45007ccab6076649ebd896c2661 2025-09-07T06:42:01.0345614Z * [new tag] trunk/d9d6dde0f42d4bcc8c97671ac50d5096c7e500ab -> trunk/d9d6dde0f42d4bcc8c97671ac50d5096c7e500ab 2025-09-07T06:42:01.0346448Z * [new tag] trunk/da4db4b33d1fdd046650cf19fdbac581a19bf2f9 -> trunk/da4db4b33d1fdd046650cf19fdbac581a19bf2f9 2025-09-07T06:42:01.0347104Z * [new tag] trunk/dac8a4b91c01c3bbc96f54e621b1ea4ffdbd29d1 -> trunk/dac8a4b91c01c3bbc96f54e621b1ea4ffdbd29d1 2025-09-07T06:42:01.0347671Z * [new tag] trunk/dbec08729fb9848bebed6048c63831b87170d061 -> trunk/dbec08729fb9848bebed6048c63831b87170d061 2025-09-07T06:42:01.0348285Z * [new tag] trunk/dcf385395d838f38c8dca25913578230dd43099a -> trunk/dcf385395d838f38c8dca25913578230dd43099a 2025-09-07T06:42:01.0348895Z * [new tag] trunk/dd2519abe83ec3c40d4797492434e41fe3b47e17 -> trunk/dd2519abe83ec3c40d4797492434e41fe3b47e17 2025-09-07T06:42:01.0349504Z * [new tag] trunk/dec72ea4b006dd0fbcaaaa106ad273d73807ab9d -> trunk/dec72ea4b006dd0fbcaaaa106ad273d73807ab9d 2025-09-07T06:42:01.0350049Z * [new tag] trunk/e0a62b266c021b910ce6dc02a6c9429210487717 -> trunk/e0a62b266c021b910ce6dc02a6c9429210487717 2025-09-07T06:42:01.0350596Z * [new tag] trunk/e19e02c84c9dcc408375e5cae3b0709c18b99228 -> trunk/e19e02c84c9dcc408375e5cae3b0709c18b99228 2025-09-07T06:42:01.0351151Z * [new tag] trunk/e304ea4e69d3a7deeb7e48c7450c214a4c953937 -> trunk/e304ea4e69d3a7deeb7e48c7450c214a4c953937 2025-09-07T06:42:01.0351713Z * [new tag] trunk/e3068cdb446adefb5a875616ba37a60235391439 -> trunk/e3068cdb446adefb5a875616ba37a60235391439 2025-09-07T06:42:01.0352250Z * [new tag] trunk/e381d4b0205d5f126c1de534f867ba776f7c3ee6 -> trunk/e381d4b0205d5f126c1de534f867ba776f7c3ee6 2025-09-07T06:42:01.0353190Z * [new tag] trunk/e4bd0ff4f8981b805df32ea5b3550621965ea4f2 -> trunk/e4bd0ff4f8981b805df32ea5b3550621965ea4f2 2025-09-07T06:42:01.0353746Z * [new tag] trunk/e532c9d4f1cdcbc1ea9628f55b9813e77847bdc7 -> trunk/e532c9d4f1cdcbc1ea9628f55b9813e77847bdc7 2025-09-07T06:42:01.0354281Z * [new tag] trunk/e92cd9415377403b6e90585e764639e2e0b5973b -> trunk/e92cd9415377403b6e90585e764639e2e0b5973b 2025-09-07T06:42:01.0354965Z * [new tag] trunk/e9481b6617b5576b099d8ca5798111592e9ad090 -> trunk/e9481b6617b5576b099d8ca5798111592e9ad090 2025-09-07T06:42:01.0357058Z * [new tag] trunk/ea1883dfd3e42defe37b11202b878bb76defa087 -> trunk/ea1883dfd3e42defe37b11202b878bb76defa087 2025-09-07T06:42:01.0357696Z * [new tag] trunk/eac3d6f04cfbbebe3d470dacd216da7d4b1f95a8 -> trunk/eac3d6f04cfbbebe3d470dacd216da7d4b1f95a8 2025-09-07T06:42:01.0361004Z * [new tag] trunk/eb18d32bda75189494d955aa001ade15f10333de -> trunk/eb18d32bda75189494d955aa001ade15f10333de 2025-09-07T06:42:01.0361634Z * [new tag] trunk/ef3be6726f7ff4b77c22db10cec5b686f9107ea9 -> trunk/ef3be6726f7ff4b77c22db10cec5b686f9107ea9 2025-09-07T06:42:01.0366521Z * [new tag] trunk/ef8aabd42422725026cb4dbf48aafa9efa226a04 -> trunk/ef8aabd42422725026cb4dbf48aafa9efa226a04 2025-09-07T06:42:01.0367164Z * [new tag] trunk/f00445b43eee57e20bb9316fa796ca23bf73373b -> trunk/f00445b43eee57e20bb9316fa796ca23bf73373b 2025-09-07T06:42:01.0367771Z * [new tag] trunk/f0c391102b754e3b145e8c59231d2df563487e37 -> trunk/f0c391102b754e3b145e8c59231d2df563487e37 2025-09-07T06:42:01.0368326Z * [new tag] trunk/f27985b7e796fb66a1b476284ba42d8cb360a751 -> trunk/f27985b7e796fb66a1b476284ba42d8cb360a751 2025-09-07T06:42:01.0368894Z * [new tag] trunk/f36f285953700f971552083a5da9d0ceacb63bbd -> trunk/f36f285953700f971552083a5da9d0ceacb63bbd 2025-09-07T06:42:01.0369500Z * [new tag] trunk/f3cebec39ebc110e1c8b06e741896585f7892dbb -> trunk/f3cebec39ebc110e1c8b06e741896585f7892dbb 2025-09-07T06:42:01.0370053Z * [new tag] trunk/f4c33cd44acac92c0b451a04da20ebe9370e5b0c -> trunk/f4c33cd44acac92c0b451a04da20ebe9370e5b0c 2025-09-07T06:42:01.0370623Z * [new tag] trunk/f612045ce105f008b2b675e2fc870163babeb2e8 -> trunk/f612045ce105f008b2b675e2fc870163babeb2e8 2025-09-07T06:42:01.0371182Z * [new tag] trunk/f8746b878dfc1e9639d42cbde832e9b9e792c86c -> trunk/f8746b878dfc1e9639d42cbde832e9b9e792c86c 2025-09-07T06:42:01.0371757Z * [new tag] trunk/f8ffa9194e26523e5f976d4a824d5cc58922727c -> trunk/f8ffa9194e26523e5f976d4a824d5cc58922727c 2025-09-07T06:42:01.0372318Z * [new tag] trunk/f981a7fa5230b98974291fdde32fe8488bc5d469 -> trunk/f981a7fa5230b98974291fdde32fe8488bc5d469 2025-09-07T06:42:01.0372875Z * [new tag] trunk/fbf3d2027daabbcb44d0af274b139be2a248a4f7 -> trunk/fbf3d2027daabbcb44d0af274b139be2a248a4f7 2025-09-07T06:42:01.0373703Z * [new tag] trunk/fca2601c9d628e1bd2d75c7318cd22c4e8c832aa -> trunk/fca2601c9d628e1bd2d75c7318cd22c4e8c832aa 2025-09-07T06:42:01.0374274Z * [new tag] trunk/fea20775ad96bdca972a1811d7d3372f368614ab -> trunk/fea20775ad96bdca972a1811d7d3372f368614ab 2025-09-07T06:42:01.0374841Z * [new tag] trunk/fefee081642f87419a21dc852f7167d4640443cd -> trunk/fefee081642f87419a21dc852f7167d4640443cd 2025-09-07T06:42:01.0375272Z * [new tag] v0.1.1 -> v0.1.1 2025-09-07T06:42:01.0375556Z * [new tag] v0.1.10 -> v0.1.10 2025-09-07T06:42:01.0375821Z * [new tag] v0.1.11 -> v0.1.11 2025-09-07T06:42:01.0376131Z * [new tag] v0.1.12 -> v0.1.12 2025-09-07T06:42:01.0376403Z * [new tag] v0.1.2 -> v0.1.2 2025-09-07T06:42:01.0376663Z * [new tag] v0.1.3 -> v0.1.3 2025-09-07T06:42:01.0376947Z * [new tag] v0.1.4 -> v0.1.4 2025-09-07T06:42:01.0377204Z * [new tag] v0.1.5 -> v0.1.5 2025-09-07T06:42:01.0377456Z * [new tag] v0.1.6 -> v0.1.6 2025-09-07T06:42:01.0377717Z * [new tag] v0.1.7 -> v0.1.7 2025-09-07T06:42:01.0377965Z * [new tag] v0.1.8 -> v0.1.8 2025-09-07T06:42:01.0378278Z * [new tag] v0.1.9 -> v0.1.9 2025-09-07T06:42:01.0378533Z * [new tag] v0.2.0 -> v0.2.0 2025-09-07T06:42:01.0378784Z * [new tag] v0.3.0 -> v0.3.0 2025-09-07T06:42:01.0379035Z * [new tag] v0.3.1 -> v0.3.1 2025-09-07T06:42:01.0379287Z * [new tag] v0.4.0 -> v0.4.0 2025-09-07T06:42:01.0379535Z * [new tag] v0.4.1 -> v0.4.1 2025-09-07T06:42:01.0379791Z * [new tag] v1.0.0 -> v1.0.0 2025-09-07T06:42:01.0380059Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-09-07T06:42:01.0380321Z * [new tag] v1.0.1 -> v1.0.1 2025-09-07T06:42:01.0380773Z * [new tag] v1.0rc0 -> v1.0rc0 2025-09-07T06:42:01.0381179Z * [new tag] v1.0rc1 -> v1.0rc1 2025-09-07T06:42:01.0381575Z * [new tag] v1.1.0 -> v1.1.0 2025-09-07T06:42:01.0382296Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-09-07T06:42:01.0382671Z * [new tag] v1.10.0 -> v1.10.0 2025-09-07T06:42:01.0382971Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-09-07T06:42:01.0383264Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-09-07T06:42:01.0383577Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-09-07T06:42:01.0383856Z * [new tag] v1.10.1 -> v1.10.1 2025-09-07T06:42:01.0384147Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-09-07T06:42:01.0384434Z * [new tag] v1.10.2 -> v1.10.2 2025-09-07T06:42:01.0384760Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-09-07T06:42:01.0385046Z * [new tag] v1.11.0 -> v1.11.0 2025-09-07T06:42:01.0385313Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-09-07T06:42:01.0385586Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-09-07T06:42:01.0386051Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-09-07T06:42:01.0386346Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-09-07T06:42:01.0386778Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-09-07T06:42:01.0387064Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-09-07T06:42:01.0387351Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-09-07T06:42:01.0387633Z * [new tag] v1.12.0 -> v1.12.0 2025-09-07T06:42:01.0387911Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-09-07T06:42:01.0388180Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-09-07T06:42:01.0388451Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-09-07T06:42:01.0388721Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-09-07T06:42:01.0389350Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-09-07T06:42:01.0390456Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-09-07T06:42:01.0390786Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-09-07T06:42:01.0391056Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-09-07T06:42:01.0391486Z * [new tag] v1.12.1 -> v1.12.1 2025-09-07T06:42:01.0391938Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-09-07T06:42:01.0392349Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-09-07T06:42:01.0393875Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-09-07T06:42:01.0394536Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-09-07T06:42:01.0394976Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-09-07T06:42:01.0395397Z * [new tag] v1.13.0 -> v1.13.0 2025-09-07T06:42:01.0395945Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-09-07T06:42:01.0396450Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-09-07T06:42:01.0397001Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-09-07T06:42:01.0397957Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-09-07T06:42:01.0398368Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-09-07T06:42:01.0398770Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-09-07T06:42:01.0399182Z * [new tag] v1.13.1 -> v1.13.1 2025-09-07T06:42:01.0399604Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-09-07T06:42:01.0401983Z * [new tag] v1.2.0 -> v1.2.0 2025-09-07T06:42:01.0402530Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-09-07T06:42:01.0402964Z * [new tag] v1.3.0 -> v1.3.0 2025-09-07T06:42:01.0403383Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-09-07T06:42:01.0403686Z * [new tag] v1.3.1 -> v1.3.1 2025-09-07T06:42:01.0403956Z * [new tag] v1.4.0 -> v1.4.0 2025-09-07T06:42:01.0404226Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-09-07T06:42:01.0404489Z * [new tag] v1.4.1 -> v1.4.1 2025-09-07T06:42:01.0406402Z * [new tag] v1.5.0 -> v1.5.0 2025-09-07T06:42:01.0406929Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-09-07T06:42:01.0407365Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-09-07T06:42:01.0407838Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-09-07T06:42:01.0408293Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-09-07T06:42:01.0408989Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-09-07T06:42:01.0409501Z * [new tag] v1.5.1 -> v1.5.1 2025-09-07T06:42:01.0409789Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-09-07T06:42:01.0410076Z * [new tag] v1.6.0 -> v1.6.0 2025-09-07T06:42:01.0410502Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-09-07T06:42:01.0410794Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-09-07T06:42:01.0411149Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-09-07T06:42:01.0411672Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-09-07T06:42:01.0412381Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-09-07T06:42:01.0413007Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-09-07T06:42:01.0413604Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-09-07T06:42:01.0413980Z * [new tag] v1.7.0 -> v1.7.0 2025-09-07T06:42:01.0414562Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-09-07T06:42:01.0417113Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-09-07T06:42:01.0417446Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-09-07T06:42:01.0417718Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-09-07T06:42:01.0417988Z * [new tag] v1.7.1 -> v1.7.1 2025-09-07T06:42:01.0418578Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-09-07T06:42:01.0419397Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-09-07T06:42:01.0419982Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-09-07T06:42:01.0420295Z * [new tag] v1.8.0 -> v1.8.0 2025-09-07T06:42:01.0420643Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-09-07T06:42:01.0420979Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-09-07T06:42:01.0421329Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-09-07T06:42:01.0422618Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-09-07T06:42:01.0422919Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-09-07T06:42:01.0423219Z * [new tag] v1.8.1 -> v1.8.1 2025-09-07T06:42:01.0423515Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-09-07T06:42:01.0423789Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-09-07T06:42:01.0424073Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-09-07T06:42:01.0425977Z * [new tag] v1.8.2 -> v1.8.2 2025-09-07T06:42:01.0426325Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-09-07T06:42:01.0426618Z * [new tag] v1.9.0 -> v1.9.0 2025-09-07T06:42:01.0427136Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-09-07T06:42:01.0427694Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-09-07T06:42:01.0427993Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-09-07T06:42:01.0428432Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-09-07T06:42:01.0429436Z * [new tag] v1.9.1 -> v1.9.1 2025-09-07T06:42:01.0430072Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-09-07T06:42:01.0430337Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-09-07T06:42:01.0431648Z * [new tag] v2.0.0 -> v2.0.0 2025-09-07T06:42:01.0431921Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-09-07T06:42:01.0432822Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-09-07T06:42:01.0433485Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-09-07T06:42:01.0436181Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-09-07T06:42:01.0436458Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-09-07T06:42:01.0436763Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-09-07T06:42:01.0437034Z * [new tag] v2.0.1 -> v2.0.1 2025-09-07T06:42:01.0437298Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-09-07T06:42:01.0437555Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-09-07T06:42:01.0441195Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-09-07T06:42:01.0441472Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-09-07T06:42:01.0441735Z * [new tag] v2.1.0 -> v2.1.0 2025-09-07T06:42:01.0442033Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-09-07T06:42:01.0442308Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-09-07T06:42:01.0442583Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-09-07T06:42:01.0442859Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-09-07T06:42:01.0444389Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-09-07T06:42:01.0444758Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-09-07T06:42:01.0445020Z * [new tag] v2.1.1 -> v2.1.1 2025-09-07T06:42:01.0445286Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-09-07T06:42:01.0445553Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-09-07T06:42:01.0445819Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-09-07T06:42:01.0446227Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-09-07T06:42:01.0446621Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-09-07T06:42:01.0446910Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-09-07T06:42:01.0447187Z * [new tag] v2.1.2 -> v2.1.2 2025-09-07T06:42:01.0447703Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-09-07T06:42:01.0448293Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-09-07T06:42:01.0448766Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-09-07T06:42:01.0452260Z * [new tag] v2.2.0 -> v2.2.0 2025-09-07T06:42:01.0452809Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-09-07T06:42:01.0453246Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-09-07T06:42:01.0454110Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-09-07T06:42:01.0456735Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-09-07T06:42:01.0457164Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-09-07T06:42:01.0457557Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-09-07T06:42:01.0457973Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-09-07T06:42:01.0458337Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-09-07T06:42:01.0458756Z * [new tag] v2.2.1 -> v2.2.1 2025-09-07T06:42:01.0459160Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-09-07T06:42:01.0459439Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-09-07T06:42:01.0459722Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-09-07T06:42:01.0459994Z * [new tag] v2.2.2 -> v2.2.2 2025-09-07T06:42:01.0460404Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-09-07T06:42:01.0460675Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-09-07T06:42:01.0460949Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-09-07T06:42:01.0461215Z * [new tag] v2.3.0 -> v2.3.0 2025-09-07T06:42:01.0461469Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-09-07T06:42:01.0461750Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-09-07T06:42:01.0462031Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-09-07T06:42:01.0462314Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-09-07T06:42:01.0462592Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-09-07T06:42:01.0462860Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-09-07T06:42:01.0463140Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-09-07T06:42:01.0463829Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-09-07T06:42:01.0464107Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-09-07T06:42:01.0465149Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-09-07T06:42:01.0465424Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-09-07T06:42:01.0466126Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-09-07T06:42:01.0467661Z * [new tag] v2.3.1 -> v2.3.1 2025-09-07T06:42:01.0467947Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-09-07T06:42:01.0468205Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-09-07T06:42:01.0468959Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-09-07T06:42:01.0472928Z * [new tag] v2.4.0 -> v2.4.0 2025-09-07T06:42:01.0473359Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-09-07T06:42:01.0473769Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-09-07T06:42:01.0474165Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-09-07T06:42:01.0474975Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-09-07T06:42:01.0475303Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-09-07T06:42:01.0475601Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-09-07T06:42:01.0475875Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-09-07T06:42:01.0480331Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-09-07T06:42:01.0481760Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-09-07T06:42:01.0482077Z * [new tag] v2.4.1 -> v2.4.1 2025-09-07T06:42:01.0482362Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-09-07T06:42:01.0482637Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-09-07T06:42:01.0482910Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-09-07T06:42:01.0483181Z * [new tag] v2.5.0 -> v2.5.0 2025-09-07T06:42:01.0483446Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-09-07T06:42:01.0483737Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-09-07T06:42:01.0484013Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-09-07T06:42:01.0484277Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-09-07T06:42:01.0484552Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-09-07T06:42:01.0484806Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-09-07T06:42:01.0485216Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-09-07T06:42:01.0485481Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-09-07T06:42:01.0485743Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-09-07T06:42:01.0485998Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-09-07T06:42:01.0486263Z * [new tag] v2.5.1 -> v2.5.1 2025-09-07T06:42:01.0486528Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-09-07T06:42:01.0486791Z * [new tag] v2.6.0 -> v2.6.0 2025-09-07T06:42:01.0487078Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-09-07T06:42:01.0487336Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-09-07T06:42:01.0487605Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-09-07T06:42:01.0487872Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-09-07T06:42:01.0488129Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-09-07T06:42:01.0488388Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-09-07T06:42:01.0488640Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-09-07T06:42:01.0492149Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-09-07T06:42:01.0494636Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-09-07T06:42:01.0495089Z * [new tag] v2.7.0 -> v2.7.0 2025-09-07T06:42:01.0495391Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-09-07T06:42:01.0495688Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-09-07T06:42:01.0495980Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-09-07T06:42:01.0496271Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-09-07T06:42:01.0496535Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-09-07T06:42:01.0496797Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-09-07T06:42:01.0497058Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-09-07T06:42:01.0497317Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-09-07T06:42:01.0497569Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-09-07T06:42:01.0497836Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-09-07T06:42:01.0498107Z * [new tag] v2.7.1 -> v2.7.1 2025-09-07T06:42:01.0500517Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-09-07T06:42:01.0500818Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-09-07T06:42:01.0501081Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-09-07T06:42:01.0501355Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-09-07T06:42:01.0501617Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-09-07T06:42:01.0501882Z * [new tag] v2.8.0 -> v2.8.0 2025-09-07T06:42:01.0502144Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-09-07T06:42:01.0502415Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-09-07T06:42:01.0504477Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-09-07T06:42:01.0504783Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-09-07T06:42:01.0505056Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-09-07T06:42:01.0505330Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-09-07T06:42:01.0505594Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-09-07T06:42:01.0506252Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-09-07T06:42:01.0506542Z * [new tag] whc_flight_1 -> whc_flight_1 2025-09-07T06:42:01.0506837Z * [new tag] whc_flight_2 -> whc_flight_2 2025-09-07T06:42:01.0509277Z * [new tag] whc_flight_4 -> whc_flight_4 2025-09-07T06:42:01.0982959Z [command]/usr/bin/git rev-parse --verify --quiet 93fb23d6fae7c4e82c4239a1033e522088742634^{object} 2025-09-07T06:42:01.1013571Z 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:42:01.1014665Z ##[endgroup] 2025-09-07T06:42:01.1015001Z ##[group]Determining the checkout info 2025-09-07T06:42:01.1015858Z ##[endgroup] 2025-09-07T06:42:01.1020870Z [command]/usr/bin/git sparse-checkout disable 2025-09-07T06:42:01.1068289Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-09-07T06:42:01.1093858Z ##[group]Checking out the ref 2025-09-07T06:42:01.1099010Z [command]/usr/bin/git checkout --progress --force 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:42:02.1488305Z Updating files: 98% (19033/19405) 2025-09-07T06:42:02.1612100Z Updating files: 99% (19211/19405) 2025-09-07T06:42:02.1612428Z Updating files: 100% (19405/19405) 2025-09-07T06:42:02.1612645Z Updating files: 100% (19405/19405), done. 2025-09-07T06:42:02.1841772Z Note: switching to '93fb23d6fae7c4e82c4239a1033e522088742634'. 2025-09-07T06:42:02.1842065Z 2025-09-07T06:42:02.1842264Z You are in 'detached HEAD' state. You can look around, make experimental 2025-09-07T06:42:02.1842954Z changes and commit them, and you can discard any commits you make in this 2025-09-07T06:42:02.1843348Z state without impacting any branches by switching back to a branch. 2025-09-07T06:42:02.1843544Z 2025-09-07T06:42:02.1843689Z If you want to create a new branch to retain commits you create, you may 2025-09-07T06:42:02.1844008Z do so (now or later) by using -c with the switch command. Example: 2025-09-07T06:42:02.1844209Z 2025-09-07T06:42:02.1844297Z git switch -c 2025-09-07T06:42:02.1844440Z 2025-09-07T06:42:02.1844546Z Or undo this operation with: 2025-09-07T06:42:02.1844680Z 2025-09-07T06:42:02.1844755Z git switch - 2025-09-07T06:42:02.1844847Z 2025-09-07T06:42:02.1845008Z Turn off this advice by setting config variable advice.detachedHead to false 2025-09-07T06:42:02.1845218Z 2025-09-07T06:42:02.1845339Z HEAD is now at 93fb23d6fae Build vLLM nightly wheels (#162000) 2025-09-07T06:42:02.1891182Z ##[endgroup] 2025-09-07T06:42:02.1891593Z ##[group]Setting up auth for fetching submodules 2025-09-07T06:42:02.1895275Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-09-07T06:42:02.1975549Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-09-07T06:42:02.2007563Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-09-07T06:42:02.2042628Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-09-07T06:42:02.2085664Z ##[endgroup] 2025-09-07T06:42:02.2086354Z ##[group]Fetching submodules 2025-09-07T06:42:02.2086764Z [command]/usr/bin/git submodule sync --recursive 2025-09-07T06:42:02.2418468Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-09-07T06:42:02.3142452Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-09-07T06:42:02.3143193Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-09-07T06:42:02.3143822Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-09-07T06:42:02.3163321Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-09-07T06:42:02.3168488Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-09-07T06:42:02.3169385Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-09-07T06:42:02.3170113Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-09-07T06:42:02.3196006Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-09-07T06:42:02.3196737Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-09-07T06:42:02.3197462Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-09-07T06:42:02.3208736Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-09-07T06:42:02.3209353Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-09-07T06:42:02.3211256Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-09-07T06:42:02.3214226Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-09-07T06:42:02.3233233Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-09-07T06:42:02.3234445Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-09-07T06:42:02.3242614Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-09-07T06:42:02.3258359Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-09-07T06:42:02.3259456Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-09-07T06:42:02.3260334Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-09-07T06:42:02.3261120Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-09-07T06:42:02.3277308Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-09-07T06:42:02.3279917Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-09-07T06:42:02.3286777Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-09-07T06:42:02.3300600Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-09-07T06:42:02.3303107Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-09-07T06:42:02.3307988Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-09-07T06:42:02.3308889Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-09-07T06:42:02.3333841Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-09-07T06:42:02.3334683Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-09-07T06:42:02.3335326Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-09-07T06:42:02.3358175Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-09-07T06:42:02.3359485Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-09-07T06:42:02.3364126Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-09-07T06:42:02.3367680Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-09-07T06:42:02.3386502Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-09-07T06:42:02.3387422Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-09-07T06:42:02.3422800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-09-07T06:42:02.5634174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-09-07T06:42:02.5635076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-09-07T06:42:02.5635879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-09-07T06:42:02.5662303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-09-07T06:42:02.8860488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-09-07T06:42:02.8861988Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-09-07T06:42:02.8863016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-09-07T06:42:02.8864026Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-09-07T06:42:02.8865014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-09-07T06:42:02.8866118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-09-07T06:42:02.8867160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-09-07T06:42:02.8868046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-09-07T06:42:02.8868983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-09-07T06:42:02.9069896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-09-07T06:42:02.9773262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-09-07T06:42:04.1365761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-09-07T06:42:04.1366339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-09-07T06:42:04.1366847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-09-07T06:42:04.1367406Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-09-07T06:42:04.1367887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-09-07T06:42:04.1368377Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-09-07T06:42:04.1368855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-09-07T06:42:04.1369342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-09-07T06:42:04.1369834Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-09-07T06:42:04.1370293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-09-07T06:42:04.1370753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-09-07T06:42:04.2273877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-09-07T06:42:16.3624676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-09-07T06:42:16.3625309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-09-07T06:42:16.3625865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-09-07T06:42:16.3626375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-09-07T06:42:16.3626934Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-09-07T06:42:16.3627478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-09-07T06:42:16.3628040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-09-07T06:42:16.3628587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-09-07T06:42:16.3629104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-09-07T06:42:16.3762005Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-09-07T06:42:16.3876134Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-09-07T06:42:16.3965351Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-09-07T06:42:16.4168868Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-09-07T06:42:16.4834314Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-09-07T06:42:16.5300848Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-09-07T06:42:17.0752865Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-09-07T06:42:17.2087879Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-09-07T06:42:17.2103764Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T06:42:17.2133521Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-09-07T06:42:21.1720026Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-09-07T06:42:21.1933579Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-09-07T06:42:21.4520746Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-09-07T06:42:21.4927294Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-09-07T06:42:21.5811836Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-09-07T06:42:21.6216625Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6' 2025-09-07T06:42:22.1525797Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491' 2025-09-07T06:42:22.2751724Z Submodule path 'third_party/fbgemm': checked out '4b39c551efe15e6bbade20565b0ceb2d8ce3352d' 2025-09-07T06:42:22.2769343Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-09-07T06:42:22.2773481Z Submodule 'external/composable_kernel' (https://github.com/jwfromm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-09-07T06:42:22.2774242Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-09-07T06:42:22.2774869Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-09-07T06:42:22.2775821Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-09-07T06:42:22.2776570Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-09-07T06:42:22.2777275Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-09-07T06:42:22.2804472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-09-07T06:42:23.5261762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-09-07T06:42:23.5262392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-09-07T06:42:23.5262983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-09-07T06:42:23.5263572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-09-07T06:42:23.6264771Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-09-07T06:42:24.6209385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-09-07T06:42:28.8210743Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-09-07T06:42:29.0248520Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9' 2025-09-07T06:42:29.1145042Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-09-07T06:42:29.6478752Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '311f3c8e51dc0eb56310cfc6980bf63d0fbd7917' 2025-09-07T06:42:29.6900495Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-09-07T06:42:29.7017050Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-09-07T06:42:29.7912813Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-09-07T06:42:29.8502189Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-09-07T06:42:29.8516931Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T06:42:29.8517693Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-09-07T06:42:29.8542842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-09-07T06:42:33.6248049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-09-07T06:42:33.8116138Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-09-07T06:42:34.2953586Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-09-07T06:42:34.4094220Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-09-07T06:42:34.4404427Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-09-07T06:42:34.4766387Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-09-07T06:42:34.4987831Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2' 2025-09-07T06:42:34.5421881Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-09-07T06:42:34.5550588Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-09-07T06:42:34.5568191Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-09-07T06:42:34.5589775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-09-07T06:42:48.1859921Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-09-07T06:42:48.2037993Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-09-07T06:42:48.2913892Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658' 2025-09-07T06:42:48.2935578Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T06:42:48.2936428Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T06:42:48.2937179Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T06:42:48.2972760Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-09-07T06:42:48.9091386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-09-07T06:42:49.3934050Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-09-07T06:42:49.4661782Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-09-07T06:42:49.4680987Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T06:42:49.4681946Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T06:42:49.4682810Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T06:42:49.4683610Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T06:42:49.4684394Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T06:42:49.4685439Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T06:42:49.4686409Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T06:42:49.4687291Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T06:42:49.4720432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-09-07T06:42:50.9684022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-09-07T06:42:50.9684742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-09-07T06:42:50.9685408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-09-07T06:42:50.9686275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-09-07T06:42:50.9686971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-09-07T06:42:50.9687628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-09-07T06:42:51.0684375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-09-07T06:42:56.2488192Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-09-07T06:42:56.2651742Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-09-07T06:42:56.2990595Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-09-07T06:42:56.3120969Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-09-07T06:42:56.3137936Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T06:42:56.3164043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-09-07T06:42:56.8219193Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-09-07T06:42:56.8399978Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-09-07T06:42:56.8784706Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-09-07T06:42:56.9663900Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-09-07T06:42:56.9818408Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-09-07T06:42:57.0136844Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-09-07T06:42:57.0666736Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-09-07T06:42:57.1057426Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7' 2025-09-07T06:42:57.2676185Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-09-07T06:42:57.3650496Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-09-07T06:42:57.6697018Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-09-07T06:42:57.6734852Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-09-07T06:42:57.6762013Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-09-07T06:42:58.7376370Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-09-07T06:42:58.7974036Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-09-07T06:42:58.7996945Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T06:42:58.7998073Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T06:42:58.8001908Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T06:42:58.8002580Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T06:42:58.8003404Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T06:42:58.8004298Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T06:42:58.8005906Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T06:42:58.8010121Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T06:42:58.8036337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-09-07T06:42:59.1844278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-09-07T06:42:59.1845276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-09-07T06:42:59.1845946Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-09-07T06:42:59.1846632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-09-07T06:42:59.2851402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-09-07T06:42:59.8682952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-09-07T06:43:07.1307163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-09-07T06:43:07.4427476Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-09-07T06:43:07.4788799Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-09-07T06:43:07.4959410Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-09-07T06:43:07.5863269Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-09-07T06:43:07.5998448Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-09-07T06:43:07.6135516Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-09-07T06:43:07.6279817Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-09-07T06:43:07.6290932Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T06:43:07.6291823Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T06:43:07.6318873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-09-07T06:43:09.3852583Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-09-07T06:43:09.6015417Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-09-07T06:43:09.6417996Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-09-07T06:43:09.9744656Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-09-07T06:43:09.9863170Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-09-07T06:43:10.2093063Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-09-07T06:43:10.2110553Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-09-07T06:43:10.2111322Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-09-07T06:43:10.2142481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-09-07T06:43:10.7405269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-09-07T06:43:11.1695110Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-09-07T06:43:11.2328325Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-09-07T06:43:11.2420030Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-09-07T06:43:11.2541123Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-09-07T06:43:11.2880644Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-09-07T06:43:11.3123521Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-09-07T06:43:11.3502666Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-09-07T06:43:11.3716468Z Submodule path 'third_party/tensorpipe': checked out 'af0118d13e52f5a08841464a768e01a0bf3e3075' 2025-09-07T06:43:11.3735937Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-09-07T06:43:11.3741017Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-09-07T06:43:11.3741706Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-09-07T06:43:11.3742365Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T06:43:11.3768091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-09-07T06:43:12.3045019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-09-07T06:43:12.3328032Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-09-07T06:43:12.5802336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-09-07T06:43:12.6316929Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-09-07T06:43:12.6463446Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-09-07T06:43:12.7113565Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-09-07T06:43:12.7372616Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-09-07T06:43:12.7386725Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T06:43:12.7414993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-09-07T06:43:12.9380675Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-09-07T06:43:12.9417806Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-09-07T06:43:12.9742629Z Entering 'android/libs/fbjni' 2025-09-07T06:43:12.9783424Z Entering 'third_party/FP16' 2025-09-07T06:43:12.9822634Z Entering 'third_party/FXdiv' 2025-09-07T06:43:12.9861687Z Entering 'third_party/NNPACK' 2025-09-07T06:43:12.9900617Z Entering 'third_party/NVTX' 2025-09-07T06:43:12.9943679Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T06:43:12.9983054Z Entering 'third_party/XNNPACK' 2025-09-07T06:43:13.0038143Z Entering 'third_party/aiter' 2025-09-07T06:43:13.0083659Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T06:43:13.0130997Z Entering 'third_party/benchmark' 2025-09-07T06:43:13.0170397Z Entering 'third_party/composable_kernel' 2025-09-07T06:43:13.0217179Z Entering 'third_party/cpp-httplib' 2025-09-07T06:43:13.0261726Z Entering 'third_party/cpuinfo' 2025-09-07T06:43:13.0302975Z Entering 'third_party/cudnn_frontend' 2025-09-07T06:43:13.0345153Z Entering 'third_party/cutlass' 2025-09-07T06:43:13.0394304Z Entering 'third_party/fbgemm' 2025-09-07T06:43:13.0444894Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T06:43:13.0486090Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T06:43:13.0538849Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T06:43:13.0579495Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T06:43:13.0625182Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T06:43:13.0665032Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T06:43:13.0703906Z Entering 'third_party/fbgemm/external/json' 2025-09-07T06:43:13.0750291Z Entering 'third_party/flash-attention' 2025-09-07T06:43:13.0789937Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T06:43:13.0838280Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T06:43:13.0884559Z Entering 'third_party/flatbuffers' 2025-09-07T06:43:13.0928071Z Entering 'third_party/fmt' 2025-09-07T06:43:13.0970857Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T06:43:13.1009775Z Entering 'third_party/gloo' 2025-09-07T06:43:13.1054721Z Entering 'third_party/googletest' 2025-09-07T06:43:13.1098980Z Entering 'third_party/ideep' 2025-09-07T06:43:13.1138889Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T06:43:13.1187305Z Entering 'third_party/ittapi' 2025-09-07T06:43:13.1224691Z Entering 'third_party/kineto' 2025-09-07T06:43:13.1266326Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T06:43:13.1303444Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T06:43:13.1345744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T06:43:13.1389965Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T06:43:13.1430573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T06:43:13.1471733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T06:43:13.1510682Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T06:43:13.1551811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T06:43:13.1599398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T06:43:13.1641873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T06:43:13.1685138Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T06:43:13.1730464Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T06:43:13.1779093Z Entering 'third_party/kleidiai' 2025-09-07T06:43:13.1814651Z Entering 'third_party/mimalloc' 2025-09-07T06:43:13.1867001Z Entering 'third_party/nlohmann' 2025-09-07T06:43:13.1903770Z Entering 'third_party/onnx' 2025-09-07T06:43:13.1962421Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T06:43:13.2002615Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T06:43:13.2044226Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T06:43:13.2086609Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T06:43:13.2127186Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T06:43:13.2171250Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T06:43:13.2215150Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T06:43:13.2258573Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T06:43:13.2296432Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T06:43:13.2339244Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T06:43:13.2378964Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T06:43:13.2418639Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T06:43:13.2481948Z Entering 'third_party/pocketfft' 2025-09-07T06:43:13.2530270Z Entering 'third_party/protobuf' 2025-09-07T06:43:13.2576689Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T06:43:13.2614090Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T06:43:13.2664735Z Entering 'third_party/psimd' 2025-09-07T06:43:13.2702240Z Entering 'third_party/pthreadpool' 2025-09-07T06:43:13.2745920Z Entering 'third_party/pybind11' 2025-09-07T06:43:13.2784891Z Entering 'third_party/python-peachpy' 2025-09-07T06:43:13.2823793Z Entering 'third_party/sleef' 2025-09-07T06:43:13.2869942Z Entering 'third_party/tensorpipe' 2025-09-07T06:43:13.2907499Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T06:43:13.2945793Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T06:43:13.2985294Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T06:43:13.3024728Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T06:43:13.3061743Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T06:43:13.3113946Z ##[endgroup] 2025-09-07T06:43:13.3114357Z ##[group]Persisting credentials for submodules 2025-09-07T06:43:13.3121251Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-09-07T06:43:13.3428548Z Entering 'android/libs/fbjni' 2025-09-07T06:43:13.3484105Z Entering 'third_party/FP16' 2025-09-07T06:43:13.3537921Z Entering 'third_party/FXdiv' 2025-09-07T06:43:13.3587248Z Entering 'third_party/NNPACK' 2025-09-07T06:43:13.3643536Z Entering 'third_party/NVTX' 2025-09-07T06:43:13.3698253Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T06:43:13.3764886Z Entering 'third_party/XNNPACK' 2025-09-07T06:43:13.3826954Z Entering 'third_party/aiter' 2025-09-07T06:43:13.3887189Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T06:43:13.3955939Z Entering 'third_party/benchmark' 2025-09-07T06:43:13.4007123Z Entering 'third_party/composable_kernel' 2025-09-07T06:43:13.4071444Z Entering 'third_party/cpp-httplib' 2025-09-07T06:43:13.4129433Z Entering 'third_party/cpuinfo' 2025-09-07T06:43:13.4187363Z Entering 'third_party/cudnn_frontend' 2025-09-07T06:43:13.4239863Z Entering 'third_party/cutlass' 2025-09-07T06:43:13.4301719Z Entering 'third_party/fbgemm' 2025-09-07T06:43:13.4356879Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T06:43:13.4411900Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T06:43:13.4473491Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T06:43:13.4528040Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T06:43:13.4594370Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T06:43:13.4656143Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T06:43:13.4711334Z Entering 'third_party/fbgemm/external/json' 2025-09-07T06:43:13.4779236Z Entering 'third_party/flash-attention' 2025-09-07T06:43:13.4826171Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T06:43:13.4895760Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T06:43:13.4957182Z Entering 'third_party/flatbuffers' 2025-09-07T06:43:13.5013234Z Entering 'third_party/fmt' 2025-09-07T06:43:13.5073734Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T06:43:13.5126181Z Entering 'third_party/gloo' 2025-09-07T06:43:13.5179191Z Entering 'third_party/googletest' 2025-09-07T06:43:13.5234485Z Entering 'third_party/ideep' 2025-09-07T06:43:13.5289655Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T06:43:13.5350562Z Entering 'third_party/ittapi' 2025-09-07T06:43:13.5403701Z Entering 'third_party/kineto' 2025-09-07T06:43:13.5459234Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T06:43:13.5509307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T06:43:13.5571339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T06:43:13.5624617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T06:43:13.5682281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T06:43:13.5734523Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T06:43:13.5792660Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T06:43:13.5841530Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T06:43:13.5899109Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T06:43:13.5956705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T06:43:13.6013762Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T06:43:13.6067141Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T06:43:13.6117819Z Entering 'third_party/kleidiai' 2025-09-07T06:43:13.6174833Z Entering 'third_party/mimalloc' 2025-09-07T06:43:13.6234991Z Entering 'third_party/nlohmann' 2025-09-07T06:43:13.6293876Z Entering 'third_party/onnx' 2025-09-07T06:43:13.6362129Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T06:43:13.6417196Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T06:43:13.6478563Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T06:43:13.6540973Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T06:43:13.6598674Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T06:43:13.6654189Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T06:43:13.6712383Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T06:43:13.6771002Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T06:43:13.6820543Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T06:43:13.6870278Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T06:43:13.6922928Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T06:43:13.6984356Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T06:43:13.7048143Z Entering 'third_party/pocketfft' 2025-09-07T06:43:13.7105616Z Entering 'third_party/protobuf' 2025-09-07T06:43:13.7162002Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T06:43:13.7214189Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T06:43:13.7281166Z Entering 'third_party/psimd' 2025-09-07T06:43:13.7335915Z Entering 'third_party/pthreadpool' 2025-09-07T06:43:13.7392908Z Entering 'third_party/pybind11' 2025-09-07T06:43:13.7447771Z Entering 'third_party/python-peachpy' 2025-09-07T06:43:13.7500134Z Entering 'third_party/sleef' 2025-09-07T06:43:13.7558956Z Entering 'third_party/tensorpipe' 2025-09-07T06:43:13.7611707Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T06:43:13.7673095Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T06:43:13.7727951Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T06:43:13.7786329Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T06:43:13.7845832Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T06:43:13.7931113Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-09-07T06:43:13.8262530Z Entering 'android/libs/fbjni' 2025-09-07T06:43:13.8307629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-09-07T06:43:13.8331520Z Entering 'third_party/FP16' 2025-09-07T06:43:13.8381593Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-09-07T06:43:13.8398550Z Entering 'third_party/FXdiv' 2025-09-07T06:43:13.8445533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-09-07T06:43:13.8463810Z Entering 'third_party/NNPACK' 2025-09-07T06:43:13.8513947Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-09-07T06:43:13.8534071Z Entering 'third_party/NVTX' 2025-09-07T06:43:13.8581527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-09-07T06:43:13.8599127Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T06:43:13.8657396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-09-07T06:43:13.8677844Z Entering 'third_party/XNNPACK' 2025-09-07T06:43:13.8729748Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-09-07T06:43:13.8750393Z Entering 'third_party/aiter' 2025-09-07T06:43:13.8798934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-09-07T06:43:13.8816404Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T06:43:13.8866601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-09-07T06:43:13.8894550Z Entering 'third_party/benchmark' 2025-09-07T06:43:13.8942039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-09-07T06:43:13.8958064Z Entering 'third_party/composable_kernel' 2025-09-07T06:43:13.9009808Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-09-07T06:43:13.9030364Z Entering 'third_party/cpp-httplib' 2025-09-07T06:43:13.9082057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-09-07T06:43:13.9095533Z Entering 'third_party/cpuinfo' 2025-09-07T06:43:13.9145904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-09-07T06:43:13.9170016Z Entering 'third_party/cudnn_frontend' 2025-09-07T06:43:13.9218144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-09-07T06:43:13.9240081Z Entering 'third_party/cutlass' 2025-09-07T06:43:13.9286963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-09-07T06:43:13.9312671Z Entering 'third_party/fbgemm' 2025-09-07T06:43:13.9366458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-09-07T06:43:13.9383320Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T06:43:13.9436358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-09-07T06:43:13.9455022Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T06:43:13.9503780Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-09-07T06:43:13.9524385Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T06:43:13.9576380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-09-07T06:43:13.9602771Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T06:43:13.9643964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-09-07T06:43:13.9660138Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T06:43:13.9708682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-09-07T06:43:13.9738876Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T06:43:13.9784771Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-09-07T06:43:13.9796844Z Entering 'third_party/fbgemm/external/json' 2025-09-07T06:43:13.9848455Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-09-07T06:43:13.9869674Z Entering 'third_party/flash-attention' 2025-09-07T06:43:13.9918408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-09-07T06:43:13.9937331Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T06:43:13.9983226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-09-07T06:43:14.0001366Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T06:43:14.0054079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-09-07T06:43:14.0081088Z Entering 'third_party/flatbuffers' 2025-09-07T06:43:14.0132458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-09-07T06:43:14.0146736Z Entering 'third_party/fmt' 2025-09-07T06:43:14.0196954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-09-07T06:43:14.0209490Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T06:43:14.0260315Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-09-07T06:43:14.0276012Z Entering 'third_party/gloo' 2025-09-07T06:43:14.0321982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-09-07T06:43:14.0340984Z Entering 'third_party/googletest' 2025-09-07T06:43:14.0389724Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-09-07T06:43:14.0406381Z Entering 'third_party/ideep' 2025-09-07T06:43:14.0462033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-09-07T06:43:14.0477528Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T06:43:14.0525886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-09-07T06:43:14.0549155Z Entering 'third_party/ittapi' 2025-09-07T06:43:14.0595156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-09-07T06:43:14.0614348Z Entering 'third_party/kineto' 2025-09-07T06:43:14.0670432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-09-07T06:43:14.0685170Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T06:43:14.0735111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-09-07T06:43:14.0757839Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T06:43:14.0804069Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-09-07T06:43:14.0822181Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T06:43:14.0867934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-09-07T06:43:14.0881425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T06:43:14.0938220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-09-07T06:43:14.0959294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T06:43:14.1007217Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-09-07T06:43:14.1021941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T06:43:14.1069867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-09-07T06:43:14.1089959Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T06:43:14.1139674Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-09-07T06:43:14.1161245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T06:43:14.1210198Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-09-07T06:43:14.1223149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T06:43:14.1272375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-09-07T06:43:14.1289524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T06:43:14.1347020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-09-07T06:43:14.1370867Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T06:43:14.1417088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-09-07T06:43:14.1438348Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T06:43:14.1487863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-09-07T06:43:14.1502785Z Entering 'third_party/kleidiai' 2025-09-07T06:43:14.1556035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-09-07T06:43:14.1577712Z Entering 'third_party/mimalloc' 2025-09-07T06:43:14.1621829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-09-07T06:43:14.1646783Z Entering 'third_party/nlohmann' 2025-09-07T06:43:14.1693110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-09-07T06:43:14.1709498Z Entering 'third_party/onnx' 2025-09-07T06:43:14.1760463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-09-07T06:43:14.1788461Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T06:43:14.1844069Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-09-07T06:43:14.1859528Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T06:43:14.1913306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-09-07T06:43:14.1931820Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T06:43:14.1979811Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-09-07T06:43:14.1996085Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T06:43:14.2047767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-09-07T06:43:14.2064559Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T06:43:14.2113432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-09-07T06:43:14.2136124Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T06:43:14.2188738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-09-07T06:43:14.2205971Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T06:43:14.2260181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-09-07T06:43:14.2277638Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T06:43:14.2325069Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-09-07T06:43:14.2343085Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T06:43:14.2391790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-09-07T06:43:14.2407593Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T06:43:14.2456923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-09-07T06:43:14.2474007Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T06:43:14.2527891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-09-07T06:43:14.2554236Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T06:43:14.2597518Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-09-07T06:43:14.2626444Z Entering 'third_party/pocketfft' 2025-09-07T06:43:14.2679136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-09-07T06:43:14.2700108Z Entering 'third_party/protobuf' 2025-09-07T06:43:14.2744493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-09-07T06:43:14.2764314Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T06:43:14.2807334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-09-07T06:43:14.2825284Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T06:43:14.2879398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-09-07T06:43:14.2898374Z Entering 'third_party/psimd' 2025-09-07T06:43:14.2947108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-09-07T06:43:14.2962761Z Entering 'third_party/pthreadpool' 2025-09-07T06:43:14.3009522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-09-07T06:43:14.3027871Z Entering 'third_party/pybind11' 2025-09-07T06:43:14.3077573Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-09-07T06:43:14.3090612Z Entering 'third_party/python-peachpy' 2025-09-07T06:43:14.3141374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-09-07T06:43:14.3162847Z Entering 'third_party/sleef' 2025-09-07T06:43:14.3205458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-09-07T06:43:14.3219379Z Entering 'third_party/tensorpipe' 2025-09-07T06:43:14.3274213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-09-07T06:43:14.3286612Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T06:43:14.3337219Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-09-07T06:43:14.3357525Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T06:43:14.3403473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-09-07T06:43:14.3420164Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T06:43:14.3472214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-09-07T06:43:14.3488602Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T06:43:14.3539426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-09-07T06:43:14.3554982Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T06:43:14.3599951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-09-07T06:43:14.4822149Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-09-07T06:43:14.5137610Z Entering 'android/libs/fbjni' 2025-09-07T06:43:14.5174631Z Entering 'third_party/FP16' 2025-09-07T06:43:14.5215603Z Entering 'third_party/FXdiv' 2025-09-07T06:43:14.5255871Z Entering 'third_party/NNPACK' 2025-09-07T06:43:14.5298889Z Entering 'third_party/NVTX' 2025-09-07T06:43:14.5339000Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T06:43:14.5381663Z Entering 'third_party/XNNPACK' 2025-09-07T06:43:14.5431625Z Entering 'third_party/aiter' 2025-09-07T06:43:14.5477082Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T06:43:14.5529392Z Entering 'third_party/benchmark' 2025-09-07T06:43:14.5575222Z Entering 'third_party/composable_kernel' 2025-09-07T06:43:14.5623418Z Entering 'third_party/cpp-httplib' 2025-09-07T06:43:14.5666284Z Entering 'third_party/cpuinfo' 2025-09-07T06:43:14.5705292Z Entering 'third_party/cudnn_frontend' 2025-09-07T06:43:14.5748372Z Entering 'third_party/cutlass' 2025-09-07T06:43:14.5796544Z Entering 'third_party/fbgemm' 2025-09-07T06:43:14.5838839Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T06:43:14.5876521Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T06:43:14.5920527Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T06:43:14.5962108Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T06:43:14.6009063Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T06:43:14.6052249Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T06:43:14.6092774Z Entering 'third_party/fbgemm/external/json' 2025-09-07T06:43:14.6136949Z Entering 'third_party/flash-attention' 2025-09-07T06:43:14.6179999Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T06:43:14.6219113Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T06:43:14.6266791Z Entering 'third_party/flatbuffers' 2025-09-07T06:43:14.6310237Z Entering 'third_party/fmt' 2025-09-07T06:43:14.6348414Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T06:43:14.6386300Z Entering 'third_party/gloo' 2025-09-07T06:43:14.6426338Z Entering 'third_party/googletest' 2025-09-07T06:43:14.6471901Z Entering 'third_party/ideep' 2025-09-07T06:43:14.6505446Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T06:43:14.6559807Z Entering 'third_party/ittapi' 2025-09-07T06:43:14.6601093Z Entering 'third_party/kineto' 2025-09-07T06:43:14.6641407Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T06:43:14.6678693Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T06:43:14.6721140Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T06:43:14.6762048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T06:43:14.6800465Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T06:43:14.6837020Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T06:43:14.6885071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T06:43:14.6928136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T06:43:14.6966402Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T06:43:14.7010964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T06:43:14.7052608Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T06:43:14.7097349Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T06:43:14.7146717Z Entering 'third_party/kleidiai' 2025-09-07T06:43:14.7187794Z Entering 'third_party/mimalloc' 2025-09-07T06:43:14.7233392Z Entering 'third_party/nlohmann' 2025-09-07T06:43:14.7267152Z Entering 'third_party/onnx' 2025-09-07T06:43:14.7318026Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T06:43:14.7361966Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T06:43:14.7405004Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T06:43:14.7447317Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T06:43:14.7497918Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T06:43:14.7530587Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T06:43:14.7573043Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T06:43:14.7610721Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T06:43:14.7655682Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T06:43:14.7694560Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T06:43:14.7736285Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T06:43:14.7778300Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T06:43:14.7831917Z Entering 'third_party/pocketfft' 2025-09-07T06:43:14.7874696Z Entering 'third_party/protobuf' 2025-09-07T06:43:14.7924098Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T06:43:14.7956090Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T06:43:14.7998060Z Entering 'third_party/psimd' 2025-09-07T06:43:14.8038831Z Entering 'third_party/pthreadpool' 2025-09-07T06:43:14.8077314Z Entering 'third_party/pybind11' 2025-09-07T06:43:14.8118195Z Entering 'third_party/python-peachpy' 2025-09-07T06:43:14.8160789Z Entering 'third_party/sleef' 2025-09-07T06:43:14.8204879Z Entering 'third_party/tensorpipe' 2025-09-07T06:43:14.8247671Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T06:43:14.8284809Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T06:43:14.8329353Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T06:43:14.8371384Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T06:43:14.8406470Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T06:43:14.8471519Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-09-07T06:43:14.8797213Z Entering 'android/libs/fbjni' 2025-09-07T06:43:14.8837036Z Entering 'third_party/FP16' 2025-09-07T06:43:14.8878824Z Entering 'third_party/FXdiv' 2025-09-07T06:43:14.8916308Z Entering 'third_party/NNPACK' 2025-09-07T06:43:14.8966866Z Entering 'third_party/NVTX' 2025-09-07T06:43:14.9004466Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T06:43:14.9049390Z Entering 'third_party/XNNPACK' 2025-09-07T06:43:14.9095647Z Entering 'third_party/aiter' 2025-09-07T06:43:14.9139135Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T06:43:14.9183921Z Entering 'third_party/benchmark' 2025-09-07T06:43:14.9224924Z Entering 'third_party/composable_kernel' 2025-09-07T06:43:14.9273492Z Entering 'third_party/cpp-httplib' 2025-09-07T06:43:14.9316833Z Entering 'third_party/cpuinfo' 2025-09-07T06:43:14.9362672Z Entering 'third_party/cudnn_frontend' 2025-09-07T06:43:14.9400646Z Entering 'third_party/cutlass' 2025-09-07T06:43:14.9452231Z Entering 'third_party/fbgemm' 2025-09-07T06:43:14.9492971Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T06:43:14.9536165Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T06:43:14.9579987Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T06:43:14.9618116Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T06:43:14.9664472Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T06:43:14.9705061Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T06:43:14.9750534Z Entering 'third_party/fbgemm/external/json' 2025-09-07T06:43:14.9798546Z Entering 'third_party/flash-attention' 2025-09-07T06:43:14.9841746Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T06:43:14.9887835Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T06:43:14.9940690Z Entering 'third_party/flatbuffers' 2025-09-07T06:43:14.9983857Z Entering 'third_party/fmt' 2025-09-07T06:43:15.0024666Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T06:43:15.0068275Z Entering 'third_party/gloo' 2025-09-07T06:43:15.0111125Z Entering 'third_party/googletest' 2025-09-07T06:43:15.0156876Z Entering 'third_party/ideep' 2025-09-07T06:43:15.0192900Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T06:43:15.0239886Z Entering 'third_party/ittapi' 2025-09-07T06:43:15.0280991Z Entering 'third_party/kineto' 2025-09-07T06:43:15.0323836Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T06:43:15.0365939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T06:43:15.0411069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T06:43:15.0460944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T06:43:15.0500326Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T06:43:15.0539743Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T06:43:15.0585183Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T06:43:15.0626516Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T06:43:15.0674967Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T06:43:15.0710813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T06:43:15.0755868Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T06:43:15.0795808Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T06:43:15.0840662Z Entering 'third_party/kleidiai' 2025-09-07T06:43:15.0887157Z Entering 'third_party/mimalloc' 2025-09-07T06:43:15.0933443Z Entering 'third_party/nlohmann' 2025-09-07T06:43:15.0977719Z Entering 'third_party/onnx' 2025-09-07T06:43:15.1024480Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T06:43:15.1066724Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T06:43:15.1109387Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T06:43:15.1158898Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T06:43:15.1202665Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T06:43:15.1244265Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T06:43:15.1284283Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T06:43:15.1319053Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T06:43:15.1364220Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T06:43:15.1399326Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T06:43:15.1442297Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T06:43:15.1485707Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T06:43:15.1538933Z Entering 'third_party/pocketfft' 2025-09-07T06:43:15.1580628Z Entering 'third_party/protobuf' 2025-09-07T06:43:15.1624709Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T06:43:15.1666852Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T06:43:15.1711001Z Entering 'third_party/psimd' 2025-09-07T06:43:15.1757292Z Entering 'third_party/pthreadpool' 2025-09-07T06:43:15.1795846Z Entering 'third_party/pybind11' 2025-09-07T06:43:15.1839219Z Entering 'third_party/python-peachpy' 2025-09-07T06:43:15.1884471Z Entering 'third_party/sleef' 2025-09-07T06:43:15.1929082Z Entering 'third_party/tensorpipe' 2025-09-07T06:43:15.1968143Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T06:43:15.2005473Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T06:43:15.2047083Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T06:43:15.2089510Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T06:43:15.2128313Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T06:43:15.2176398Z ##[endgroup] 2025-09-07T06:43:15.2210126Z [command]/usr/bin/git log -1 --format=%H 2025-09-07T06:43:15.2239164Z 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:43:15.2336983Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-09-07T06:43:15.2337245Z cd "${GITHUB_WORKSPACE}" 2025-09-07T06:43:15.2337460Z # Clean stale submodule dirs 2025-09-07T06:43:15.2337676Z if [ -z "${NO_SUDO}" ]; then 2025-09-07T06:43:15.2337941Z  sudo git submodule foreach --recursive git clean -ffdx 2025-09-07T06:43:15.2338189Z else 2025-09-07T06:43:15.2338400Z  git submodule foreach --recursive git clean -ffdx 2025-09-07T06:43:15.2338630Z fi 2025-09-07T06:43:15.2346699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:15.2346965Z env: 2025-09-07T06:43:15.2347149Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:15.2347335Z NO_SUDO: true 2025-09-07T06:43:15.2347506Z ##[endgroup] 2025-09-07T06:43:15.2680300Z Entering 'android/libs/fbjni' 2025-09-07T06:43:15.2714489Z Entering 'third_party/FP16' 2025-09-07T06:43:15.2746746Z Entering 'third_party/FXdiv' 2025-09-07T06:43:15.2776539Z Entering 'third_party/NNPACK' 2025-09-07T06:43:15.2810406Z Entering 'third_party/NVTX' 2025-09-07T06:43:15.2845545Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T06:43:15.2879926Z Entering 'third_party/XNNPACK' 2025-09-07T06:43:15.2980812Z Entering 'third_party/aiter' 2025-09-07T06:43:15.3014968Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T06:43:15.3105324Z Entering 'third_party/benchmark' 2025-09-07T06:43:15.3135254Z Entering 'third_party/composable_kernel' 2025-09-07T06:43:15.3226986Z Entering 'third_party/cpp-httplib' 2025-09-07T06:43:15.3260341Z Entering 'third_party/cpuinfo' 2025-09-07T06:43:15.3294141Z Entering 'third_party/cudnn_frontend' 2025-09-07T06:43:15.3328625Z Entering 'third_party/cutlass' 2025-09-07T06:43:15.3404548Z Entering 'third_party/fbgemm' 2025-09-07T06:43:15.3457952Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T06:43:15.3485976Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T06:43:15.3574294Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T06:43:15.3609687Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T06:43:15.3688926Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T06:43:15.3718714Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T06:43:15.3758457Z Entering 'third_party/fbgemm/external/json' 2025-09-07T06:43:15.3801212Z Entering 'third_party/flash-attention' 2025-09-07T06:43:15.3839389Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T06:43:15.3912211Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T06:43:15.3992040Z Entering 'third_party/flatbuffers' 2025-09-07T06:43:15.4053141Z Entering 'third_party/fmt' 2025-09-07T06:43:15.4088337Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T06:43:15.4115794Z Entering 'third_party/gloo' 2025-09-07T06:43:15.4145222Z Entering 'third_party/googletest' 2025-09-07T06:43:15.4181969Z Entering 'third_party/ideep' 2025-09-07T06:43:15.4210750Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T06:43:15.4285319Z Entering 'third_party/ittapi' 2025-09-07T06:43:15.4318813Z Entering 'third_party/kineto' 2025-09-07T06:43:15.4351326Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T06:43:15.4385222Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T06:43:15.4424196Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T06:43:15.4456615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T06:43:15.4486255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T06:43:15.4514595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T06:43:15.4552459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T06:43:15.4581867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T06:43:15.4612298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T06:43:15.4653405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T06:43:15.4690281Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T06:43:15.4719090Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T06:43:15.4751324Z Entering 'third_party/kleidiai' 2025-09-07T06:43:15.4783327Z Entering 'third_party/mimalloc' 2025-09-07T06:43:15.4817122Z Entering 'third_party/nlohmann' 2025-09-07T06:43:15.4862391Z Entering 'third_party/onnx' 2025-09-07T06:43:15.5090231Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T06:43:15.5127473Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T06:43:15.5172960Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T06:43:15.5204551Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T06:43:15.5236958Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T06:43:15.5266008Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T06:43:15.5306303Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T06:43:15.5331264Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T06:43:15.5370576Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T06:43:15.5399908Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T06:43:15.5443117Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T06:43:15.5480146Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T06:43:15.5661761Z Entering 'third_party/pocketfft' 2025-09-07T06:43:15.5694184Z Entering 'third_party/protobuf' 2025-09-07T06:43:15.5760705Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T06:43:15.5795083Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T06:43:15.5831286Z Entering 'third_party/psimd' 2025-09-07T06:43:15.5862870Z Entering 'third_party/pthreadpool' 2025-09-07T06:43:15.5892168Z Entering 'third_party/pybind11' 2025-09-07T06:43:15.5927553Z Entering 'third_party/python-peachpy' 2025-09-07T06:43:15.5963501Z Entering 'third_party/sleef' 2025-09-07T06:43:15.5995671Z Entering 'third_party/tensorpipe' 2025-09-07T06:43:15.6030942Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T06:43:15.6064278Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T06:43:15.6095187Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T06:43:15.6128015Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T06:43:15.6160398Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T06:43:15.6292160Z Prepare all required actions 2025-09-07T06:43:15.6292589Z Getting action download info 2025-09-07T06:43:15.7643662Z ##[group]Run ./.github/actions/setup-linux 2025-09-07T06:43:15.7643891Z env: 2025-09-07T06:43:15.7644050Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:15.7644218Z ##[endgroup] 2025-09-07T06:43:15.7682033Z ##[group]Run set -euo pipefail 2025-09-07T06:43:15.7682310Z set -euo pipefail 2025-09-07T06:43:15.7682511Z function get_ec2_metadata() { 2025-09-07T06:43:15.7682753Z  # Pulled from instance metadata endpoint for EC2 2025-09-07T06:43:15.7683152Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-09-07T06:43:15.7683495Z  category=$1 2025-09-07T06:43:15.7683736Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-09-07T06:43:15.7684008Z  runner_name_str=i-085acfb4aecab35f4 2025-09-07T06:43:15.7684287Z  if [[ -f /.inarc ]]; then 2025-09-07T06:43:15.7684693Z  echo "ARC Runner, no info on ec2 metadata" 2025-09-07T06:43:15.7684937Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-09-07T06:43:15.7685233Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-09-07T06:43:15.7685495Z  else 2025-09-07T06:43:15.7686029Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-09-07T06:43:15.7686562Z  fi 2025-09-07T06:43:15.7686710Z } 2025-09-07T06:43:15.7686894Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-09-07T06:43:15.7687165Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-09-07T06:43:15.7687458Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-09-07T06:43:15.7687709Z echo "system info $(uname -a)" 2025-09-07T06:43:15.7694263Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:15.7694503Z env: 2025-09-07T06:43:15.7694658Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:15.7694826Z ##[endgroup] 2025-09-07T06:43:15.7826161Z ami-id: ami-05ffe3c48a9991133 2025-09-07T06:43:15.7923074Z instance-id: i-085acfb4aecab35f4 2025-09-07T06:43:15.8020744Z instance-type: m7i-flex.8xlarge 2025-09-07T06:43:15.8035923Z system info Linux ip-10-0-10-208.ec2.internal 6.1.141-155.222.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 17 10:29:47 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-09-07T06:43:15.8065968Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-09-07T06:43:15.8066616Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-09-07T06:43:15.8072137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:15.8072436Z env: 2025-09-07T06:43:15.8072601Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:15.8072795Z ##[endgroup] 2025-09-07T06:43:15.8123461Z ##[group]Run if systemctl is-active --quiet docker; then 2025-09-07T06:43:15.8123768Z if systemctl is-active --quiet docker; then 2025-09-07T06:43:15.8124024Z  echo "Docker daemon is running..."; 2025-09-07T06:43:15.8124239Z else 2025-09-07T06:43:15.8124477Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-09-07T06:43:15.8124732Z fi 2025-09-07T06:43:15.8128535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:15.8128775Z env: 2025-09-07T06:43:15.8128933Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:15.8129108Z ##[endgroup] 2025-09-07T06:43:15.8241300Z Docker daemon is running... 2025-09-07T06:43:15.8287034Z ##[group]Run nick-fields/retry@v3.0.0 2025-09-07T06:43:15.8287272Z with: 2025-09-07T06:43:15.8287428Z shell: bash 2025-09-07T06:43:15.8287752Z timeout_minutes: 5 2025-09-07T06:43:15.8288000Z max_attempts: 3 2025-09-07T06:43:15.8288189Z retry_wait_seconds: 30 2025-09-07T06:43:15.8289693Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-09-07T06:43:15.8291267Z polling_interval_seconds: 1 2025-09-07T06:43:15.8291487Z warning_on_retry: true 2025-09-07T06:43:15.8291689Z continue_on_error: false 2025-09-07T06:43:15.8291877Z env: 2025-09-07T06:43:15.8292141Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:15.8292339Z AWS_RETRY_MODE: standard 2025-09-07T06:43:15.8292532Z AWS_MAX_ATTEMPTS: 5 2025-09-07T06:43:15.8292722Z AWS_DEFAULT_REGION: us-east-1 2025-09-07T06:43:15.8292926Z ##[endgroup] 2025-09-07T06:43:16.8783422Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-09-07T06:43:16.8783883Z Configure a credential helper to remove this warning. See 2025-09-07T06:43:16.8784387Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-09-07T06:43:16.8784653Z 2025-09-07T06:43:16.8784741Z Login Succeeded 2025-09-07T06:43:16.9981810Z Command completed after 1 attempt(s). 2025-09-07T06:43:17.0037542Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-09-07T06:43:17.0037935Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-09-07T06:43:17.0038253Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-09-07T06:43:17.0044626Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:17.0044913Z env: 2025-09-07T06:43:17.0045091Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:17.0045290Z ##[endgroup] 2025-09-07T06:43:17.0141888Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-09-07T06:43:17.0142296Z # ignore expansion of "docker ps -q" since it could be empty 2025-09-07T06:43:17.0142593Z # shellcheck disable=SC2046 2025-09-07T06:43:17.0142837Z docker stop $(docker ps -q) || true 2025-09-07T06:43:17.0143078Z # Prune all of the docker images 2025-09-07T06:43:17.0143314Z docker system prune -af 2025-09-07T06:43:17.0148502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:17.0148763Z env: 2025-09-07T06:43:17.0148927Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:17.0149125Z ##[endgroup] 2025-09-07T06:43:17.0618862Z "docker stop" requires at least 1 argument. 2025-09-07T06:43:17.0619202Z See 'docker stop --help'. 2025-09-07T06:43:17.0619383Z 2025-09-07T06:43:17.0619522Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-09-07T06:43:17.0619871Z 2025-09-07T06:43:17.0619962Z Stop one or more running containers 2025-09-07T06:43:17.0825305Z Total reclaimed space: 0B 2025-09-07T06:43:17.0868856Z ##[group]Run set +e 2025-09-07T06:43:17.0869070Z set +e 2025-09-07T06:43:17.0869240Z set -x 2025-09-07T06:43:17.0869397Z  2025-09-07T06:43:17.0869572Z PT_DOMAIN=download.pytorch.org 2025-09-07T06:43:17.0869940Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2025-09-07T06:43:17.0870397Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2025-09-07T06:43:17.0870724Z # one is returned at random 2025-09-07T06:43:17.0870990Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2025-09-07T06:43:17.0871233Z  2025-09-07T06:43:17.0871515Z if [ -z "${RESOLVED_IP}" ]; then 2025-09-07T06:43:17.0871808Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2025-09-07T06:43:17.0872136Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2025-09-07T06:43:17.0872384Z  2025-09-07T06:43:17.0872555Z  if [ -z "${RESOLVED_IP}" ]; then 2025-09-07T06:43:17.0872826Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2025-09-07T06:43:17.0873078Z  exit 1 2025-09-07T06:43:17.0873240Z  fi 2025-09-07T06:43:17.0873384Z fi 2025-09-07T06:43:17.0873534Z  2025-09-07T06:43:17.0873716Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2025-09-07T06:43:17.0873957Z  # Clean up any old records first 2025-09-07T06:43:17.0874184Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2025-09-07T06:43:17.0874395Z fi 2025-09-07T06:43:17.0874538Z  2025-09-07T06:43:17.0874749Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2025-09-07T06:43:17.0875111Z cat /etc/hosts 2025-09-07T06:43:17.0880013Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:17.0880253Z env: 2025-09-07T06:43:17.0880414Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:17.0880590Z ##[endgroup] 2025-09-07T06:43:17.0903814Z + PT_DOMAIN=download.pytorch.org 2025-09-07T06:43:17.0912085Z ++ dig -4 +short download.pytorch.org 2025-09-07T06:43:17.0912362Z ++ tail -n1 2025-09-07T06:43:17.1617501Z + RESOLVED_IP=18.160.10.28 2025-09-07T06:43:17.1621470Z + '[' -z 18.160.10.28 ']' 2025-09-07T06:43:17.1621775Z + grep -r download.pytorch.org /etc/hosts 2025-09-07T06:43:17.1638256Z + sudo tee -a /etc/hosts 2025-09-07T06:43:17.1643744Z + echo '18.160.10.28 download.pytorch.org' 2025-09-07T06:43:17.4220868Z 18.160.10.28 download.pytorch.org 2025-09-07T06:43:17.4243363Z + cat /etc/hosts 2025-09-07T06:43:17.4251905Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2025-09-07T06:43:17.4263621Z ::1 localhost6 localhost6.localdomain6 2025-09-07T06:43:17.4263935Z 18.160.10.28 download.pytorch.org 2025-09-07T06:43:17.4383267Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-09-07T06:43:17.4383589Z with: 2025-09-07T06:43:17.4384153Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4384759Z use-custom-docker-registry: true 2025-09-07T06:43:17.4384991Z docker-build-dir: .ci/docker 2025-09-07T06:43:17.4385209Z docker-build-script: ./build.sh 2025-09-07T06:43:17.4385424Z working-directory: . 2025-09-07T06:43:17.4385897Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:17.4386192Z force-push: false 2025-09-07T06:43:17.4386374Z env: 2025-09-07T06:43:17.4386550Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:17.4386745Z ##[endgroup] 2025-09-07T06:43:17.4405480Z ##[group]Run set -ex 2025-09-07T06:43:17.4405738Z set -ex 2025-09-07T06:43:17.4405905Z  2025-09-07T06:43:17.4406226Z # If the docker build directory or the build script doesn't exist, the action will 2025-09-07T06:43:17.4406665Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-09-07T06:43:17.4407012Z # job could then download the pre-built image as usual 2025-09-07T06:43:17.4407440Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-09-07T06:43:17.4407846Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4408061Z else 2025-09-07T06:43:17.4408247Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4408527Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4408784Z  2025-09-07T06:43:17.4409137Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-09-07T06:43:17.4409526Z  exit 0 2025-09-07T06:43:17.4409678Z fi 2025-09-07T06:43:17.4409827Z  2025-09-07T06:43:17.4410054Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-09-07T06:43:17.4410445Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-09-07T06:43:17.4410781Z  # use it as it is, but first let's extract the tag 2025-09-07T06:43:17.4411082Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-09-07T06:43:17.4411404Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4411712Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4411968Z else 2025-09-07T06:43:17.4412157Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-09-07T06:43:17.4412507Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-09-07T06:43:17.4412764Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-09-07T06:43:17.4412981Z  fi 2025-09-07T06:43:17.4413274Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-09-07T06:43:17.4413644Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4414036Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4414462Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4414731Z fi 2025-09-07T06:43:17.4421572Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:17.4421834Z env: 2025-09-07T06:43:17.4422009Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:17.4422206Z REPO_NAME: pytorch 2025-09-07T06:43:17.4422898Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4423502Z DOCKER_BUILD_DIR: .ci/docker 2025-09-07T06:43:17.4423707Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-09-07T06:43:17.4423981Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:17.4424269Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-09-07T06:43:17.4424486Z CUSTOM_TAG_PREFIX: 2025-09-07T06:43:17.4424667Z ##[endgroup] 2025-09-07T06:43:17.4450798Z + [[ -d .ci/docker ]] 2025-09-07T06:43:17.4454701Z + [[ -f .ci/docker/./build.sh ]] 2025-09-07T06:43:17.4455070Z + [[ true == \t\r\u\e ]] 2025-09-07T06:43:17.4455709Z + echo skip=false 2025-09-07T06:43:17.4456565Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-09-07T06:43:17.4457708Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4458326Z ++ awk -F '[:,]' '{print $2}' 2025-09-07T06:43:17.4484107Z + DOCKER_TAG=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4488720Z + echo docker-tag=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4491050Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4513395Z ##[group]Run set +e 2025-09-07T06:43:17.4513644Z set +e 2025-09-07T06:43:17.4513812Z set -x 2025-09-07T06:43:17.4513965Z  2025-09-07T06:43:17.4514107Z login() { 2025-09-07T06:43:17.4514430Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-09-07T06:43:17.4514766Z } 2025-09-07T06:43:17.4514916Z  2025-09-07T06:43:17.4515057Z retry () { 2025-09-07T06:43:17.4515246Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-09-07T06:43:17.4515454Z } 2025-09-07T06:43:17.4515600Z  2025-09-07T06:43:17.4515756Z retry login "${DOCKER_REGISTRY}" 2025-09-07T06:43:17.4515956Z  2025-09-07T06:43:17.4516111Z START_TIME=$(date +%s) 2025-09-07T06:43:17.4516317Z # Wait up to 120 minutes 2025-09-07T06:43:17.4516553Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-09-07T06:43:17.4516868Z  # Check if image already exists, if it does then skip building it 2025-09-07T06:43:17.4517181Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-09-07T06:43:17.4517416Z  exit 0 2025-09-07T06:43:17.4518665Z  fi 2025-09-07T06:43:17.4518811Z  2025-09-07T06:43:17.4519065Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-09-07T06:43:17.4519466Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-09-07T06:43:17.4520101Z  # latter, it will wait for the Docker images to become available before continuing 2025-09-07T06:43:17.4520434Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-09-07T06:43:17.4520691Z  # It's a Docker build job, let's build the image 2025-09-07T06:43:17.4520918Z  break 2025-09-07T06:43:17.4521123Z  else 2025-09-07T06:43:17.4521355Z  # It's a regular build job, wait for the image to become available 2025-09-07T06:43:17.4521608Z  sleep 300 2025-09-07T06:43:17.4521778Z  fi 2025-09-07T06:43:17.4521931Z done 2025-09-07T06:43:17.4522082Z  2025-09-07T06:43:17.4522309Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-09-07T06:43:17.4522744Z # be empty. The default action would be to continue rebuild the image 2025-09-07T06:43:17.4523073Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-09-07T06:43:17.4523366Z  # if we're on the base branch then use the parent commit 2025-09-07T06:43:17.4523628Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-09-07T06:43:17.4523831Z else 2025-09-07T06:43:17.4524054Z  # otherwise we're on a PR, so use the most recent base commit 2025-09-07T06:43:17.4524362Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-09-07T06:43:17.4524596Z fi 2025-09-07T06:43:17.4524744Z  2025-09-07T06:43:17.4524914Z if [[ -z "${MERGE_BASE}" ]]; then 2025-09-07T06:43:17.4525161Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4525378Z  2025-09-07T06:43:17.4525673Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-09-07T06:43:17.4526010Z  exit 0 2025-09-07T06:43:17.4526177Z fi 2025-09-07T06:43:17.4526319Z  2025-09-07T06:43:17.4526513Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-09-07T06:43:17.4526923Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-09-07T06:43:17.4527277Z  exit 1 2025-09-07T06:43:17.4527434Z fi 2025-09-07T06:43:17.4527578Z  2025-09-07T06:43:17.4527810Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-09-07T06:43:17.4528210Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-09-07T06:43:17.4528570Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-09-07T06:43:17.4528988Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-09-07T06:43:17.4529432Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-09-07T06:43:17.4529696Z fi 2025-09-07T06:43:17.4529835Z  2025-09-07T06:43:17.4530012Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-09-07T06:43:17.4534546Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:17.4534778Z env: 2025-09-07T06:43:17.4534937Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:17.4535122Z DOCKER_BUILD_DIR: .ci/docker 2025-09-07T06:43:17.4535351Z BASE_REVISION: 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:43:17.4535932Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4536649Z DOCKER_TAG: pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:17.4537164Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:17.4537409Z DOCKER_PUSH: 2025-09-07T06:43:17.4537571Z ##[endgroup] 2025-09-07T06:43:17.4559836Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:17.4560158Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:17.4560443Z + aws ecr get-login-password --region us-east-1 2025-09-07T06:43:17.4565024Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:17.8882440Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-09-07T06:43:17.8882798Z Login Succeeded 2025-09-07T06:43:17.8883042Z Configure a credential helper to remove this warning. See 2025-09-07T06:43:17.8883428Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-09-07T06:43:17.8883662Z 2025-09-07T06:43:17.8901319Z ++ date +%s 2025-09-07T06:43:17.8912923Z + START_TIME=1757227397 2025-09-07T06:43:17.8917300Z ++ date +%s 2025-09-07T06:43:17.8930628Z + [[ 1757220197 -lt 1757227397 ]] 2025-09-07T06:43:17.8931340Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:18.1124876Z { 2025-09-07T06:43:18.1129096Z "schemaVersion": 2, 2025-09-07T06:43:18.1133854Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-09-07T06:43:18.1138854Z "config": { 2025-09-07T06:43:18.1140937Z + exit 0 2025-09-07T06:43:18.1141206Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-09-07T06:43:18.1141570Z "size": 30269, 2025-09-07T06:43:18.1141943Z "digest": "sha256:662d8c9dfc7db2f5d004293de4f2b7647941dee4c916479ef082d17fcdfd9c47" 2025-09-07T06:43:18.1142250Z }, 2025-09-07T06:43:18.1142405Z "layers": [ 2025-09-07T06:43:18.1142562Z { 2025-09-07T06:43:18.1142842Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1143176Z "size": 30448359, 2025-09-07T06:43:18.1143508Z "digest": "sha256:e6fdc8487bfe6d764301ef3634bc6c043841dc3ab05ca14f81e69c0f92562d46" 2025-09-07T06:43:18.1143837Z }, 2025-09-07T06:43:18.1143978Z { 2025-09-07T06:43:18.1144212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1144496Z "size": 1554, 2025-09-07T06:43:18.1144822Z "digest": "sha256:18a5ee5b0e2e283bf6d7b9c4c312b0448c75eff1c43446c22c5139a3aeec97fe" 2025-09-07T06:43:18.1145163Z }, 2025-09-07T06:43:18.1145341Z { 2025-09-07T06:43:18.1145602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1146104Z "size": 313297813, 2025-09-07T06:43:18.1146461Z "digest": "sha256:572424b92528ee46c84fdf3e9e1f5fd75e302621ad75dcf4257ad06778885094" 2025-09-07T06:43:18.1146850Z }, 2025-09-07T06:43:18.1147005Z { 2025-09-07T06:43:18.1147247Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1147558Z "size": 793, 2025-09-07T06:43:18.1147872Z "digest": "sha256:1c35b7d4b67c6769f59f96a643d69c214c5b00291a4968cdd395eedbce82b9c0" 2025-09-07T06:43:18.1148198Z }, 2025-09-07T06:43:18.1148340Z { 2025-09-07T06:43:18.1148575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1148865Z "size": 106, 2025-09-07T06:43:18.1149180Z "digest": "sha256:68c20f3c23bb0bddb9b69e6ce2e45bcd5b1fcfd9b37dbe3de26b8a5f0e81ff13" 2025-09-07T06:43:18.1149519Z }, 2025-09-07T06:43:18.1149651Z { 2025-09-07T06:43:18.1149893Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1150183Z "size": 704, 2025-09-07T06:43:18.1150471Z "digest": "sha256:7efa39950d3273a15b20bc5f6659373b2b4eb62e36328d96b289834c48d2e408" 2025-09-07T06:43:18.1150788Z }, 2025-09-07T06:43:18.1150929Z { 2025-09-07T06:43:18.1151169Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1151756Z "size": 1214, 2025-09-07T06:43:18.1152062Z "digest": "sha256:a10eb16a7271e996ea9f1d769ba6bd2ec69358f2a79cf26649595a8cea38275f" 2025-09-07T06:43:18.1152396Z }, 2025-09-07T06:43:18.1152543Z { 2025-09-07T06:43:18.1152785Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1153074Z "size": 485, 2025-09-07T06:43:18.1153360Z "digest": "sha256:7d52cf57965449440c17f257fe4c522f9685019961eaa9853d7c820cfe39f5cc" 2025-09-07T06:43:18.1153652Z }, 2025-09-07T06:43:18.1153787Z { 2025-09-07T06:43:18.1153991Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1154249Z "size": 110343705, 2025-09-07T06:43:18.1154530Z "digest": "sha256:cb6a20fcf4e24ec2e1f72ecf361b26e058f3e6194947a9b3a25312223d43516e" 2025-09-07T06:43:18.1154824Z }, 2025-09-07T06:43:18.1154945Z { 2025-09-07T06:43:18.1155154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1155411Z "size": 4787, 2025-09-07T06:43:18.1155680Z "digest": "sha256:46fb6a8b3e1d4eac9b3a21577824410003ed38f194b4b1486b747e324b32ef6a" 2025-09-07T06:43:18.1156047Z }, 2025-09-07T06:43:18.1156184Z { 2025-09-07T06:43:18.1156433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1156690Z "size": 1709, 2025-09-07T06:43:18.1156964Z "digest": "sha256:5ad6977cc38e4ea8a6545d6a4fc0e2fdde705a7af96eb496cfe20f264fbc1e74" 2025-09-07T06:43:18.1157262Z }, 2025-09-07T06:43:18.1157397Z { 2025-09-07T06:43:18.1157603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1157872Z "size": 724, 2025-09-07T06:43:18.1158132Z "digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031" 2025-09-07T06:43:18.1158419Z }, 2025-09-07T06:43:18.1158545Z { 2025-09-07T06:43:18.1158754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1159011Z "size": 543, 2025-09-07T06:43:18.1159279Z "digest": "sha256:78243fdb9906cb588921ddaa67a3ca915aa9447ca675faac1a9ebc420a561d83" 2025-09-07T06:43:18.1159568Z }, 2025-09-07T06:43:18.1159703Z { 2025-09-07T06:43:18.1159912Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1160174Z "size": 3395447162, 2025-09-07T06:43:18.1160453Z "digest": "sha256:6f70d5d50abaab8988f460b5590d92b6d1d340575ddee981662c24034d7d20af" 2025-09-07T06:43:18.1160744Z }, 2025-09-07T06:43:18.1160876Z { 2025-09-07T06:43:18.1161086Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1161336Z "size": 32, 2025-09-07T06:43:18.1161604Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1161897Z }, 2025-09-07T06:43:18.1162029Z { 2025-09-07T06:43:18.1162231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1162487Z "size": 380, 2025-09-07T06:43:18.1162750Z "digest": "sha256:69715d3ad3c493436abde51f5a575e79f7d55b46c653f5607f3c7722ad9a05db" 2025-09-07T06:43:18.1163040Z }, 2025-09-07T06:43:18.1163164Z { 2025-09-07T06:43:18.1163377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1163635Z "size": 235844, 2025-09-07T06:43:18.1163907Z "digest": "sha256:7ace90c063f3f3ce8f04b541afe935088868930e5c074824af2b2c327779a3b5" 2025-09-07T06:43:18.1164188Z }, 2025-09-07T06:43:18.1164321Z { 2025-09-07T06:43:18.1164533Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1164793Z "size": 230, 2025-09-07T06:43:18.1165063Z "digest": "sha256:acbd5447dd1406dab8e46234f6a034a75ad9794f76c24f817b0ecf28b6a69c78" 2025-09-07T06:43:18.1165351Z }, 2025-09-07T06:43:18.1165480Z { 2025-09-07T06:43:18.1165683Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1165929Z "size": 3396092, 2025-09-07T06:43:18.1166199Z "digest": "sha256:744523d9b7f5a3e7abfc646c2d5222e7379024242430b93cb4b8093574e69022" 2025-09-07T06:43:18.1166523Z }, 2025-09-07T06:43:18.1166650Z { 2025-09-07T06:43:18.1166846Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1167091Z "size": 1477, 2025-09-07T06:43:18.1167339Z "digest": "sha256:5bd615a7b945084e11bcb40190f9d6e50367297237146df7b008fa8c668f29c8" 2025-09-07T06:43:18.1167610Z }, 2025-09-07T06:43:18.1167730Z { 2025-09-07T06:43:18.1167928Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1168170Z "size": 482, 2025-09-07T06:43:18.1168430Z "digest": "sha256:f4986a00e3aecf1d56beaada7aba8c49fbb3683db3c99790ab0aa4caaa34f76f" 2025-09-07T06:43:18.1168712Z }, 2025-09-07T06:43:18.1168840Z { 2025-09-07T06:43:18.1169044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1169296Z "size": 196, 2025-09-07T06:43:18.1169540Z "digest": "sha256:21902f6e4f8cb76c82e755b8fc9f72e1912bf925ab345ab5b4cc2210f4887a64" 2025-09-07T06:43:18.1169823Z }, 2025-09-07T06:43:18.1169957Z { 2025-09-07T06:43:18.1170165Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1170457Z "size": 608, 2025-09-07T06:43:18.1170721Z "digest": "sha256:d80602abf3ccf0c0b527848a403dfde36e1cf1db1416852385feda5c44bf4363" 2025-09-07T06:43:18.1171010Z }, 2025-09-07T06:43:18.1171140Z { 2025-09-07T06:43:18.1171338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1171590Z "size": 226, 2025-09-07T06:43:18.1171850Z "digest": "sha256:3c51bf0bc362d34a17911f73c5146cbd668c4d1cf1b944cbf40a604d71cd623a" 2025-09-07T06:43:18.1172141Z }, 2025-09-07T06:43:18.1172267Z { 2025-09-07T06:43:18.1172482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1172749Z "size": 828, 2025-09-07T06:43:18.1173007Z "digest": "sha256:119ab3bceafa6f2cab4b1f71161195139792990263ee8de82230c6284f0ae20a" 2025-09-07T06:43:18.1173283Z }, 2025-09-07T06:43:18.1173411Z { 2025-09-07T06:43:18.1173618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1173873Z "size": 32, 2025-09-07T06:43:18.1174134Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1174420Z }, 2025-09-07T06:43:18.1174548Z { 2025-09-07T06:43:18.1174751Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1174993Z "size": 104, 2025-09-07T06:43:18.1175255Z "digest": "sha256:af8eadc9eaabdaf6c5e01031d63061605327153e07568ddd159966ecea75cd07" 2025-09-07T06:43:18.1175543Z }, 2025-09-07T06:43:18.1175675Z { 2025-09-07T06:43:18.1175882Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1176140Z "size": 1495, 2025-09-07T06:43:18.1176408Z "digest": "sha256:e7769b0d7a8262f3cc32a9d96080de5318dac3d2617e10508a167e689016e40c" 2025-09-07T06:43:18.1176694Z }, 2025-09-07T06:43:18.1176818Z { 2025-09-07T06:43:18.1177034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1177300Z "size": 453908015, 2025-09-07T06:43:18.1177586Z "digest": "sha256:ba263639b0f4634277ef3b8903e3457ac27ce012f1bbeeeeb773191c2c3b222b" 2025-09-07T06:43:18.1177859Z }, 2025-09-07T06:43:18.1177989Z { 2025-09-07T06:43:18.1178196Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1178446Z "size": 164, 2025-09-07T06:43:18.1178705Z "digest": "sha256:a5ab7a280382a797dd5ba6a6716f667a231540ad1e0e7c8ba48bb24d5ab80ef0" 2025-09-07T06:43:18.1178989Z }, 2025-09-07T06:43:18.1179121Z { 2025-09-07T06:43:18.1179329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1179580Z "size": 346, 2025-09-07T06:43:18.1179853Z "digest": "sha256:80b2232d952f55c3662cffd657ba30fe825f08dfcc5bbea13e2bc6de4482b7e4" 2025-09-07T06:43:18.1180148Z }, 2025-09-07T06:43:18.1180283Z { 2025-09-07T06:43:18.1180486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1180796Z "size": 32, 2025-09-07T06:43:18.1181069Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1181367Z }, 2025-09-07T06:43:18.1181492Z { 2025-09-07T06:43:18.1181712Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1181983Z "size": 106, 2025-09-07T06:43:18.1182271Z "digest": "sha256:cc93cd65e90f0a9c50194579c93e96897f4e582b9777a1c4d7df7b913ddcdded" 2025-09-07T06:43:18.1182576Z }, 2025-09-07T06:43:18.1182721Z { 2025-09-07T06:43:18.1182945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1183214Z "size": 425, 2025-09-07T06:43:18.1183493Z "digest": "sha256:0eed4c15712bc470dac7df87e33b3570a1510344019dd9cc0e95b8beb1f98372" 2025-09-07T06:43:18.1183791Z }, 2025-09-07T06:43:18.1183923Z { 2025-09-07T06:43:18.1184130Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1184381Z "size": 19309387, 2025-09-07T06:43:18.1184658Z "digest": "sha256:092516f71fe325518f9737f105bcd65c40cd35c3019098889757e2c84c03c8a8" 2025-09-07T06:43:18.1185007Z }, 2025-09-07T06:43:18.1185149Z { 2025-09-07T06:43:18.1185364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1185717Z "size": 108, 2025-09-07T06:43:18.1186022Z "digest": "sha256:8c0825014a6270f765ff514da8583d55874f3278bef76e5617e29115f91ee654" 2025-09-07T06:43:18.1186333Z }, 2025-09-07T06:43:18.1186467Z { 2025-09-07T06:43:18.1186693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1186973Z "size": 636, 2025-09-07T06:43:18.1187259Z "digest": "sha256:8e0d2f63da0a8ff07657d7e06cdbc1ad9d5db95614d640a9f7a9aa8c30c9986d" 2025-09-07T06:43:18.1187587Z }, 2025-09-07T06:43:18.1187723Z { 2025-09-07T06:43:18.1187951Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1188242Z "size": 724, 2025-09-07T06:43:18.1188526Z "digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031" 2025-09-07T06:43:18.1188842Z }, 2025-09-07T06:43:18.1188996Z { 2025-09-07T06:43:18.1189212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1189461Z "size": 148, 2025-09-07T06:43:18.1189724Z "digest": "sha256:73aae7958ba1a16c5f5625d39b06208e1def8c7816bb75028bf0845f553a5068" 2025-09-07T06:43:18.1190014Z }, 2025-09-07T06:43:18.1190150Z { 2025-09-07T06:43:18.1190353Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1190610Z "size": 136, 2025-09-07T06:43:18.1190873Z "digest": "sha256:ac6077ec9fa50fc0822d387d2ee35e1b6f1f56612402fe7195378180b25087bc" 2025-09-07T06:43:18.1191163Z }, 2025-09-07T06:43:18.1191290Z { 2025-09-07T06:43:18.1191508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1191780Z "size": 140, 2025-09-07T06:43:18.1192066Z "digest": "sha256:bf4ee4e45e92ef179f7fc64e2c7c6755905a969c37cf82c39aafbadd9290ff04" 2025-09-07T06:43:18.1192381Z }, 2025-09-07T06:43:18.1192515Z { 2025-09-07T06:43:18.1192742Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1193022Z "size": 18617175577, 2025-09-07T06:43:18.1193325Z "digest": "sha256:c1b766f9b961bcc863d6f89d623815fd7dfe9797ddcfd5d15ef06ffe7d177359" 2025-09-07T06:43:18.1193628Z }, 2025-09-07T06:43:18.1193765Z { 2025-09-07T06:43:18.1193980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1194237Z "size": 223, 2025-09-07T06:43:18.1194499Z "digest": "sha256:6e726ef07b5d5cfe2fb9f06d43fc931fc64c381fd37eaf0c169e0dd84796f152" 2025-09-07T06:43:18.1194791Z }, 2025-09-07T06:43:18.1194931Z { 2025-09-07T06:43:18.1195194Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1195456Z "size": 274477524, 2025-09-07T06:43:18.1195722Z "digest": "sha256:364070434a64fa913f3907ada910a4051707e693e0e6124f57bc97aa57791da1" 2025-09-07T06:43:18.1196086Z }, 2025-09-07T06:43:18.1196222Z { 2025-09-07T06:43:18.1196440Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1196701Z "size": 6451569004, 2025-09-07T06:43:18.1196975Z "digest": "sha256:71f708151a84685fc366b85e914dac9f5279313eff07358d79ecaaeecb0f1c42" 2025-09-07T06:43:18.1197281Z }, 2025-09-07T06:43:18.1197421Z { 2025-09-07T06:43:18.1197653Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1197907Z "size": 129, 2025-09-07T06:43:18.1198182Z "digest": "sha256:622d8cfb39ea4dda608d2819c6a9de45df81b6f8319ee8ab4a24c36d81b9a132" 2025-09-07T06:43:18.1198482Z }, 2025-09-07T06:43:18.1198619Z { 2025-09-07T06:43:18.1198826Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1199089Z "size": 778, 2025-09-07T06:43:18.1199354Z "digest": "sha256:284119a92cb13dacff06926444aab4f99756039acb48abba7b75d35c367ed3f1" 2025-09-07T06:43:18.1199648Z }, 2025-09-07T06:43:18.1199774Z { 2025-09-07T06:43:18.1200005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1200307Z "size": 724, 2025-09-07T06:43:18.1200570Z "digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031" 2025-09-07T06:43:18.1200846Z }, 2025-09-07T06:43:18.1200979Z { 2025-09-07T06:43:18.1201186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1201445Z "size": 140, 2025-09-07T06:43:18.1201692Z "digest": "sha256:96695940d842555623cfe4fb7b52e949423e8c8f383e55d02363e7e5c5804afa" 2025-09-07T06:43:18.1201978Z }, 2025-09-07T06:43:18.1202110Z { 2025-09-07T06:43:18.1202320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1202573Z "size": 32, 2025-09-07T06:43:18.1202869Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1203222Z }, 2025-09-07T06:43:18.1203356Z { 2025-09-07T06:43:18.1203554Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1203814Z "size": 160, 2025-09-07T06:43:18.1204080Z "digest": "sha256:7ddca6c4c050460204097ba875dc0fa03eca6265122a18c0b8dc5504152aea53" 2025-09-07T06:43:18.1204369Z }, 2025-09-07T06:43:18.1204496Z { 2025-09-07T06:43:18.1204706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1204975Z "size": 1012, 2025-09-07T06:43:18.1205271Z "digest": "sha256:a95e1f2f1aadef03514a7cdbdac1fe83d4eebedbb80df9be868a223f27e1c263" 2025-09-07T06:43:18.1205648Z }, 2025-09-07T06:43:18.1205788Z { 2025-09-07T06:43:18.1206005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1206271Z "size": 724, 2025-09-07T06:43:18.1206545Z "digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031" 2025-09-07T06:43:18.1206905Z }, 2025-09-07T06:43:18.1207043Z { 2025-09-07T06:43:18.1207262Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1207526Z "size": 135, 2025-09-07T06:43:18.1207815Z "digest": "sha256:8085756b0cc0f9588f23a73c27840a5dff48cc18c3a2f0311e4d1ef291855679" 2025-09-07T06:43:18.1208117Z }, 2025-09-07T06:43:18.1208258Z { 2025-09-07T06:43:18.1208472Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1208752Z "size": 32, 2025-09-07T06:43:18.1209043Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1209361Z }, 2025-09-07T06:43:18.1209493Z { 2025-09-07T06:43:18.1209716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1209999Z "size": 158, 2025-09-07T06:43:18.1210277Z "digest": "sha256:7e9ff0c6f103b18756f01c60b4d57a951660f17bffb1810b330e3ff703caf216" 2025-09-07T06:43:18.1210576Z }, 2025-09-07T06:43:18.1210718Z { 2025-09-07T06:43:18.1210940Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1211213Z "size": 1369, 2025-09-07T06:43:18.1211549Z "digest": "sha256:a625cbbc05b983aeb4c28702a4a5b65c68191ab1b8d17978f7d98cc17ddf3c52" 2025-09-07T06:43:18.1211864Z }, 2025-09-07T06:43:18.1212006Z { 2025-09-07T06:43:18.1212224Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1212484Z "size": 32, 2025-09-07T06:43:18.1212766Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1213056Z }, 2025-09-07T06:43:18.1213188Z { 2025-09-07T06:43:18.1213388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1213645Z "size": 136, 2025-09-07T06:43:18.1213898Z "digest": "sha256:4e28486424310870c8d6815524440f17c6e0afe7572eaa173a811b98b4920bed" 2025-09-07T06:43:18.1214178Z }, 2025-09-07T06:43:18.1214307Z { 2025-09-07T06:43:18.1214522Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1214777Z "size": 380, 2025-09-07T06:43:18.1215043Z "digest": "sha256:5e944f1ed1bef9442f5b1b86225d3958ea8f2f7f4c6aa7b92dc5d0c810c260bc" 2025-09-07T06:43:18.1215329Z }, 2025-09-07T06:43:18.1215501Z { 2025-09-07T06:43:18.1215716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1215972Z "size": 32, 2025-09-07T06:43:18.1216234Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1216525Z }, 2025-09-07T06:43:18.1216657Z { 2025-09-07T06:43:18.1216864Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1217112Z "size": 104, 2025-09-07T06:43:18.1217373Z "digest": "sha256:41619248f604c60e038a02bfd462af96ee2996b77be5f59f05e9ac5fe4790e5a" 2025-09-07T06:43:18.1217661Z }, 2025-09-07T06:43:18.1217792Z { 2025-09-07T06:43:18.1217994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1218251Z "size": 407, 2025-09-07T06:43:18.1218521Z "digest": "sha256:be86f8c4f654b9ae64a20eb7f960e6ce4baa5b46e0a1f5e1312b11492a40bcd4" 2025-09-07T06:43:18.1218818Z }, 2025-09-07T06:43:18.1218945Z { 2025-09-07T06:43:18.1219157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1219414Z "size": 32, 2025-09-07T06:43:18.1220053Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1220350Z }, 2025-09-07T06:43:18.1220486Z { 2025-09-07T06:43:18.1220701Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1220973Z "size": 109, 2025-09-07T06:43:18.1221248Z "digest": "sha256:ef1340e22a4bc8cf42e1d40961cb32d183cd3da8f0b785b5425c32ee067690c1" 2025-09-07T06:43:18.1221562Z }, 2025-09-07T06:43:18.1221708Z { 2025-09-07T06:43:18.1221935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1222207Z "size": 1897, 2025-09-07T06:43:18.1222501Z "digest": "sha256:da8d8b696333cbf6b9f339ab859639c905d6752d7e65fea14c23c3c2dcba553e" 2025-09-07T06:43:18.1222830Z }, 2025-09-07T06:43:18.1222974Z { 2025-09-07T06:43:18.1223199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1223480Z "size": 243443118, 2025-09-07T06:43:18.1223778Z "digest": "sha256:386b0c49c4982a821fb6f427fbc7d9c7d2012e97c96a514a9c7a09304e76b935" 2025-09-07T06:43:18.1224092Z }, 2025-09-07T06:43:18.1224229Z { 2025-09-07T06:43:18.1224456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1224733Z "size": 106, 2025-09-07T06:43:18.1225024Z "digest": "sha256:2b1d0ea7efe0bf86e86df804d2cddbf83b113fdecd03f3ddfca728da30546f34" 2025-09-07T06:43:18.1225353Z }, 2025-09-07T06:43:18.1225491Z { 2025-09-07T06:43:18.1225786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1226087Z "size": 163, 2025-09-07T06:43:18.1226372Z "digest": "sha256:04c04be7408f20625b1bd8454e5a08c91fcf04d4f79ab3ec1b75ae6b1824174d" 2025-09-07T06:43:18.1226699Z }, 2025-09-07T06:43:18.1226961Z { 2025-09-07T06:43:18.1227187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1227451Z "size": 7943, 2025-09-07T06:43:18.1227737Z "digest": "sha256:f8690caa3ac5e845f2dcc25ad12815b5c7452285c3838a87c780bd03ecf072a3" 2025-09-07T06:43:18.1228046Z }, 2025-09-07T06:43:18.1228184Z { 2025-09-07T06:43:18.1228397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1228656Z "size": 8074, 2025-09-07T06:43:18.1228925Z "digest": "sha256:2908d6baaa6b21331dee5f210472cae0874d22b98b0a35420cad4fd753ed215f" 2025-09-07T06:43:18.1229216Z }, 2025-09-07T06:43:18.1229347Z { 2025-09-07T06:43:18.1229550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1229804Z "size": 303, 2025-09-07T06:43:18.1230065Z "digest": "sha256:37e2336101eba2c73995d34431e4fae8782d9e9700c42621777922490b2158ed" 2025-09-07T06:43:18.1230346Z }, 2025-09-07T06:43:18.1230472Z { 2025-09-07T06:43:18.1230683Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1230946Z "size": 32, 2025-09-07T06:43:18.1231266Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1231557Z }, 2025-09-07T06:43:18.1231688Z { 2025-09-07T06:43:18.1231904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1232160Z "size": 108, 2025-09-07T06:43:18.1232411Z "digest": "sha256:f1ac881fde33994861be4324231269058643168b9aee60c699552d0d92d965da" 2025-09-07T06:43:18.1232704Z }, 2025-09-07T06:43:18.1232844Z { 2025-09-07T06:43:18.1233065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1233339Z "size": 54145699, 2025-09-07T06:43:18.1233605Z "digest": "sha256:43b14c67347e2813c5f63e928c14db60dbb35c330ccc865510cf79739d8b78a1" 2025-09-07T06:43:18.1233886Z }, 2025-09-07T06:43:18.1234013Z { 2025-09-07T06:43:18.1234212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-09-07T06:43:18.1234467Z "size": 32, 2025-09-07T06:43:18.1234729Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-09-07T06:43:18.1235012Z } 2025-09-07T06:43:18.1235136Z ] 2025-09-07T06:43:18.1235269Z } 2025-09-07T06:43:18.1263203Z ##[group]Run set -eux 2025-09-07T06:43:18.1263419Z set -eux 2025-09-07T06:43:18.1263708Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-09-07T06:43:18.1264449Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-09-07T06:43:18.1271632Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:18.1271881Z env: 2025-09-07T06:43:18.1272043Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:18.1272228Z ##[endgroup] 2025-09-07T06:43:18.1299974Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-09-07T06:43:18.1300422Z + docker login --username pytorchbot --password-stdin 2025-09-07T06:43:18.1300677Z + jq --raw-output .SecretString 2025-09-07T06:43:18.1300895Z + jq -r .docker_hub_readonly_token 2025-09-07T06:43:18.6160600Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-09-07T06:43:18.6160984Z Login Succeeded 2025-09-07T06:43:18.6162183Z Configure a credential helper to remove this warning. See 2025-09-07T06:43:18.6162615Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-09-07T06:43:18.6162877Z 2025-09-07T06:43:18.6246701Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-09-07T06:43:18.6246960Z tag=${ECR_DOCKER_IMAGE##*:} 2025-09-07T06:43:18.6247227Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-09-07T06:43:18.6251902Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:18.6252182Z env: 2025-09-07T06:43:18.6252465Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:18.6253006Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:18.6253542Z ##[endgroup] 2025-09-07T06:43:18.6280837Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:18.6369744Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-09-07T06:43:18.6370026Z with: 2025-09-07T06:43:18.6370559Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:18.6371206Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:18.6371462Z env: 2025-09-07T06:43:18.6371619Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:18.6371809Z ##[endgroup] 2025-09-07T06:43:18.6391583Z ##[group]Run set -x 2025-09-07T06:43:18.6391830Z set -x 2025-09-07T06:43:18.6392028Z set +e 2025-09-07T06:43:18.6392214Z  2025-09-07T06:43:18.6392391Z login() { 2025-09-07T06:43:18.6392782Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-09-07T06:43:18.6393182Z } 2025-09-07T06:43:18.6393373Z  2025-09-07T06:43:18.6393590Z retry () { 2025-09-07T06:43:18.6393824Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-09-07T06:43:18.6394068Z } 2025-09-07T06:43:18.6394236Z  2025-09-07T06:43:18.6394428Z retry login "${DOCKER_REGISTRY}" 2025-09-07T06:43:18.6394666Z  2025-09-07T06:43:18.6395028Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-09-07T06:43:18.6395519Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-09-07T06:43:18.6395823Z  2025-09-07T06:43:18.6395991Z set -e 2025-09-07T06:43:18.6396251Z # ignore output since only exit code is used for conditional 2025-09-07T06:43:18.6396616Z # only pull docker image if it's not available locally 2025-09-07T06:43:18.6397011Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-09-07T06:43:18.6397394Z  retry docker pull "${DOCKER_IMAGE}" 2025-09-07T06:43:18.6397647Z fi 2025-09-07T06:43:18.6402460Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:43:18.6402717Z env: 2025-09-07T06:43:18.6402887Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:43:18.6403478Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:18.6404123Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:18.6404406Z ##[endgroup] 2025-09-07T06:43:18.6426049Z + set +e 2025-09-07T06:43:18.6426327Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:18.6426639Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:18.6434728Z + aws ecr get-login-password --region us-east-1 2025-09-07T06:43:18.6439668Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-09-07T06:43:19.0573801Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-09-07T06:43:19.0574186Z Login Succeeded 2025-09-07T06:43:19.0574616Z Configure a credential helper to remove this warning. See 2025-09-07T06:43:19.0575643Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-09-07T06:43:19.0576009Z 2025-09-07T06:43:19.0604445Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:19.0605196Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-09-07T06:43:19.2836658Z + IMAGE_SIZE=28579.020259857178 2025-09-07T06:43:19.2836974Z Compressed size of image in MB: 28579.020259857178 2025-09-07T06:43:19.2837420Z + echo 'Compressed size of image in MB: 28579.020259857178' 2025-09-07T06:43:19.2837760Z + set -e 2025-09-07T06:43:19.2838729Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:19.2966260Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:19.2967387Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:43:19.5183589Z pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77: Pulling from pytorch/ci-image 2025-09-07T06:43:19.5184232Z e6fdc8487bfe: Pulling fs layer 2025-09-07T06:43:19.5184454Z 18a5ee5b0e2e: Pulling fs layer 2025-09-07T06:43:19.5184653Z 572424b92528: Pulling fs layer 2025-09-07T06:43:19.5184881Z 1c35b7d4b67c: Pulling fs layer 2025-09-07T06:43:19.5185096Z 68c20f3c23bb: Pulling fs layer 2025-09-07T06:43:19.5185296Z 7efa39950d32: Pulling fs layer 2025-09-07T06:43:19.5185574Z a10eb16a7271: Pulling fs layer 2025-09-07T06:43:19.5185986Z 7d52cf579654: Pulling fs layer 2025-09-07T06:43:19.5186205Z cb6a20fcf4e2: Pulling fs layer 2025-09-07T06:43:19.5186412Z 46fb6a8b3e1d: Pulling fs layer 2025-09-07T06:43:19.5186641Z 5ad6977cc38e: Pulling fs layer 2025-09-07T06:43:19.5186860Z da63046995a2: Pulling fs layer 2025-09-07T06:43:19.5187071Z 78243fdb9906: Pulling fs layer 2025-09-07T06:43:19.5187254Z 6f70d5d50aba: Pulling fs layer 2025-09-07T06:43:19.5187434Z 4f4fb700ef54: Pulling fs layer 2025-09-07T06:43:19.5187606Z 69715d3ad3c4: Pulling fs layer 2025-09-07T06:43:19.5187787Z 7ace90c063f3: Pulling fs layer 2025-09-07T06:43:19.5187971Z acbd5447dd14: Pulling fs layer 2025-09-07T06:43:19.5188150Z 744523d9b7f5: Pulling fs layer 2025-09-07T06:43:19.5188322Z 5bd615a7b945: Pulling fs layer 2025-09-07T06:43:19.5188501Z f4986a00e3ae: Pulling fs layer 2025-09-07T06:43:19.5188680Z 21902f6e4f8c: Pulling fs layer 2025-09-07T06:43:19.5188866Z d80602abf3cc: Pulling fs layer 2025-09-07T06:43:19.5189061Z 3c51bf0bc362: Pulling fs layer 2025-09-07T06:43:19.5189254Z 119ab3bceafa: Pulling fs layer 2025-09-07T06:43:19.5189455Z af8eadc9eaab: Pulling fs layer 2025-09-07T06:43:19.5189653Z e7769b0d7a82: Pulling fs layer 2025-09-07T06:43:19.5189844Z ba263639b0f4: Pulling fs layer 2025-09-07T06:43:19.5190066Z a5ab7a280382: Pulling fs layer 2025-09-07T06:43:19.5190248Z 80b2232d952f: Pulling fs layer 2025-09-07T06:43:19.5190430Z cc93cd65e90f: Pulling fs layer 2025-09-07T06:43:19.5190606Z 0eed4c15712b: Pulling fs layer 2025-09-07T06:43:19.5190786Z 092516f71fe3: Pulling fs layer 2025-09-07T06:43:19.5190969Z 68c20f3c23bb: Waiting 2025-09-07T06:43:19.5191142Z 8c0825014a62: Pulling fs layer 2025-09-07T06:43:19.5191318Z 8e0d2f63da0a: Pulling fs layer 2025-09-07T06:43:19.5191503Z 7efa39950d32: Waiting 2025-09-07T06:43:19.5191670Z 73aae7958ba1: Pulling fs layer 2025-09-07T06:43:19.5191853Z ac6077ec9fa5: Pulling fs layer 2025-09-07T06:43:19.5192023Z a10eb16a7271: Waiting 2025-09-07T06:43:19.5192191Z bf4ee4e45e92: Pulling fs layer 2025-09-07T06:43:19.5192374Z c1b766f9b961: Pulling fs layer 2025-09-07T06:43:19.5192555Z 6e726ef07b5d: Pulling fs layer 2025-09-07T06:43:19.5192729Z 364070434a64: Pulling fs layer 2025-09-07T06:43:19.5192909Z 71f708151a84: Pulling fs layer 2025-09-07T06:43:19.5193092Z 622d8cfb39ea: Pulling fs layer 2025-09-07T06:43:19.5193274Z 284119a92cb1: Pulling fs layer 2025-09-07T06:43:19.5193446Z 96695940d842: Pulling fs layer 2025-09-07T06:43:19.5193627Z 7ddca6c4c050: Pulling fs layer 2025-09-07T06:43:19.5193814Z a95e1f2f1aad: Pulling fs layer 2025-09-07T06:43:19.5193992Z 8085756b0cc0: Pulling fs layer 2025-09-07T06:43:19.5194431Z 7e9ff0c6f103: Pulling fs layer 2025-09-07T06:43:19.5194613Z a625cbbc05b9: Pulling fs layer 2025-09-07T06:43:19.5194791Z 4e2848642431: Pulling fs layer 2025-09-07T06:43:19.5194968Z 5e944f1ed1be: Pulling fs layer 2025-09-07T06:43:19.5195151Z 41619248f604: Pulling fs layer 2025-09-07T06:43:19.5195334Z be86f8c4f654: Pulling fs layer 2025-09-07T06:43:19.5195518Z ef1340e22a4b: Pulling fs layer 2025-09-07T06:43:19.5195801Z da8d8b696333: Pulling fs layer 2025-09-07T06:43:19.5195987Z 386b0c49c498: Pulling fs layer 2025-09-07T06:43:19.5196170Z 2b1d0ea7efe0: Pulling fs layer 2025-09-07T06:43:19.5196353Z 04c04be7408f: Pulling fs layer 2025-09-07T06:43:19.5196522Z 7d52cf579654: Waiting 2025-09-07T06:43:19.5196699Z f8690caa3ac5: Pulling fs layer 2025-09-07T06:43:19.5196894Z cb6a20fcf4e2: Waiting 2025-09-07T06:43:19.5197074Z 2908d6baaa6b: Pulling fs layer 2025-09-07T06:43:19.5197261Z 1c35b7d4b67c: Waiting 2025-09-07T06:43:19.5197433Z 46fb6a8b3e1d: Waiting 2025-09-07T06:43:19.5197615Z 37e2336101eb: Pulling fs layer 2025-09-07T06:43:19.5197808Z 5ad6977cc38e: Waiting 2025-09-07T06:43:19.5197982Z f1ac881fde33: Pulling fs layer 2025-09-07T06:43:19.5198176Z da63046995a2: Waiting 2025-09-07T06:43:19.5198341Z 43b14c67347e: Pulling fs layer 2025-09-07T06:43:19.5198518Z acbd5447dd14: Waiting 2025-09-07T06:43:19.5198671Z 744523d9b7f5: Waiting 2025-09-07T06:43:19.5198828Z 5bd615a7b945: Waiting 2025-09-07T06:43:19.5198990Z 284119a92cb1: Waiting 2025-09-07T06:43:19.5199144Z 41619248f604: Waiting 2025-09-07T06:43:19.5199293Z be86f8c4f654: Waiting 2025-09-07T06:43:19.5199451Z 0eed4c15712b: Waiting 2025-09-07T06:43:19.5199607Z 092516f71fe3: Waiting 2025-09-07T06:43:19.5199763Z 78243fdb9906: Waiting 2025-09-07T06:43:19.5199911Z 96695940d842: Waiting 2025-09-07T06:43:19.5200064Z c1b766f9b961: Waiting 2025-09-07T06:43:19.5200221Z 6e726ef07b5d: Waiting 2025-09-07T06:43:19.5200400Z 7ddca6c4c050: Waiting 2025-09-07T06:43:19.5200551Z 364070434a64: Waiting 2025-09-07T06:43:19.5200710Z a95e1f2f1aad: Waiting 2025-09-07T06:43:19.5200875Z 8085756b0cc0: Waiting 2025-09-07T06:43:19.5201032Z 8c0825014a62: Waiting 2025-09-07T06:43:19.5201182Z 71f708151a84: Waiting 2025-09-07T06:43:19.5201339Z 7e9ff0c6f103: Waiting 2025-09-07T06:43:19.5201497Z 8e0d2f63da0a: Waiting 2025-09-07T06:43:19.5201657Z 622d8cfb39ea: Waiting 2025-09-07T06:43:19.5201808Z 4e2848642431: Waiting 2025-09-07T06:43:19.5201972Z 5e944f1ed1be: Waiting 2025-09-07T06:43:19.5202139Z 73aae7958ba1: Waiting 2025-09-07T06:43:19.5202300Z bf4ee4e45e92: Waiting 2025-09-07T06:43:19.5202453Z f4986a00e3ae: Waiting 2025-09-07T06:43:19.5202611Z ef1340e22a4b: Waiting 2025-09-07T06:43:19.5202772Z a625cbbc05b9: Waiting 2025-09-07T06:43:19.5202932Z da8d8b696333: Waiting 2025-09-07T06:43:19.5203081Z 386b0c49c498: Waiting 2025-09-07T06:43:19.5203241Z ac6077ec9fa5: Waiting 2025-09-07T06:43:19.5203399Z 2908d6baaa6b: Waiting 2025-09-07T06:43:19.5203561Z 2b1d0ea7efe0: Waiting 2025-09-07T06:43:19.5203713Z 04c04be7408f: Waiting 2025-09-07T06:43:19.5203871Z f8690caa3ac5: Waiting 2025-09-07T06:43:19.5204036Z f1ac881fde33: Waiting 2025-09-07T06:43:19.5204197Z 43b14c67347e: Waiting 2025-09-07T06:43:19.5204348Z 37e2336101eb: Waiting 2025-09-07T06:43:19.5204506Z 4f4fb700ef54: Waiting 2025-09-07T06:43:19.5204666Z 69715d3ad3c4: Waiting 2025-09-07T06:43:19.5204818Z 7ace90c063f3: Waiting 2025-09-07T06:43:19.5204978Z 21902f6e4f8c: Waiting 2025-09-07T06:43:19.5205135Z cc93cd65e90f: Waiting 2025-09-07T06:43:19.5205298Z d80602abf3cc: Waiting 2025-09-07T06:43:19.5205451Z a5ab7a280382: Waiting 2025-09-07T06:43:19.5205609Z 3c51bf0bc362: Waiting 2025-09-07T06:43:19.5205766Z ba263639b0f4: Waiting 2025-09-07T06:43:19.5205921Z 80b2232d952f: Waiting 2025-09-07T06:43:19.5206072Z e7769b0d7a82: Waiting 2025-09-07T06:43:19.5206234Z af8eadc9eaab: Waiting 2025-09-07T06:43:19.5206395Z 6f70d5d50aba: Waiting 2025-09-07T06:43:19.5878745Z 18a5ee5b0e2e: Verifying Checksum 2025-09-07T06:43:19.5880449Z 18a5ee5b0e2e: Download complete 2025-09-07T06:43:19.6765443Z 1c35b7d4b67c: Verifying Checksum 2025-09-07T06:43:19.6765742Z 1c35b7d4b67c: Download complete 2025-09-07T06:43:19.7748923Z 68c20f3c23bb: Verifying Checksum 2025-09-07T06:43:19.7749229Z 68c20f3c23bb: Download complete 2025-09-07T06:43:19.8689011Z 7efa39950d32: Verifying Checksum 2025-09-07T06:43:19.8689305Z 7efa39950d32: Download complete 2025-09-07T06:43:19.8795823Z e6fdc8487bfe: Verifying Checksum 2025-09-07T06:43:19.8801614Z e6fdc8487bfe: Download complete 2025-09-07T06:43:19.9530326Z a10eb16a7271: Verifying Checksum 2025-09-07T06:43:19.9530652Z a10eb16a7271: Download complete 2025-09-07T06:43:19.9690638Z 7d52cf579654: Verifying Checksum 2025-09-07T06:43:19.9691123Z 7d52cf579654: Download complete 2025-09-07T06:43:20.0522347Z 46fb6a8b3e1d: Verifying Checksum 2025-09-07T06:43:20.0527832Z 46fb6a8b3e1d: Download complete 2025-09-07T06:43:20.1455196Z 5ad6977cc38e: Verifying Checksum 2025-09-07T06:43:20.1456223Z 5ad6977cc38e: Download complete 2025-09-07T06:43:20.2195563Z da63046995a2: Verifying Checksum 2025-09-07T06:43:20.2195893Z da63046995a2: Download complete 2025-09-07T06:43:20.3140093Z 78243fdb9906: Download complete 2025-09-07T06:43:21.0403309Z e6fdc8487bfe: Pull complete 2025-09-07T06:43:21.0562799Z 18a5ee5b0e2e: Pull complete 2025-09-07T06:43:21.1255493Z cb6a20fcf4e2: Verifying Checksum 2025-09-07T06:43:21.1255835Z cb6a20fcf4e2: Download complete 2025-09-07T06:43:21.1314448Z 4f4fb700ef54: Verifying Checksum 2025-09-07T06:43:21.1315193Z 4f4fb700ef54: Download complete 2025-09-07T06:43:21.2001468Z 69715d3ad3c4: Verifying Checksum 2025-09-07T06:43:21.2001766Z 69715d3ad3c4: Download complete 2025-09-07T06:43:21.2699811Z 7ace90c063f3: Download complete 2025-09-07T06:43:21.3428731Z acbd5447dd14: Verifying Checksum 2025-09-07T06:43:21.3429098Z acbd5447dd14: Download complete 2025-09-07T06:43:21.4359290Z 744523d9b7f5: Download complete 2025-09-07T06:43:21.5117393Z 5bd615a7b945: Verifying Checksum 2025-09-07T06:43:21.5122384Z 5bd615a7b945: Download complete 2025-09-07T06:43:21.5929723Z f4986a00e3ae: Verifying Checksum 2025-09-07T06:43:21.5933792Z f4986a00e3ae: Download complete 2025-09-07T06:43:21.6675419Z 21902f6e4f8c: Verifying Checksum 2025-09-07T06:43:21.6675958Z 21902f6e4f8c: Download complete 2025-09-07T06:43:21.7571728Z d80602abf3cc: Verifying Checksum 2025-09-07T06:43:21.7572216Z d80602abf3cc: Download complete 2025-09-07T06:43:21.8126720Z 3c51bf0bc362: Download complete 2025-09-07T06:43:21.8890383Z 119ab3bceafa: Verifying Checksum 2025-09-07T06:43:21.8890707Z 119ab3bceafa: Download complete 2025-09-07T06:43:21.9822373Z af8eadc9eaab: Download complete 2025-09-07T06:43:22.0499580Z e7769b0d7a82: Verifying Checksum 2025-09-07T06:43:22.0499864Z e7769b0d7a82: Download complete 2025-09-07T06:43:22.7127724Z 572424b92528: Verifying Checksum 2025-09-07T06:43:22.7131177Z 572424b92528: Download complete 2025-09-07T06:43:22.7691504Z a5ab7a280382: Verifying Checksum 2025-09-07T06:43:22.7693179Z a5ab7a280382: Download complete 2025-09-07T06:43:22.8457189Z 80b2232d952f: Verifying Checksum 2025-09-07T06:43:22.8459570Z 80b2232d952f: Download complete 2025-09-07T06:43:22.9175649Z cc93cd65e90f: Verifying Checksum 2025-09-07T06:43:22.9176201Z cc93cd65e90f: Download complete 2025-09-07T06:43:22.9904703Z 0eed4c15712b: Download complete 2025-09-07T06:43:23.2307584Z 092516f71fe3: Verifying Checksum 2025-09-07T06:43:23.2308089Z 092516f71fe3: Download complete 2025-09-07T06:43:23.3049536Z 8c0825014a62: Verifying Checksum 2025-09-07T06:43:23.3053883Z 8c0825014a62: Download complete 2025-09-07T06:43:23.3769158Z 8e0d2f63da0a: Verifying Checksum 2025-09-07T06:43:23.3769481Z 8e0d2f63da0a: Download complete 2025-09-07T06:43:23.4516121Z 73aae7958ba1: Verifying Checksum 2025-09-07T06:43:23.4516434Z 73aae7958ba1: Download complete 2025-09-07T06:43:23.5251078Z ac6077ec9fa5: Verifying Checksum 2025-09-07T06:43:23.5251416Z ac6077ec9fa5: Download complete 2025-09-07T06:43:23.6145909Z bf4ee4e45e92: Verifying Checksum 2025-09-07T06:43:23.6146243Z bf4ee4e45e92: Download complete 2025-09-07T06:43:26.6370439Z ba263639b0f4: Verifying Checksum 2025-09-07T06:43:26.6370758Z ba263639b0f4: Download complete 2025-09-07T06:43:26.7033666Z 6e726ef07b5d: Download complete 2025-09-07T06:43:29.5002919Z 364070434a64: Verifying Checksum 2025-09-07T06:43:29.5003224Z 364070434a64: Download complete 2025-09-07T06:43:33.6847279Z 572424b92528: Pull complete 2025-09-07T06:43:33.8663993Z 1c35b7d4b67c: Pull complete 2025-09-07T06:43:34.0547431Z 68c20f3c23bb: Pull complete 2025-09-07T06:43:34.2423829Z 7efa39950d32: Pull complete 2025-09-07T06:43:34.4516117Z a10eb16a7271: Pull complete 2025-09-07T06:43:34.6296832Z 7d52cf579654: Pull complete 2025-09-07T06:43:38.2361927Z cb6a20fcf4e2: Pull complete 2025-09-07T06:43:38.5001857Z 46fb6a8b3e1d: Pull complete 2025-09-07T06:43:38.7531014Z 5ad6977cc38e: Pull complete 2025-09-07T06:43:39.0504370Z da63046995a2: Pull complete 2025-09-07T06:43:39.3116682Z 78243fdb9906: Pull complete 2025-09-07T06:43:54.3198167Z 6f70d5d50aba: Verifying Checksum 2025-09-07T06:43:54.3198659Z 6f70d5d50aba: Download complete 2025-09-07T06:43:54.4049375Z 622d8cfb39ea: Verifying Checksum 2025-09-07T06:43:54.4049871Z 622d8cfb39ea: Download complete 2025-09-07T06:43:54.4941095Z 284119a92cb1: Verifying Checksum 2025-09-07T06:43:54.4943692Z 284119a92cb1: Download complete 2025-09-07T06:43:54.5783814Z 96695940d842: Download complete 2025-09-07T06:43:54.6528741Z 7ddca6c4c050: Verifying Checksum 2025-09-07T06:43:54.6529066Z 7ddca6c4c050: Download complete 2025-09-07T06:43:54.7195911Z a95e1f2f1aad: Verifying Checksum 2025-09-07T06:43:54.7196272Z a95e1f2f1aad: Download complete 2025-09-07T06:43:54.8078358Z 8085756b0cc0: Verifying Checksum 2025-09-07T06:43:54.8078867Z 8085756b0cc0: Download complete 2025-09-07T06:43:54.8721184Z 7e9ff0c6f103: Verifying Checksum 2025-09-07T06:43:54.8721500Z 7e9ff0c6f103: Download complete 2025-09-07T06:43:54.9348732Z a625cbbc05b9: Verifying Checksum 2025-09-07T06:43:55.0126862Z a625cbbc05b9: Download complete 2025-09-07T06:43:55.0127190Z 4e2848642431: Verifying Checksum 2025-09-07T06:43:55.0133047Z 4e2848642431: Download complete 2025-09-07T06:43:55.0905402Z 5e944f1ed1be: Verifying Checksum 2025-09-07T06:43:55.0906006Z 5e944f1ed1be: Download complete 2025-09-07T06:43:55.1694651Z 41619248f604: Verifying Checksum 2025-09-07T06:43:55.1701307Z 41619248f604: Download complete 2025-09-07T06:43:55.2466050Z be86f8c4f654: Download complete 2025-09-07T06:43:55.3228588Z ef1340e22a4b: Verifying Checksum 2025-09-07T06:43:55.3229073Z ef1340e22a4b: Download complete 2025-09-07T06:43:55.3993187Z da8d8b696333: Download complete 2025-09-07T06:43:57.8752619Z 386b0c49c498: Verifying Checksum 2025-09-07T06:43:57.8756359Z 386b0c49c498: Download complete 2025-09-07T06:43:57.9729627Z 2b1d0ea7efe0: Download complete 2025-09-07T06:43:58.0651887Z 04c04be7408f: Verifying Checksum 2025-09-07T06:43:58.0652359Z 04c04be7408f: Download complete 2025-09-07T06:43:58.1440417Z f8690caa3ac5: Verifying Checksum 2025-09-07T06:43:58.1440746Z f8690caa3ac5: Download complete 2025-09-07T06:43:58.2281887Z 2908d6baaa6b: Verifying Checksum 2025-09-07T06:43:58.2285334Z 2908d6baaa6b: Download complete 2025-09-07T06:43:58.3021212Z 37e2336101eb: Download complete 2025-09-07T06:43:58.3942963Z f1ac881fde33: Verifying Checksum 2025-09-07T06:43:58.3943280Z f1ac881fde33: Download complete 2025-09-07T06:43:58.9737345Z 43b14c67347e: Verifying Checksum 2025-09-07T06:43:58.9737628Z 43b14c67347e: Download complete 2025-09-07T06:44:34.0795069Z 71f708151a84: Verifying Checksum 2025-09-07T06:44:34.0795416Z 71f708151a84: Download complete 2025-09-07T06:45:13.0353223Z 6f70d5d50aba: Pull complete 2025-09-07T06:45:13.3063928Z 4f4fb700ef54: Pull complete 2025-09-07T06:45:13.6216989Z 69715d3ad3c4: Pull complete 2025-09-07T06:45:13.9694560Z 7ace90c063f3: Pull complete 2025-09-07T06:45:14.1295331Z acbd5447dd14: Pull complete 2025-09-07T06:45:14.7246077Z 744523d9b7f5: Pull complete 2025-09-07T06:45:15.1577133Z 5bd615a7b945: Pull complete 2025-09-07T06:45:15.4642870Z f4986a00e3ae: Pull complete 2025-09-07T06:45:15.7195551Z 21902f6e4f8c: Pull complete 2025-09-07T06:45:16.0430759Z d80602abf3cc: Pull complete 2025-09-07T06:45:16.3681484Z 3c51bf0bc362: Pull complete 2025-09-07T06:45:16.7345882Z 119ab3bceafa: Pull complete 2025-09-07T06:45:17.5224603Z af8eadc9eaab: Pull complete 2025-09-07T06:45:18.0383912Z e7769b0d7a82: Pull complete 2025-09-07T06:45:29.5745348Z ba263639b0f4: Pull complete 2025-09-07T06:45:30.0664074Z a5ab7a280382: Pull complete 2025-09-07T06:45:30.5984504Z 80b2232d952f: Pull complete 2025-09-07T06:45:31.4788079Z cc93cd65e90f: Pull complete 2025-09-07T06:45:31.9030464Z 0eed4c15712b: Pull complete 2025-09-07T06:45:32.7065531Z 092516f71fe3: Pull complete 2025-09-07T06:45:33.1410063Z 8c0825014a62: Pull complete 2025-09-07T06:45:33.6817858Z 8e0d2f63da0a: Pull complete 2025-09-07T06:45:34.4940929Z 73aae7958ba1: Pull complete 2025-09-07T06:45:34.9006357Z ac6077ec9fa5: Pull complete 2025-09-07T06:45:35.2139918Z bf4ee4e45e92: Pull complete 2025-09-07T06:46:29.8361162Z c1b766f9b961: Verifying Checksum 2025-09-07T06:46:29.8365347Z c1b766f9b961: Download complete 2025-09-07T06:50:43.5607787Z c1b766f9b961: Pull complete 2025-09-07T06:50:43.5860433Z 6e726ef07b5d: Pull complete 2025-09-07T06:50:45.8368402Z 364070434a64: Pull complete 2025-09-07T06:53:15.2551865Z 71f708151a84: Pull complete 2025-09-07T06:53:15.2837938Z 622d8cfb39ea: Pull complete 2025-09-07T06:53:15.3142275Z 284119a92cb1: Pull complete 2025-09-07T06:53:15.3630887Z 96695940d842: Pull complete 2025-09-07T06:53:15.4185354Z 7ddca6c4c050: Pull complete 2025-09-07T06:53:15.4455473Z a95e1f2f1aad: Pull complete 2025-09-07T06:53:15.5002980Z 8085756b0cc0: Pull complete 2025-09-07T06:53:15.5540679Z 7e9ff0c6f103: Pull complete 2025-09-07T06:53:15.5815804Z a625cbbc05b9: Pull complete 2025-09-07T06:53:15.6360651Z 4e2848642431: Pull complete 2025-09-07T06:53:15.6620902Z 5e944f1ed1be: Pull complete 2025-09-07T06:53:15.7183330Z 41619248f604: Pull complete 2025-09-07T06:53:15.7446255Z be86f8c4f654: Pull complete 2025-09-07T06:53:15.7956246Z ef1340e22a4b: Pull complete 2025-09-07T06:53:15.8223749Z da8d8b696333: Pull complete 2025-09-07T06:53:25.8465195Z 386b0c49c498: Pull complete 2025-09-07T06:53:26.3364446Z 2b1d0ea7efe0: Pull complete 2025-09-07T06:53:26.8473804Z 04c04be7408f: Pull complete 2025-09-07T06:53:27.3850734Z f8690caa3ac5: Pull complete 2025-09-07T06:53:27.7423912Z 2908d6baaa6b: Pull complete 2025-09-07T06:53:28.1418782Z 37e2336101eb: Pull complete 2025-09-07T06:53:29.1819859Z f1ac881fde33: Pull complete 2025-09-07T06:53:32.0696994Z 43b14c67347e: Pull complete 2025-09-07T06:53:32.6307439Z Digest: sha256:383efb45082f20b8c808cb0ba4df693a01359592233f641f1f486911ac320a9a 2025-09-07T06:53:32.7271678Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:53:32.7699199Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:53:32.7751434Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-09-07T06:53:32.7752051Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-09-07T06:53:32.7760981Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:32.7761244Z env: 2025-09-07T06:53:32.7761446Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:32.7761646Z ##[endgroup] 2025-09-07T06:53:32.7843714Z Prepare all required actions 2025-09-07T06:53:32.8087279Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-09-07T06:53:32.8087520Z with: 2025-09-07T06:53:32.8088111Z github-token: *** 2025-09-07T06:53:32.8088271Z env: 2025-09-07T06:53:32.8088429Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:32.8088669Z ##[endgroup] 2025-09-07T06:53:32.8233486Z ##[group]Run set -eux 2025-09-07T06:53:32.8233708Z set -eux 2025-09-07T06:53:32.8234013Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-09-07T06:53:32.8238915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:32.8239406Z env: 2025-09-07T06:53:32.8239572Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:32.8239976Z GITHUB_TOKEN: *** 2025-09-07T06:53:32.8240169Z ##[endgroup] 2025-09-07T06:53:32.8263986Z + python3 .github/scripts/get_workflow_job_id.py 17524754606 i-085acfb4aecab35f4 2025-09-07T06:53:33.4928023Z Setting output job-id=49774397867 2025-09-07T06:53:33.4933437Z Setting output job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:53:33.5206497Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-09-07T06:53:33.5206987Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-09-07T06:53:33.5207592Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-09-07T06:53:33.5208100Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-09-07T06:53:33.5213614Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:33.5213864Z env: 2025-09-07T06:53:33.5214032Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:33.5214221Z JOB_ID: 49774397867 2025-09-07T06:53:33.5214525Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:53:33.5214859Z WORKFLOW_NAME: inductor 2025-09-07T06:53:33.5215046Z WORKFLOW_RUN_ID: 17524754606 2025-09-07T06:53:33.5215280Z MONITOR_LOG_INTERVAL: 5 2025-09-07T06:53:33.5215464Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-09-07T06:53:33.5215666Z ##[endgroup] 2025-09-07T06:53:34.1981244Z Defaulting to user installation because normal site-packages is not writeable 2025-09-07T06:53:34.5087597Z Collecting psutil==5.9.8 2025-09-07T06:53:34.5241400Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-09-07T06:53:34.6707433Z Collecting dataclasses_json==0.6.7 2025-09-07T06:53:34.6746934Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-09-07T06:53:34.7328156Z Collecting nvidia-ml-py==11.525.84 2025-09-07T06:53:34.7378751Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-09-07T06:53:34.8684061Z Collecting marshmallow<4.0.0,>=3.18.0 2025-09-07T06:53:34.8719891Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-09-07T06:53:34.9759188Z Collecting typing-inspect<1,>=0.4.0 2025-09-07T06:53:34.9802388Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-09-07T06:53:35.1183760Z Collecting packaging>=17.0 2025-09-07T06:53:35.1220988Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-09-07T06:53:35.2608054Z Collecting typing-extensions>=3.7.4 2025-09-07T06:53:35.2645877Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-09-07T06:53:35.3661432Z Collecting mypy-extensions>=0.3.0 2025-09-07T06:53:35.3694244Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-09-07T06:53:35.5814317Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-09-07T06:53:36.2582583Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-09-07T06:53:36.5326432Z Prepare all required actions 2025-09-07T06:53:36.5326917Z Getting action download info 2025-09-07T06:53:36.7026662Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-09-07T06:53:37.3224583Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-09-07T06:53:38.3095518Z ##[group]Run ./.github/actions/download-build-artifacts 2025-09-07T06:53:38.3095785Z with: 2025-09-07T06:53:38.3095978Z name: linux-jammy-py3.9-gcc11-build 2025-09-07T06:53:38.3096200Z s3-bucket: gha-artifacts 2025-09-07T06:53:38.3096517Z env: 2025-09-07T06:53:38.3096680Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:38.3096864Z ##[endgroup] 2025-09-07T06:53:38.3126099Z ##[group]Run seemethere/download-artifact-s3@v4 2025-09-07T06:53:38.3126336Z with: 2025-09-07T06:53:38.3126514Z name: linux-jammy-py3.9-gcc11-build 2025-09-07T06:53:38.3126736Z s3-bucket: gha-artifacts 2025-09-07T06:53:38.3126970Z region: us-east-1 2025-09-07T06:53:38.3127126Z env: 2025-09-07T06:53:38.3127280Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:38.3127457Z ##[endgroup] 2025-09-07T06:53:38.6989417Z (node:48162) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-09-07T06:53:38.6993525Z 2025-09-07T06:53:38.6994064Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-09-07T06:53:38.6994508Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-09-07T06:53:38.6994936Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-09-07T06:53:38.9980935Z Found 1 objects with prefix pytorch/pytorch/17524754606/linux-jammy-py3.9-gcc11-build/ 2025-09-07T06:53:38.9981564Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-09-07T06:53:45.9847061Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-09-07T06:53:45.9851777Z Artifact download has finished successfully 2025-09-07T06:53:46.0041399Z ##[group]Run unzip -o artifacts.zip 2025-09-07T06:53:46.0041653Z unzip -o artifacts.zip 2025-09-07T06:53:46.0046155Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:46.0046399Z env: 2025-09-07T06:53:46.0046561Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:46.0046744Z ##[endgroup] 2025-09-07T06:53:46.0946307Z Archive: artifacts.zip 2025-09-07T06:53:46.0946838Z creating: dist/ 2025-09-07T06:53:47.2048974Z inflating: dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl 2025-09-07T06:53:47.2053863Z creating: dist/vision/ 2025-09-07T06:53:47.2128258Z inflating: dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-09-07T06:53:47.2130699Z creating: dist/audio/ 2025-09-07T06:53:47.2154992Z inflating: dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl 2025-09-07T06:53:47.2155323Z creating: dist/ao/ 2025-09-07T06:53:47.2192617Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-09-07T06:53:47.2312168Z inflating: dist/.ninja_log 2025-09-07T06:53:47.2317427Z creating: build/custom_test_artifacts/ 2025-09-07T06:53:47.2317777Z creating: build/custom_test_artifacts/custom-op-build/ 2025-09-07T06:53:47.2318133Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-09-07T06:53:47.2318566Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-09-07T06:53:47.2319058Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-09-07T06:53:47.2319461Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/ 2025-09-07T06:53:47.2320087Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-09-07T06:53:47.2320522Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-09-07T06:53:47.2321388Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-09-07T06:53:47.2321862Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-09-07T06:53:47.2322380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-09-07T06:53:47.2322855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-09-07T06:53:47.2323360Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-09-07T06:53:47.2323823Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-09-07T06:53:47.2324489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-09-07T06:53:47.2324994Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-09-07T06:53:47.2325468Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-09-07T06:53:47.2326016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-09-07T06:53:47.2326748Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-09-07T06:53:47.2327256Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-09-07T06:53:47.2327685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-09-07T06:53:47.2328121Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-09-07T06:53:47.2328602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-09-07T06:53:47.2329134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-09-07T06:53:47.2329651Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-09-07T06:53:47.2330131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-09-07T06:53:47.2330628Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-09-07T06:53:47.2331166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-09-07T06:53:47.2331723Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-09-07T06:53:47.2332230Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-09-07T06:53:47.2332703Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-09-07T06:53:47.2353025Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-09-07T06:53:47.2525599Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-09-07T06:53:47.2526128Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-09-07T06:53:47.2526613Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-09-07T06:53:47.2527137Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-09-07T06:53:47.2527649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-09-07T06:53:47.2528119Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-09-07T06:53:47.2528614Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-09-07T06:53:47.2529089Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-09-07T06:53:47.2529905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-09-07T06:53:47.2530381Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-09-07T06:53:47.2530851Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-09-07T06:53:47.2547032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-09-07T06:53:47.2621297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-09-07T06:53:47.2621917Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-09-07T06:53:47.2622791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-09-07T06:53:47.2623271Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-09-07T06:53:47.2623720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-09-07T06:53:47.2624180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-09-07T06:53:47.2624646Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json 2025-09-07T06:53:47.2625089Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-09-07T06:53:47.2625478Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-09-07T06:53:47.2626124Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-09-07T06:53:47.2786778Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-09-07T06:53:47.2835804Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-09-07T06:53:47.2839920Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-09-07T06:53:47.2840356Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-09-07T06:53:47.2840783Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-09-07T06:53:47.2841266Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-09-07T06:53:47.2841711Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/ 2025-09-07T06:53:47.2842101Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-09-07T06:53:47.2842623Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-09-07T06:53:47.2843033Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-09-07T06:53:47.2843571Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-09-07T06:53:47.2844048Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-09-07T06:53:47.2844539Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-09-07T06:53:47.2844969Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-09-07T06:53:47.2845369Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-09-07T06:53:47.2848649Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-09-07T06:53:47.2849256Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-09-07T06:53:47.2852742Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-09-07T06:53:47.2853250Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-09-07T06:53:47.2853795Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-09-07T06:53:47.2854249Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-09-07T06:53:47.2854929Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-09-07T06:53:47.2855354Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-09-07T06:53:47.2855810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-09-07T06:53:47.2856327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-09-07T06:53:47.2856806Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-09-07T06:53:47.2857264Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-09-07T06:53:47.2857828Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-09-07T06:53:47.2858338Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-09-07T06:53:47.2858836Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-09-07T06:53:47.2859356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-09-07T06:53:47.2859872Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-09-07T06:53:47.2874234Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-09-07T06:53:47.2929521Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-09-07T06:53:47.2931447Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-09-07T06:53:47.2931950Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-09-07T06:53:47.2932382Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-09-07T06:53:47.2932791Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-09-07T06:53:47.2933312Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-09-07T06:53:47.2938261Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json 2025-09-07T06:53:47.2941249Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-09-07T06:53:47.2946816Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-09-07T06:53:47.2947656Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-09-07T06:53:47.2968475Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-09-07T06:53:47.2969108Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-09-07T06:53:47.2969668Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-09-07T06:53:47.2970117Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-09-07T06:53:47.2970568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-09-07T06:53:47.2970984Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/ 2025-09-07T06:53:47.2971397Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-09-07T06:53:47.2971867Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-09-07T06:53:47.2972287Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-09-07T06:53:47.2972946Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-09-07T06:53:47.2973590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-09-07T06:53:47.2974461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-09-07T06:53:47.2975053Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-09-07T06:53:47.2975500Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-09-07T06:53:47.2976009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-09-07T06:53:47.2976606Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-09-07T06:53:47.2977380Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-09-07T06:53:47.2979126Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-09-07T06:53:47.2980447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-09-07T06:53:47.2980961Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-09-07T06:53:47.2981384Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-09-07T06:53:47.2981828Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-09-07T06:53:47.2982322Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-09-07T06:53:47.2982907Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-09-07T06:53:47.2983466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-09-07T06:53:47.2983992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-09-07T06:53:47.2984546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-09-07T06:53:47.2985109Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-09-07T06:53:47.2985667Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-09-07T06:53:47.2986317Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-09-07T06:53:47.2986891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-09-07T06:53:47.2988687Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-09-07T06:53:47.3098502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-09-07T06:53:47.3100635Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-09-07T06:53:47.3101354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-09-07T06:53:47.3106929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-09-07T06:53:47.3112610Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-09-07T06:53:47.3113377Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-09-07T06:53:47.3114573Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-09-07T06:53:47.3115248Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-09-07T06:53:47.3115933Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-09-07T06:53:47.3116773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-09-07T06:53:47.3117373Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-09-07T06:53:47.3118006Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-09-07T06:53:47.3169060Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-09-07T06:53:47.3172682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-09-07T06:53:47.3173259Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-09-07T06:53:47.3174026Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-09-07T06:53:47.3174459Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-09-07T06:53:47.3174924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-09-07T06:53:47.3175411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json 2025-09-07T06:53:47.3175856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-09-07T06:53:47.3176257Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-09-07T06:53:47.3176675Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-09-07T06:53:47.3263141Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-09-07T06:53:47.3298490Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-09-07T06:53:47.3299350Z creating: build/lib/ 2025-09-07T06:53:47.3374472Z inflating: build/lib/libprotobuf-lite.a 2025-09-07T06:53:47.3777141Z inflating: build/lib/libprotobuf.a 2025-09-07T06:53:47.4235983Z inflating: build/lib/libprotoc.a 2025-09-07T06:53:47.4247778Z inflating: build/lib/libpthreadpool.a 2025-09-07T06:53:47.4252868Z inflating: build/lib/libcpuinfo.a 2025-09-07T06:53:47.4261666Z inflating: build/lib/libcpuinfo_internals.a 2025-09-07T06:53:47.4261982Z inflating: build/lib/libclog.a 2025-09-07T06:53:47.4280350Z inflating: build/lib/libpytorch_qnnpack.a 2025-09-07T06:53:47.4286666Z inflating: build/lib/libnnpack_reference_layers.a 2025-09-07T06:53:47.4457430Z inflating: build/lib/libmicrokernels-prod.a 2025-09-07T06:53:47.4479277Z inflating: build/lib/libnnpack.a 2025-09-07T06:53:47.5301648Z inflating: build/lib/libmicrokernels-all.a 2025-09-07T06:53:47.5368410Z inflating: build/lib/libgtest.a 2025-09-07T06:53:47.5385571Z inflating: build/lib/libgmock.a 2025-09-07T06:53:47.5386666Z inflating: build/lib/libgmock_main.a 2025-09-07T06:53:47.5469343Z inflating: build/lib/libXNNPACK.a 2025-09-07T06:53:47.5469666Z inflating: build/lib/libgtest_main.a 2025-09-07T06:53:47.5539535Z inflating: build/lib/libbenchmark.a 2025-09-07T06:53:47.5539948Z inflating: build/lib/libbenchmark_main.a 2025-09-07T06:53:47.5540204Z inflating: build/lib/libjitprofiling.a 2025-09-07T06:53:47.5545268Z inflating: build/lib/libittnotify.a 2025-09-07T06:53:47.5610217Z inflating: build/lib/libasmjit.a 2025-09-07T06:53:47.6670367Z inflating: build/lib/libfbgemm.a 2025-09-07T06:53:47.6698201Z inflating: build/lib/libtensorpipe_uv.a 2025-09-07T06:53:47.7200106Z inflating: build/lib/libtensorpipe.a 2025-09-07T06:53:47.7313403Z inflating: build/lib/libgloo.a 2025-09-07T06:53:47.7356025Z inflating: build/lib/libonnx_proto.a 2025-09-07T06:53:47.8016691Z inflating: build/lib/libonnx.a 2025-09-07T06:53:48.7311237Z inflating: build/lib/libdnnl.a 2025-09-07T06:53:48.7331425Z inflating: build/lib/libfmt.a 2025-09-07T06:53:48.7577205Z inflating: build/lib/libkineto.a 2025-09-07T06:53:48.7682815Z inflating: build/lib/libc10.so 2025-09-07T06:53:48.7683475Z inflating: build/lib/libtorch_global_deps.so 2025-09-07T06:53:51.5785880Z inflating: build/lib/libtorch_cpu.so 2025-09-07T06:53:51.5786234Z inflating: build/lib/libtorch.so 2025-09-07T06:53:51.5855430Z inflating: build/lib/libtorchbind_test.so 2025-09-07T06:53:51.5872524Z inflating: build/lib/libjitbackend_test.so 2025-09-07T06:53:51.5895049Z inflating: build/lib/libbackend_with_compiler.so 2025-09-07T06:53:51.5920847Z inflating: build/lib/libaoti_custom_ops.so 2025-09-07T06:53:51.5926981Z inflating: build/lib/libshm.so 2025-09-07T06:53:51.7869150Z inflating: build/lib/libtorch_python.so 2025-09-07T06:53:51.7905262Z inflating: build/lib/libnnapi_backend.so 2025-09-07T06:53:51.7906147Z creating: build/bin/ 2025-09-07T06:53:51.7906360Z creating: build/bin/CMakeFiles/ 2025-09-07T06:53:51.7906614Z inflating: build/bin/cmake_install.cmake 2025-09-07T06:53:51.7906880Z inflating: build/bin/CTestTestfile.cmake 2025-09-07T06:53:51.8335488Z inflating: build/bin/protoc-3.13.0.0 2025-09-07T06:53:51.8758190Z inflating: build/bin/protoc 2025-09-07T06:53:51.8811921Z inflating: build/bin/c10_AllocatorConfig_test 2025-09-07T06:53:51.8864926Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-09-07T06:53:51.8918143Z inflating: build/bin/c10_DeviceGuard_test 2025-09-07T06:53:51.8971330Z inflating: build/bin/c10_Device_test 2025-09-07T06:53:51.9034055Z inflating: build/bin/c10_DispatchKeySet_test 2025-09-07T06:53:51.9084146Z inflating: build/bin/c10_StreamGuard_test 2025-09-07T06:53:51.9140929Z inflating: build/bin/c10_SymInt_test 2025-09-07T06:53:51.9197703Z inflating: build/bin/c10_Scalar_test 2025-09-07T06:53:51.9253651Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-09-07T06:53:51.9306183Z inflating: build/bin/c10_InlineStreamGuard_test 2025-09-07T06:53:51.9366562Z inflating: build/bin/c10_SizesAndStrides_test 2025-09-07T06:53:51.9418325Z inflating: build/bin/c10_Bitset_test 2025-09-07T06:53:51.9491635Z inflating: build/bin/c10_cow_test 2025-09-07T06:53:51.9540691Z inflating: build/bin/c10_ConstexprCrc_test 2025-09-07T06:53:51.9592668Z inflating: build/bin/c10_ArrayRef_test 2025-09-07T06:53:51.9645262Z inflating: build/bin/c10_DeadlockDetection_test 2025-09-07T06:53:51.9700920Z inflating: build/bin/c10_Enumerate_test 2025-09-07T06:53:51.9760408Z inflating: build/bin/c10_IntrusiveList_test 2025-09-07T06:53:51.9813566Z inflating: build/bin/c10_LeftRight_test 2025-09-07T06:53:51.9865630Z inflating: build/bin/c10_Half_test 2025-09-07T06:53:51.9920929Z inflating: build/bin/c10_Metaprogramming_test 2025-09-07T06:53:51.9975335Z inflating: build/bin/c10_NetworkFlow_test 2025-09-07T06:53:52.0027449Z inflating: build/bin/c10_Semaphore_test 2025-09-07T06:53:52.0088635Z inflating: build/bin/c10_ThreadLocal_test 2025-09-07T06:53:52.0136936Z inflating: build/bin/c10_TypeList_test 2025-09-07T06:53:52.0191258Z inflating: build/bin/c10_Synchronized_test 2025-09-07T06:53:52.0249962Z inflating: build/bin/c10_TypeIndex_test 2025-09-07T06:53:52.0300091Z inflating: build/bin/c10_TypeTraits_test 2025-09-07T06:53:52.0351779Z inflating: build/bin/c10_accumulate_test 2025-09-07T06:53:52.0404046Z inflating: build/bin/c10_bit_cast_test 2025-09-07T06:53:52.0463317Z inflating: build/bin/c10_bfloat16_test 2025-09-07T06:53:52.0524720Z inflating: build/bin/c10_complex_math_test 2025-09-07T06:53:52.0574900Z inflating: build/bin/c10_error_test 2025-09-07T06:53:52.0633091Z inflating: build/bin/c10_complex_test 2025-09-07T06:53:52.0687038Z inflating: build/bin/c10_exception_test 2025-09-07T06:53:52.0736200Z inflating: build/bin/c10_flags_test 2025-09-07T06:53:52.0785483Z inflating: build/bin/c10_generic_math_test 2025-09-07T06:53:52.0841969Z inflating: build/bin/c10_lazy_test 2025-09-07T06:53:52.0998873Z inflating: build/bin/c10_intrusive_ptr_test 2025-09-07T06:53:52.1054518Z inflating: build/bin/c10_irange_test 2025-09-07T06:53:52.1110749Z inflating: build/bin/c10_logging_test 2025-09-07T06:53:52.1186244Z inflating: build/bin/c10_optional_test 2025-09-07T06:53:52.1242394Z inflating: build/bin/c10_registry_test 2025-09-07T06:53:52.1307377Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-09-07T06:53:52.1456144Z inflating: build/bin/c10_small_vector_test 2025-09-07T06:53:52.1521257Z inflating: build/bin/c10_string_util_test 2025-09-07T06:53:52.1572778Z inflating: build/bin/c10_ssize_test 2025-09-07T06:53:52.1624013Z inflating: build/bin/c10_tempfile_test 2025-09-07T06:53:52.1679451Z inflating: build/bin/c10_string_view_test 2025-09-07T06:53:52.1724681Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-09-07T06:53:52.1783389Z inflating: build/bin/c10_typeid_test 2025-09-07T06:53:52.2342920Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-09-07T06:53:52.2920686Z inflating: build/bin/vec_test_all_types_AVX512 2025-09-07T06:53:52.3500462Z inflating: build/bin/vec_test_all_types_AVX2 2025-09-07T06:53:52.3555871Z inflating: build/bin/static_runtime_bench 2025-09-07T06:53:52.3801147Z inflating: build/bin/static_runtime_test 2025-09-07T06:53:52.3881882Z inflating: build/bin/Dict_test 2025-09-07T06:53:52.3935895Z inflating: build/bin/Dimname_test 2025-09-07T06:53:52.4002471Z inflating: build/bin/MaybeOwned_test 2025-09-07T06:53:52.4058108Z inflating: build/bin/NamedTensor_test 2025-09-07T06:53:52.4124918Z inflating: build/bin/apply_utils_test 2025-09-07T06:53:52.4182949Z inflating: build/bin/atest 2025-09-07T06:53:52.4253847Z inflating: build/bin/basic 2025-09-07T06:53:52.4306167Z inflating: build/bin/broadcast_test 2025-09-07T06:53:52.4359716Z inflating: build/bin/cpu_allocator_test 2025-09-07T06:53:52.4420846Z inflating: build/bin/cpu_generator_test 2025-09-07T06:53:52.4473002Z inflating: build/bin/cpu_profiling_allocator_test 2025-09-07T06:53:52.4567603Z inflating: build/bin/cpu_rng_test 2025-09-07T06:53:52.4618321Z inflating: build/bin/dlconvertor_test 2025-09-07T06:53:52.4681072Z inflating: build/bin/extension_backend_test 2025-09-07T06:53:52.4738666Z inflating: build/bin/half_test 2025-09-07T06:53:52.4835140Z inflating: build/bin/ivalue_test 2025-09-07T06:53:52.4887944Z inflating: build/bin/lazy_tensor_test 2025-09-07T06:53:52.4943400Z inflating: build/bin/math_kernel_test 2025-09-07T06:53:52.4997335Z inflating: build/bin/memory_format_test 2025-09-07T06:53:52.5053021Z inflating: build/bin/memory_overlapping_test 2025-09-07T06:53:52.5104345Z inflating: build/bin/mobile_memory_cleanup 2025-09-07T06:53:52.5164720Z inflating: build/bin/native_test 2025-09-07T06:53:52.5219113Z inflating: build/bin/operator_name_test 2025-09-07T06:53:52.5268941Z inflating: build/bin/operators_test 2025-09-07T06:53:52.5327718Z inflating: build/bin/packedtensoraccessor_test 2025-09-07T06:53:52.5395045Z inflating: build/bin/pow_test 2025-09-07T06:53:52.5454806Z inflating: build/bin/quantized_test 2025-09-07T06:53:52.5507062Z inflating: build/bin/reduce_ops_test 2025-09-07T06:53:52.5558688Z inflating: build/bin/reportMemoryUsage_test 2025-09-07T06:53:52.5620058Z inflating: build/bin/scalar_tensor_test 2025-09-07T06:53:52.5682115Z inflating: build/bin/scalar_test 2025-09-07T06:53:52.5734778Z inflating: build/bin/StorageUtils_test 2025-09-07T06:53:52.5792244Z inflating: build/bin/stride_properties_test 2025-09-07T06:53:52.5872151Z inflating: build/bin/tensor_iterator_test 2025-09-07T06:53:52.5926820Z inflating: build/bin/test_parallel 2025-09-07T06:53:52.5981113Z inflating: build/bin/thread_init_test 2025-09-07T06:53:52.6043084Z inflating: build/bin/type_ptr_test 2025-09-07T06:53:52.6103403Z inflating: build/bin/type_test 2025-09-07T06:53:52.6159683Z inflating: build/bin/undefined_tensor_test 2025-09-07T06:53:52.6210639Z inflating: build/bin/verify_api_visibility 2025-09-07T06:53:52.6285144Z inflating: build/bin/legacy_vmap_test 2025-09-07T06:53:52.6339924Z inflating: build/bin/weakref_test 2025-09-07T06:53:52.6392541Z inflating: build/bin/wrapdim_test 2025-09-07T06:53:52.6450621Z inflating: build/bin/xla_tensor_test 2025-09-07T06:53:52.6510836Z inflating: build/bin/IListRef_test 2025-09-07T06:53:52.6613096Z inflating: build/bin/List_test 2025-09-07T06:53:52.6681301Z inflating: build/bin/KernelFunction_test 2025-09-07T06:53:52.6798477Z inflating: build/bin/kernel_function_legacy_test 2025-09-07T06:53:52.6893537Z inflating: build/bin/kernel_function_test 2025-09-07T06:53:52.7020350Z inflating: build/bin/kernel_lambda_legacy_test 2025-09-07T06:53:52.7121770Z inflating: build/bin/kernel_lambda_test 2025-09-07T06:53:52.7181689Z inflating: build/bin/kernel_stackbased_test 2025-09-07T06:53:52.7277456Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-09-07T06:53:52.7330518Z inflating: build/bin/CppSignature_test 2025-09-07T06:53:52.7386609Z inflating: build/bin/backend_fallback_test 2025-09-07T06:53:52.7438686Z inflating: build/bin/op_allowlist_test 2025-09-07T06:53:52.7740116Z inflating: build/bin/op_registration_test 2025-09-07T06:53:52.7815512Z inflating: build/bin/inline_container_test 2025-09-07T06:53:52.8866978Z inflating: build/bin/test_jit 2025-09-07T06:53:52.8925148Z inflating: build/bin/BackoffTest 2025-09-07T06:53:52.8983049Z inflating: build/bin/FileStoreTest 2025-09-07T06:53:52.9041454Z inflating: build/bin/TCPStoreTest 2025-09-07T06:53:52.9397714Z inflating: build/bin/test_nativert 2025-09-07T06:53:52.9452820Z inflating: build/bin/HashStoreTest 2025-09-07T06:53:52.9517553Z inflating: build/bin/ProcessGroupGlooTest 2025-09-07T06:53:52.9517873Z inflating: build/bin/example_allreduce 2025-09-07T06:53:52.9575283Z inflating: build/bin/test_dist_autograd 2025-09-07T06:53:52.9642244Z inflating: build/bin/test_cpp_rpc 2025-09-07T06:53:53.0720076Z inflating: build/bin/test_api 2025-09-07T06:53:53.0720687Z inflating: build/bin/parallel_benchmark 2025-09-07T06:53:53.1046276Z inflating: build/bin/test_lazy 2025-09-07T06:53:53.1048130Z inflating: build/bin/torch_shm_manager 2025-09-07T06:53:53.1048571Z creating: .additional_ci_files/ 2025-09-07T06:53:53.1131773Z inflating: .additional_ci_files/test-times.json 2025-09-07T06:53:53.1451139Z inflating: .additional_ci_files/test-class-times.json 2025-09-07T06:53:53.1552035Z ##[group]Run rm artifacts.zip 2025-09-07T06:53:53.1552275Z rm artifacts.zip 2025-09-07T06:53:53.1557136Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:53.1557403Z env: 2025-09-07T06:53:53.1557562Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:53.1557743Z ##[endgroup] 2025-09-07T06:53:53.1991179Z ##[group]Run df -H 2025-09-07T06:53:53.1991390Z df -H 2025-09-07T06:53:53.1996016Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:53.1996255Z env: 2025-09-07T06:53:53.1996437Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:53.1996678Z ##[endgroup] 2025-09-07T06:53:53.2038737Z Filesystem Size Used Avail Use% Mounted on 2025-09-07T06:53:53.2039087Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-09-07T06:53:53.2039332Z tmpfs 67G 0 67G 0% /dev/shm 2025-09-07T06:53:53.2039556Z tmpfs 27G 791k 27G 1% /run 2025-09-07T06:53:53.2039766Z /dev/nvme0n1p1 215G 70G 146G 33% / 2025-09-07T06:53:53.2039981Z tmpfs 67G 13k 67G 1% /tmp 2025-09-07T06:53:53.2040205Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-09-07T06:53:53.2070951Z Prepare all required actions 2025-09-07T06:53:53.2072000Z Getting action download info 2025-09-07T06:53:53.3464444Z ##[group]Run ./.github/actions/download-td-artifacts 2025-09-07T06:53:53.3464720Z with: 2025-09-07T06:53:53.3464885Z env: 2025-09-07T06:53:53.3465057Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:53.3465252Z ##[endgroup] 2025-09-07T06:53:53.3613421Z ##[group]Run seemethere/download-artifact-s3@v4 2025-09-07T06:53:53.3613664Z with: 2025-09-07T06:53:53.3613809Z name: td_results 2025-09-07T06:53:53.3613984Z s3-bucket: gha-artifacts 2025-09-07T06:53:53.3614169Z region: us-east-1 2025-09-07T06:53:53.3614325Z env: 2025-09-07T06:53:53.3614511Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:53.3614690Z ##[endgroup] 2025-09-07T06:53:53.7094598Z (node:48184) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-09-07T06:53:53.7099067Z 2025-09-07T06:53:53.7101480Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-09-07T06:53:53.7101869Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-09-07T06:53:53.7102487Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-09-07T06:53:53.7917917Z Found 0 objects with prefix pytorch/pytorch/17524754606/td_results/ 2025-09-07T06:53:53.7926469Z Artifact download has finished successfully 2025-09-07T06:53:53.8201100Z ##[group]Run mkdir -p .additional_ci_files 2025-09-07T06:53:53.8201375Z mkdir -p .additional_ci_files 2025-09-07T06:53:53.8201667Z mv td_results.json .additional_ci_files/td_results.json || true 2025-09-07T06:53:53.8206701Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:53.8206952Z env: 2025-09-07T06:53:53.8207118Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:53.8207299Z ##[endgroup] 2025-09-07T06:53:53.9023429Z mv: cannot stat 'td_results.json': No such file or directory 2025-09-07T06:53:53.9141433Z ##[group]Run .github/scripts/parse_ref.py 2025-09-07T06:53:53.9141707Z .github/scripts/parse_ref.py 2025-09-07T06:53:53.9147078Z shell: /usr/bin/bash -e {0} 2025-09-07T06:53:53.9147287Z env: 2025-09-07T06:53:53.9147467Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:53.9147646Z ##[endgroup] 2025-09-07T06:53:53.9360594Z Setting output branch=main 2025-09-07T06:53:53.9461918Z Prepare all required actions 2025-09-07T06:53:53.9462270Z Getting action download info 2025-09-07T06:53:54.0633525Z ##[group]Run ./.github/actions/filter-test-configs 2025-09-07T06:53:54.0633876Z with: 2025-09-07T06:53:54.0634322Z github-token: *** 2025-09-07T06:53:54.0636189Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-09-07T06:53:54.0638124Z job-name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:53:54.0638469Z env: 2025-09-07T06:53:54.0638635Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:54.0638829Z ##[endgroup] 2025-09-07T06:53:54.0692173Z ##[group]Run nick-fields/retry@v3.0.0 2025-09-07T06:53:54.0692449Z with: 2025-09-07T06:53:54.0692638Z shell: bash 2025-09-07T06:53:54.0692846Z timeout_minutes: 10 2025-09-07T06:53:54.0693064Z max_attempts: 5 2025-09-07T06:53:54.0693279Z retry_wait_seconds: 30 2025-09-07T06:53:54.0693905Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-09-07T06:53:54.0694564Z polling_interval_seconds: 1 2025-09-07T06:53:54.0694823Z warning_on_retry: true 2025-09-07T06:53:54.0695053Z continue_on_error: false 2025-09-07T06:53:54.0695273Z env: 2025-09-07T06:53:54.0695454Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:54.0695885Z GITHUB_TOKEN: *** 2025-09-07T06:53:54.0696086Z ##[endgroup] 2025-09-07T06:53:54.1960107Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-09-07T06:53:54.3797553Z Defaulting to user installation because normal site-packages is not writeable 2025-09-07T06:53:54.8258326Z Collecting requests==2.27.1 2025-09-07T06:53:54.8417605Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-09-07T06:53:55.0676128Z Collecting pyyaml==6.0.2 2025-09-07T06:53:55.0715129Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-09-07T06:53:55.4808473Z Collecting charset-normalizer~=2.0.0 2025-09-07T06:53:55.4853025Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-09-07T06:53:55.5593392Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-09-07T06:53:55.6502645Z Collecting certifi>=2017.4.17 2025-09-07T06:53:55.6536683Z Downloading certifi-2025.8.3-py3-none-any.whl (161 kB) 2025-09-07T06:53:55.6801018Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-09-07T06:53:55.7414083Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-09-07T06:53:55.8482716Z Successfully installed certifi-2025.8.3 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-09-07T06:53:56.1364385Z Command completed after 1 attempt(s). 2025-09-07T06:53:56.1424058Z ##[group]Run set -x 2025-09-07T06:53:56.1424282Z set -x 2025-09-07T06:53:56.1424455Z  2025-09-07T06:53:56.1424744Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-09-07T06:53:56.1425074Z # in runner workspace 2025-09-07T06:53:56.1425546Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-09-07T06:53:56.1430732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:56.1430978Z env: 2025-09-07T06:53:56.1431133Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:56.1431316Z ##[endgroup] 2025-09-07T06:53:56.1460455Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-09-07T06:53:56.1606680Z Setting output branch=main 2025-09-07T06:53:56.1657360Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-09-07T06:53:56.1657668Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-09-07T06:53:56.1657913Z echo "Job name: ${JOB_NAME}" 2025-09-07T06:53:56.1658124Z  2025-09-07T06:53:56.1658381Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-09-07T06:53:56.1658727Z # in runner workspace 2025-09-07T06:53:56.1659038Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-09-07T06:53:56.1659367Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-09-07T06:53:56.1659603Z  --job-name "${JOB_NAME}" \ 2025-09-07T06:53:56.1661391Z  --test-matrix "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" \ 2025-09-07T06:53:56.1663217Z  --selected-test-configs "" \ 2025-09-07T06:53:56.1663463Z  --pr-number "${PR_NUMBER}" \ 2025-09-07T06:53:56.1663683Z  --tag "${TAG}" \ 2025-09-07T06:53:56.1663900Z  --event-name "${EVENT_NAME}" \ 2025-09-07T06:53:56.1664132Z  --schedule "${SCHEDULE}" \ 2025-09-07T06:53:56.1664357Z  --branch "${HEAD_BRANCH}" 2025-09-07T06:53:56.1669697Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:56.1669953Z env: 2025-09-07T06:53:56.1670131Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:56.1670656Z GITHUB_TOKEN: *** 2025-09-07T06:53:56.1670994Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:53:56.1671472Z PR_NUMBER: 2025-09-07T06:53:56.1671638Z TAG: 2025-09-07T06:53:56.1671793Z EVENT_NAME: push 2025-09-07T06:53:56.1671967Z SCHEDULE: 2025-09-07T06:53:56.1672121Z HEAD_BRANCH: main 2025-09-07T06:53:56.1672306Z ##[endgroup] 2025-09-07T06:53:56.1694251Z Workflow: inductor 2025-09-07T06:53:56.1694623Z Job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:53:56.3502902Z Setting output keep-going=True 2025-09-07T06:53:56.3503224Z Setting output ci-verbose-test-logs=False 2025-09-07T06:53:56.3503487Z Setting output ci-test-showlocals=False 2025-09-07T06:53:56.3503732Z Setting output ci-no-test-timeout=False 2025-09-07T06:53:56.3503964Z Setting output ci-no-td=False 2025-09-07T06:53:56.3504182Z Setting output ci-td-distributed=False 2025-09-07T06:53:56.3504420Z Setting output is-unstable=False 2025-09-07T06:53:56.3504644Z Setting output reenabled-issues= 2025-09-07T06:53:56.3507141Z Setting output test-matrix={"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-09-07T06:53:56.3509053Z Setting output is-test-matrix-empty=False 2025-09-07T06:53:56.3647458Z ##[group]Run echo "Filtered matrix:" 2025-09-07T06:53:56.3647726Z echo "Filtered matrix:" 2025-09-07T06:53:56.3649461Z echo "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" 2025-09-07T06:53:56.3651202Z  2025-09-07T06:53:56.3651368Z echo 2025-09-07T06:53:56.3651579Z echo "Is the current job unstable? False" 2025-09-07T06:53:56.3651821Z  2025-09-07T06:53:56.3651980Z echo 2025-09-07T06:53:56.3652175Z echo "Is keep-going label set? True" 2025-09-07T06:53:56.3652396Z  2025-09-07T06:53:56.3652553Z echo 2025-09-07T06:53:56.3652732Z echo "Reenabled issues? " 2025-09-07T06:53:56.3657604Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:56.3657868Z env: 2025-09-07T06:53:56.3658042Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:56.3658237Z ##[endgroup] 2025-09-07T06:53:56.3681300Z Filtered matrix: 2025-09-07T06:53:56.3683267Z {include: [{config: cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: inductor_torchbench_cpu_smoketest_perf, shard: 1, num_shards: 1, runner: linux.24xl.spr-metal}]} 2025-09-07T06:53:56.3685135Z 2025-09-07T06:53:56.3685230Z Is the current job unstable? False 2025-09-07T06:53:56.3685373Z 2025-09-07T06:53:56.3685452Z Is keep-going label set? True 2025-09-07T06:53:56.3685587Z 2025-09-07T06:53:56.3685660Z Reenabled issues? 2025-09-07T06:53:56.3725586Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-09-07T06:53:56.3725949Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-09-07T06:53:56.3730492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:56.3730740Z env: 2025-09-07T06:53:56.3730930Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:56.3731157Z JOB_TIMEOUT: 240 2025-09-07T06:53:56.3731316Z ##[endgroup] 2025-09-07T06:53:56.3786653Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-09-07T06:53:56.3787047Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-09-07T06:53:56.3787335Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-09-07T06:53:56.3791507Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T06:53:56.3791754Z env: 2025-09-07T06:53:56.3791919Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:56.3792105Z ##[endgroup] 2025-09-07T06:53:56.3896216Z ##[group]Run set -x 2025-09-07T06:53:56.3896485Z set -x 2025-09-07T06:53:56.3896648Z  2025-09-07T06:53:56.3896846Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-09-07T06:53:56.3897129Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-09-07T06:53:56.3897406Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-09-07T06:53:56.3897672Z  TEST_COMMAND=.ci/onnx/test.sh 2025-09-07T06:53:56.3897876Z else 2025-09-07T06:53:56.3898055Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-09-07T06:53:56.3898259Z fi 2025-09-07T06:53:56.3898398Z  2025-09-07T06:53:56.3898579Z # Leaving 1GB for the runner and other things 2025-09-07T06:53:56.3898953Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-09-07T06:53:56.3899531Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-09-07T06:53:56.3899977Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-09-07T06:53:56.3900329Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-09-07T06:53:56.3900604Z  2025-09-07T06:53:56.3900800Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-09-07T06:53:56.3901032Z  SHM_OPTS= 2025-09-07T06:53:56.3901217Z  JENKINS_USER= 2025-09-07T06:53:56.3901468Z  # ensure that docker container cleanly exits in 12 hours 2025-09-07T06:53:56.3901788Z  # if for some reason cleanup action doesn't stop container 2025-09-07T06:53:56.3902055Z  # when job is cancelled 2025-09-07T06:53:56.3902274Z  DOCKER_SHELL_CMD="sleep 12h" 2025-09-07T06:53:56.3902487Z else 2025-09-07T06:53:56.3902687Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-09-07T06:53:56.3902931Z  JENKINS_USER="--user jenkins" 2025-09-07T06:53:56.3903153Z  DOCKER_SHELL_CMD= 2025-09-07T06:53:56.3903351Z fi 2025-09-07T06:53:56.3903511Z  2025-09-07T06:53:56.3903756Z # detached container should get cleaned up by teardown_ec2_linux 2025-09-07T06:53:56.3904113Z # TODO: Stop building test binaries as part of the build phase 2025-09-07T06:53:56.3904518Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-09-07T06:53:56.3904999Z # shellcheck disable=SC2086,SC2090 2025-09-07T06:53:56.3905247Z container_name=$(docker run \ 2025-09-07T06:53:56.3905477Z  ${GPU_FLAG:-} \ 2025-09-07T06:53:56.3905906Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-09-07T06:53:56.3906173Z  -e BUILD_ENVIRONMENT \ 2025-09-07T06:53:56.3906399Z  -e PR_NUMBER \ 2025-09-07T06:53:56.3906611Z  -e GITHUB_ACTIONS \ 2025-09-07T06:53:56.3906823Z  -e GITHUB_REPOSITORY \ 2025-09-07T06:53:56.3907048Z  -e GITHUB_WORKFLOW \ 2025-09-07T06:53:56.3907263Z  -e GITHUB_JOB \ 2025-09-07T06:53:56.3907464Z  -e GITHUB_RUN_ID \ 2025-09-07T06:53:56.3907664Z  -e GITHUB_RUN_NUMBER \ 2025-09-07T06:53:56.3907885Z  -e GITHUB_RUN_ATTEMPT \ 2025-09-07T06:53:56.3908104Z  -e JOB_ID \ 2025-09-07T06:53:56.3908297Z  -e JOB_NAME \ 2025-09-07T06:53:56.3908487Z  -e BASE_SHA \ 2025-09-07T06:53:56.3908677Z  -e BRANCH \ 2025-09-07T06:53:56.3908861Z  -e SHA1 \ 2025-09-07T06:53:56.3909049Z  -e AWS_DEFAULT_REGION \ 2025-09-07T06:53:56.3909256Z  -e IN_WHEEL_TEST \ 2025-09-07T06:53:56.3909459Z  -e SHARD_NUMBER \ 2025-09-07T06:53:56.3909660Z  -e TEST_CONFIG \ 2025-09-07T06:53:56.3909859Z  -e NUM_TEST_SHARDS \ 2025-09-07T06:53:56.3910068Z  -e REENABLED_ISSUES \ 2025-09-07T06:53:56.3910289Z  -e CONTINUE_THROUGH_ERROR \ 2025-09-07T06:53:56.3910595Z  -e VERBOSE_TEST_LOGS \ 2025-09-07T06:53:56.3910811Z  -e TEST_SHOWLOCALS \ 2025-09-07T06:53:56.3911009Z  -e NO_TEST_TIMEOUT \ 2025-09-07T06:53:56.3911207Z  -e NO_TD \ 2025-09-07T06:53:56.3911399Z  -e TD_DISTRIBUTED \ 2025-09-07T06:53:56.3911606Z  -e PR_LABELS \ 2025-09-07T06:53:56.3911829Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-09-07T06:53:56.3912069Z  -e SCCACHE_BUCKET \ 2025-09-07T06:53:56.3912273Z  -e SCCACHE_REGION \ 2025-09-07T06:53:56.3912473Z  -e XLA_CUDA \ 2025-09-07T06:53:56.3912688Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-09-07T06:53:56.3912943Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-09-07T06:53:56.3913208Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-09-07T06:53:56.3913468Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-09-07T06:53:56.3913714Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-09-07T06:53:56.3913953Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-09-07T06:53:56.3914201Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-09-07T06:53:56.3914430Z  -e DASHBOARD_TAG \ 2025-09-07T06:53:56.3914652Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-09-07T06:53:56.3914900Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-09-07T06:53:56.3915185Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-09-07T06:53:56.3915477Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-09-07T06:53:56.3915749Z  --security-opt seccomp=unconfined \ 2025-09-07T06:53:56.3915990Z  --cap-add=SYS_PTRACE \ 2025-09-07T06:53:56.3916191Z  --ipc=host \ 2025-09-07T06:53:56.3916378Z  ${SHM_OPTS} \ 2025-09-07T06:53:56.3916559Z  --tty \ 2025-09-07T06:53:56.3916731Z  --detach \ 2025-09-07T06:53:56.3916919Z  --name="${container_name}" \ 2025-09-07T06:53:56.3917137Z  ${JENKINS_USER} \ 2025-09-07T06:53:56.3917389Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-09-07T06:53:56.3917661Z  -w /var/lib/jenkins/workspace \ 2025-09-07T06:53:56.3917876Z  "${DOCKER_IMAGE}" \ 2025-09-07T06:53:56.3918079Z  ${DOCKER_SHELL_CMD} 2025-09-07T06:53:56.3918267Z ) 2025-09-07T06:53:56.3918480Z # Propagate download.pytorch.org IP to container 2025-09-07T06:53:56.3918963Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2025-09-07T06:53:56.3919431Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-09-07T06:53:56.3919945Z  2025-09-07T06:53:56.3920148Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-09-07T06:53:56.3920538Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-09-07T06:53:56.3920872Z fi 2025-09-07T06:53:56.3921032Z  2025-09-07T06:53:56.3921368Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-09-07T06:53:56.3926162Z shell: /usr/bin/bash -e {0} 2025-09-07T06:53:56.3926346Z env: 2025-09-07T06:53:56.3926498Z GIT_DEFAULT_BRANCH: main 2025-09-07T06:53:56.3926720Z BUILD_ENVIRONMENT: linux-jammy-py3.9-gcc11-build 2025-09-07T06:53:56.3926951Z PR_NUMBER: 2025-09-07T06:53:56.3927122Z GITHUB_REPOSITORY: pytorch/pytorch 2025-09-07T06:53:56.3927321Z GITHUB_WORKFLOW: inductor 2025-09-07T06:53:56.3927498Z GITHUB_JOB: test 2025-09-07T06:53:56.3927667Z GITHUB_RUN_ID: 17524754606 2025-09-07T06:53:56.3927852Z GITHUB_RUN_NUMBER: 152138 2025-09-07T06:53:56.3928029Z GITHUB_RUN_ATTEMPT: 1 2025-09-07T06:53:56.3928200Z JOB_ID: 49774397867 2025-09-07T06:53:56.3928506Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:53:56.3928830Z BRANCH: main 2025-09-07T06:53:56.3929171Z SHA1: 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:53:56.3929439Z BASE_SHA: 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:53:56.3929685Z TEST_CONFIG: dynamic_cpu_inductor_huggingface 2025-09-07T06:53:56.3929903Z SHARD_NUMBER: 1 2025-09-07T06:53:56.3930056Z NUM_TEST_SHARDS: 1 2025-09-07T06:53:56.3930224Z REENABLED_ISSUES: 2025-09-07T06:53:56.3930401Z CONTINUE_THROUGH_ERROR: True 2025-09-07T06:53:56.3930597Z VERBOSE_TEST_LOGS: False 2025-09-07T06:53:56.3930772Z TEST_SHOWLOCALS: False 2025-09-07T06:53:56.3930957Z NO_TEST_TIMEOUT: False 2025-09-07T06:53:56.3931127Z NO_TD: False 2025-09-07T06:53:56.3931286Z TD_DISTRIBUTED: False 2025-09-07T06:53:56.3931495Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-09-07T06:53:56.3931737Z SCCACHE_REGION: us-east-1 2025-09-07T06:53:56.3931916Z SHM_SIZE: 1g 2025-09-07T06:53:56.3932435Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:53:56.3932972Z XLA_CUDA: 2025-09-07T06:53:56.3933213Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-09-07T06:53:56.3933499Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-09-07T06:53:56.3933714Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-09-07T06:53:56.3933931Z DASHBOARD_TAG: 2025-09-07T06:53:56.3934284Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-09-07T06:53:56.3934551Z HUGGING_FACE_HUB_TOKEN: *** 2025-09-07T06:53:56.3934823Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-09-07T06:53:56.3935157Z ARTIFACTS_FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T06:53:56.3935486Z ##[endgroup] 2025-09-07T06:53:56.3957328Z + [[ dynamic_cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-09-07T06:53:56.3957820Z + [[ linux-jammy-py3.9-gcc11-build == *onnx* ]] 2025-09-07T06:53:56.3958206Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-09-07T06:53:56.3962891Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-09-07T06:53:56.3982653Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-09-07T06:53:56.3982949Z + TOTAL_MEMORY_WITH_SWAP=125 2025-09-07T06:53:56.3983572Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-09-07T06:53:56.3983892Z + SHM_OPTS=--shm-size=1g 2025-09-07T06:53:56.3984112Z + JENKINS_USER='--user jenkins' 2025-09-07T06:53:56.3984320Z + DOCKER_SHELL_CMD= 2025-09-07T06:53:56.3987959Z +++ nproc --ignore=2 2025-09-07T06:53:56.4012804Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_17524754606 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T06:54:06.9049497Z + container_name=9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T06:54:06.9053858Z + grep download.pytorch.org /etc/hosts 2025-09-07T06:54:06.9054344Z + docker exec -i 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e sudo bash -c '/bin/cat >> /etc/hosts' 2025-09-07T06:54:06.9996636Z + echo DOCKER_CONTAINER_ID=9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T06:54:06.9997174Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-09-07T06:54:07.0000672Z ++ echo dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl 2025-09-07T06:54:07.0003514Z + docker exec -t 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e sh -c 'python3 -m pip install dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-09-07T06:54:07.3620052Z Processing ./dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl (from torch==2.9.0a0+git93fb23d) 2025-09-07T06:54:07.5851330Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.19.1) 2025-09-07T06:54:07.5858073Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (4.15.0) 2025-09-07T06:54:07.5859469Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (1.13.3) 2025-09-07T06:54:07.5860401Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (2.8.8) 2025-09-07T06:54:07.5862181Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.1.6) 2025-09-07T06:54:07.5865482Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (2025.3.0) 2025-09-07T06:54:07.5878998Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.3.0) 2025-09-07T06:54:07.6188131Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from opt-einsum>=3.3->torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (1.22.4) 2025-09-07T06:54:07.6196936Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (1.3.0) 2025-09-07T06:54:07.6243456Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from jinja2->torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.0.2) 2025-09-07T06:54:08.4126111Z Installing collected packages: torch 2025-09-07T06:54:15.7951591Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-09-07T06:54:15.7952249Z dall-e 0.1 requires torchvision, which is not installed. 2025-09-07T06:54:15.7952708Z effdet 0.4.1 requires torchvision, which is not installed. 2025-09-07T06:54:15.7953270Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-09-07T06:54:15.7953877Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-09-07T06:54:15.7954512Z timm 1.0.14 requires torchvision, which is not installed. 2025-09-07T06:54:15.7954967Z Successfully installed torch-2.9.0a0+git93fb23d 2025-09-07T06:54:15.8982438Z + export TERM=vt100 2025-09-07T06:54:15.8982696Z + TERM=vt100 2025-09-07T06:54:15.8982885Z ++ dirname .ci/pytorch/test.sh 2025-09-07T06:54:15.8984652Z + source .ci/pytorch/common.sh 2025-09-07T06:54:15.8987591Z +++ dirname .ci/pytorch/common.sh 2025-09-07T06:54:15.9000737Z ++ source .ci/pytorch/common_utils.sh 2025-09-07T06:54:15.9001076Z +++ declare -f -t trap_add 2025-09-07T06:54:15.9010467Z ++ set -ex -o pipefail 2025-09-07T06:54:15.9010727Z ++ [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-09-07T06:54:15.9011454Z ++ BUILD_TEST_LIBTORCH=0 2025-09-07T06:54:15.9011670Z ++ dirname .ci/pytorch/test.sh 2025-09-07T06:54:15.9018414Z + source .ci/pytorch/common-build.sh 2025-09-07T06:54:15.9019212Z ++ [[ linux-jammy-py3.9-gcc11-build != *win-* ]] 2025-09-07T06:54:15.9019983Z ++++ dirname .ci/pytorch/common-build.sh 2025-09-07T06:54:15.9033692Z +++ cd .ci/pytorch 2025-09-07T06:54:15.9036544Z +++ pwd -P 2025-09-07T06:54:15.9036969Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-09-07T06:54:15.9037475Z ++ [[ linux-jammy-py3.9-gcc11-build == *-pch* ]] 2025-09-07T06:54:15.9037848Z ++ which sccache 2025-09-07T06:54:15.9060763Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-09-07T06:54:15.9061206Z ++ sccache --stop-server 2025-09-07T06:54:15.9085230Z ++ true 2025-09-07T06:54:15.9090116Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-09-07T06:54:15.9093866Z ++ trap_add sccache_epilogue EXIT 2025-09-07T06:54:15.9094143Z ++ trap_add_cmd=sccache_epilogue 2025-09-07T06:54:15.9094350Z ++ shift 2025-09-07T06:54:15.9094559Z ++ for trap_add_name in "$@" 2025-09-07T06:54:15.9102269Z ++++ trap -p EXIT 2025-09-07T06:54:15.9102529Z +++ eval 'extract_trap_cmd ' 2025-09-07T06:54:15.9102766Z ++++ extract_trap_cmd 2025-09-07T06:54:15.9102957Z ++++ printf '%s\n' '' 2025-09-07T06:54:15.9103159Z +++ printf '%s\n' sccache_epilogue 2025-09-07T06:54:15.9103373Z ++ trap -- ' 2025-09-07T06:54:15.9103554Z sccache_epilogue' EXIT 2025-09-07T06:54:15.9103774Z ++ [[ -n 1 ]] 2025-09-07T06:54:15.9104078Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-09-07T06:54:15.9104476Z Skipping sccache server initialization, setting environment variables 2025-09-07T06:54:15.9104782Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-09-07T06:54:15.9104997Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-09-07T06:54:15.9105257Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-09-07T06:54:15.9105580Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-09-07T06:54:15.9106010Z ++ export RUST_LOG=sccache::server=error 2025-09-07T06:54:15.9106271Z ++ RUST_LOG=sccache::server=error 2025-09-07T06:54:15.9106493Z ++ sccache --zero-stats 2025-09-07T06:54:16.0478618Z Statistics zeroed. 2025-09-07T06:54:16.0484056Z ++ which ccache 2025-09-07T06:54:16.0511341Z + [[ linux-jammy-py3.9-gcc11-build != *rocm* ]] 2025-09-07T06:54:16.0511699Z + [[ linux-jammy-py3.9-gcc11-build != *s390x* ]] 2025-09-07T06:54:16.0511964Z + [[ -d /var/lib/jenkins/workspace ]] 2025-09-07T06:54:16.0514780Z ++ stat -c %u /var/lib/jenkins/workspace 2025-09-07T06:54:16.0525345Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-09-07T06:54:16.0525612Z + trap_add cleanup_workspace EXIT 2025-09-07T06:54:16.0525854Z + trap_add_cmd=cleanup_workspace 2025-09-07T06:54:16.0526061Z + shift 2025-09-07T06:54:16.0526235Z + for trap_add_name in "$@" 2025-09-07T06:54:16.0540372Z +++ trap -p EXIT 2025-09-07T06:54:16.0540635Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-09-07T06:54:16.0541271Z sccache_epilogue'\'' EXIT' 2025-09-07T06:54:16.0541534Z +++ extract_trap_cmd trap -- ' 2025-09-07T06:54:16.0541788Z sccache_epilogue' EXIT 2025-09-07T06:54:16.0541985Z +++ printf '%s\n' ' 2025-09-07T06:54:16.0542169Z sccache_epilogue' 2025-09-07T06:54:16.0542359Z ++ printf '%s\n' cleanup_workspace 2025-09-07T06:54:16.0542627Z + trap -- ' 2025-09-07T06:54:16.0542816Z sccache_epilogue 2025-09-07T06:54:16.0543151Z cleanup_workspace' EXIT 2025-09-07T06:54:16.0543429Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-09-07T06:54:16.4873960Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-09-07T06:54:16.4885649Z + echo 'Environment variables:' 2025-09-07T06:54:16.4885921Z Environment variables: 2025-09-07T06:54:16.4886151Z + env 2025-09-07T06:54:16.4900972Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-09-07T06:54:16.4901368Z CONTINUE_THROUGH_ERROR=True 2025-09-07T06:54:16.4901646Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-09-07T06:54:16.4902156Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-09-07T06:54:16.4902386Z HOSTNAME=9c09efa4294e 2025-09-07T06:54:16.4903120Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.4903556Z GITHUB_ACTION=__run_2 2025-09-07T06:54:16.4903765Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-09-07T06:54:16.4903994Z GITHUB_RUN_NUMBER=152138 2025-09-07T06:54:16.4904278Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-09-07T06:54:16.4904536Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-09-07T06:54:16.4904791Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-09-07T06:54:16.4905024Z SCCACHE_IDLE_TIMEOUT=0 2025-09-07T06:54:16.4905316Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-09-07T06:54:16.4905546Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-09-07T06:54:16.4905981Z GITHUB_REF_TYPE=branch 2025-09-07T06:54:16.4906205Z BASE_SHA=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.4906448Z XLA_CUDA= 2025-09-07T06:54:16.4906628Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-09-07T06:54:16.4906967Z HUGGING_FACE_HUB_TOKEN=*** 2025-09-07T06:54:16.4912953Z *** 2025-09-07T06:54:16.4913186Z GITHUB_REPOSITORY_ID=65600975 2025-09-07T06:54:16.4913414Z GITHUB_ACTIONS=true 2025-09-07T06:54:16.4913644Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-09-07T06:54:16.4913928Z SHA1=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.4914186Z GITHUB_SHA=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.4914541Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-09-07T06:54:16.4914868Z UCC_HOME=/usr 2025-09-07T06:54:16.4915039Z VERBOSE_TEST_LOGS=False 2025-09-07T06:54:16.4915226Z GITHUB_REF=refs/heads/main 2025-09-07T06:54:16.4915414Z SHARD_NUMBER=1 2025-09-07T06:54:16.4915592Z GITHUB_REF_PROTECTED=true 2025-09-07T06:54:16.4915784Z HOME=/var/lib/jenkins 2025-09-07T06:54:16.4915988Z GITHUB_API_URL=https://api.github.com 2025-09-07T06:54:16.4916231Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-09-07T06:54:16.4916441Z UCX_COMMIT= 2025-09-07T06:54:16.4916594Z USE_SYSTEM_NCCL=1 2025-09-07T06:54:16.4916766Z NUM_TEST_SHARDS=1 2025-09-07T06:54:16.4916933Z UCX_HOME=/usr 2025-09-07T06:54:16.4917322Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.4917910Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:54:16.4918454Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.4919157Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-09-07T06:54:16.4919470Z GITHUB_EVENT_NAME=push 2025-09-07T06:54:16.4919809Z DASHBOARD_TAG= 2025-09-07T06:54:16.4919972Z GITHUB_RUN_ID=17524754606 2025-09-07T06:54:16.4920157Z INSTALLED_OPENBLAS= 2025-09-07T06:54:16.4920540Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.4920956Z GITHUB_ACTOR=pytorchmergebot 2025-09-07T06:54:16.4921138Z PR_NUMBER= 2025-09-07T06:54:16.4921295Z DESIRED_CUDA= 2025-09-07T06:54:16.4921455Z GITHUB_RUN_ATTEMPT=1 2025-09-07T06:54:16.4921636Z ANACONDA_PYTHON_VERSION=3.9 2025-09-07T06:54:16.4921857Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-09-07T06:54:16.4922081Z TERM=vt100 2025-09-07T06:54:16.4922232Z INSTALLED_VISION=yes 2025-09-07T06:54:16.4922394Z BRANCH=main 2025-09-07T06:54:16.4922546Z SCCACHE_REGION=us-east-1 2025-09-07T06:54:16.4922739Z OPENSSL_ROOT_DIR=/opt/openssl 2025-09-07T06:54:16.4922931Z CUDA_PATH=/usr/local/cuda 2025-09-07T06:54:16.4923258Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-09-07T06:54:16.4923612Z GITHUB_SERVER_URL=https://github.com 2025-09-07T06:54:16.4923815Z UCC_COMMIT= 2025-09-07T06:54:16.4923965Z REENABLED_ISSUES= 2025-09-07T06:54:16.4924122Z DOCS=yes 2025-09-07T06:54:16.4924258Z SHLVL=1 2025-09-07T06:54:16.4924397Z MAX_JOBS=30 2025-09-07T06:54:16.4924548Z GITHUB_ACTOR_ID=97764156 2025-09-07T06:54:16.4924861Z GITHUB_WORKFLOW_SHA=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.4925104Z GITHUB_REF_NAME=main 2025-09-07T06:54:16.4925360Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-09-07T06:54:16.4925634Z GITHUB_JOB=test 2025-09-07T06:54:16.4925798Z NO_TEST_TIMEOUT=False 2025-09-07T06:54:16.4925960Z TD_DISTRIBUTED=False 2025-09-07T06:54:16.4926143Z GITHUB_REPOSITORY=pytorch/pytorch 2025-09-07T06:54:16.4926351Z GITHUB_RETENTION_DAYS=90 2025-09-07T06:54:16.4926530Z OPENSSL_DIR=/opt/openssl 2025-09-07T06:54:16.4926704Z GITHUB_ACTION_REPOSITORY= 2025-09-07T06:54:16.4927181Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-09-07T06:54:16.4927647Z GITHUB_BASE_REF= 2025-09-07T06:54:16.4927808Z INSTALLED_ACL= 2025-09-07T06:54:16.4928100Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T06:54:16.4928435Z CI=true 2025-09-07T06:54:16.4928600Z GITHUB_REPOSITORY_OWNER=pytorch 2025-09-07T06:54:16.4928850Z RUST_LOG=sccache::server=error 2025-09-07T06:54:16.4929027Z JOB_ID=49774397867 2025-09-07T06:54:16.4929188Z GITHUB_HEAD_REF= 2025-09-07T06:54:16.4929348Z GITHUB_ACTION_REF= 2025-09-07T06:54:16.4929548Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-09-07T06:54:16.4929768Z TEST_SHOWLOCALS=False 2025-09-07T06:54:16.4929943Z GITHUB_WORKFLOW=inductor 2025-09-07T06:54:16.4930125Z DEBIAN_FRONTEND=noninteractive 2025-09-07T06:54:16.4930506Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.4930894Z NO_TD=False 2025-09-07T06:54:16.4931062Z SKIP_SCCACHE_INITIALIZATION=1 2025-09-07T06:54:16.4931272Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-09-07T06:54:16.4931477Z _=/usr/bin/env 2025-09-07T06:54:16.4931690Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-09-07T06:54:16.5173647Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch 2025-09-07T06:54:16.5174223Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2025-09-07T06:54:16.5178420Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib 2025-09-07T06:54:16.5178929Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/test 2025-09-07T06:54:16.5179223Z + BUILD_DIR=build 2025-09-07T06:54:16.5179947Z + BUILD_RENAMED_DIR=build_renamed 2025-09-07T06:54:16.5180169Z + BUILD_BIN_DIR=build/bin 2025-09-07T06:54:16.5180359Z + SHARD_NUMBER=1 2025-09-07T06:54:16.5180524Z + NUM_TEST_SHARDS=1 2025-09-07T06:54:16.5180722Z + export TORCH_SERIALIZATION_DEBUG=1 2025-09-07T06:54:16.5180952Z + TORCH_SERIALIZATION_DEBUG=1 2025-09-07T06:54:16.5181206Z + export VALGRIND=ON 2025-09-07T06:54:16.5181390Z + VALGRIND=ON 2025-09-07T06:54:16.5181594Z + [[ linux-jammy-py3.9-gcc11-build == *clang9* ]] 2025-09-07T06:54:16.5181864Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-09-07T06:54:16.5182097Z + detect_cuda_arch 2025-09-07T06:54:16.5182295Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-09-07T06:54:16.5182554Z + [[ linux-jammy-py3.9-gcc11-build == *s390x* ]] 2025-09-07T06:54:16.5182781Z + [[ 0 == \1 ]] 2025-09-07T06:54:16.5182941Z + [[ True == \1 ]] 2025-09-07T06:54:16.5183126Z + [[ linux-jammy-py3.9-gcc11-build != *bazel* ]] 2025-09-07T06:54:16.5183366Z ++ realpath build/custom_test_artifacts 2025-09-07T06:54:16.5183771Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-09-07T06:54:16.5184109Z + [[ -n '' ]] 2025-09-07T06:54:16.5184280Z + echo 'Environment variables' 2025-09-07T06:54:16.5184487Z Environment variables 2025-09-07T06:54:16.5184664Z + env 2025-09-07T06:54:16.5212277Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-09-07T06:54:16.5212729Z CONTINUE_THROUGH_ERROR=True 2025-09-07T06:54:16.5213281Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-09-07T06:54:16.5214519Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-09-07T06:54:16.5215036Z HOSTNAME=9c09efa4294e 2025-09-07T06:54:16.5215525Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.5215956Z GITHUB_ACTION=__run_2 2025-09-07T06:54:16.5216164Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-09-07T06:54:16.5216391Z GITHUB_RUN_NUMBER=152138 2025-09-07T06:54:16.5216633Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-09-07T06:54:16.5216883Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-09-07T06:54:16.5217123Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-09-07T06:54:16.5217353Z SCCACHE_IDLE_TIMEOUT=0 2025-09-07T06:54:16.5217638Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-09-07T06:54:16.5217857Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-09-07T06:54:16.5218083Z GITHUB_REF_TYPE=branch 2025-09-07T06:54:16.5218300Z BASE_SHA=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.5218534Z XLA_CUDA= 2025-09-07T06:54:16.5218701Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-09-07T06:54:16.5219196Z HUGGING_FACE_HUB_TOKEN=*** 2025-09-07T06:54:16.5219464Z *** 2025-09-07T06:54:16.5219787Z GITHUB_REPOSITORY_ID=65600975 2025-09-07T06:54:16.5219999Z GITHUB_ACTIONS=true 2025-09-07T06:54:16.5220213Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-09-07T06:54:16.5220485Z SHA1=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.5220749Z GITHUB_SHA=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.5221112Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-09-07T06:54:16.5221426Z UCC_HOME=/usr 2025-09-07T06:54:16.5221607Z TORCH_SERIALIZATION_DEBUG=1 2025-09-07T06:54:16.5221809Z VERBOSE_TEST_LOGS=False 2025-09-07T06:54:16.5222001Z GITHUB_REF=refs/heads/main 2025-09-07T06:54:16.5222186Z SHARD_NUMBER=1 2025-09-07T06:54:16.5222359Z GITHUB_REF_PROTECTED=true 2025-09-07T06:54:16.5222550Z HOME=/var/lib/jenkins 2025-09-07T06:54:16.5222760Z GITHUB_API_URL=https://api.github.com 2025-09-07T06:54:16.5222990Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-09-07T06:54:16.5223194Z UCX_COMMIT= 2025-09-07T06:54:16.5223356Z USE_SYSTEM_NCCL=1 2025-09-07T06:54:16.5223525Z NUM_TEST_SHARDS=1 2025-09-07T06:54:16.5223683Z UCX_HOME=/usr 2025-09-07T06:54:16.5224068Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.5224626Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T06:54:16.5225274Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.5226018Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-09-07T06:54:16.5226376Z GITHUB_EVENT_NAME=push 2025-09-07T06:54:16.5226572Z DASHBOARD_TAG= 2025-09-07T06:54:16.5226754Z GITHUB_RUN_ID=17524754606 2025-09-07T06:54:16.5227056Z INSTALLED_OPENBLAS= 2025-09-07T06:54:16.5227486Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.5227937Z GITHUB_ACTOR=pytorchmergebot 2025-09-07T06:54:16.5228137Z PR_NUMBER= 2025-09-07T06:54:16.5228291Z DESIRED_CUDA= 2025-09-07T06:54:16.5228460Z GITHUB_RUN_ATTEMPT=1 2025-09-07T06:54:16.5228639Z VALGRIND=ON 2025-09-07T06:54:16.5228834Z ANACONDA_PYTHON_VERSION=3.9 2025-09-07T06:54:16.5229067Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-09-07T06:54:16.5229317Z TERM=vt100 2025-09-07T06:54:16.5229477Z INSTALLED_VISION=yes 2025-09-07T06:54:16.5229653Z BRANCH=main 2025-09-07T06:54:16.5229814Z SCCACHE_REGION=us-east-1 2025-09-07T06:54:16.5230019Z OPENSSL_ROOT_DIR=/opt/openssl 2025-09-07T06:54:16.5230222Z CUDA_PATH=/usr/local/cuda 2025-09-07T06:54:16.5230574Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-09-07T06:54:16.5230954Z GITHUB_SERVER_URL=https://github.com 2025-09-07T06:54:16.5231170Z UCC_COMMIT= 2025-09-07T06:54:16.5231329Z REENABLED_ISSUES= 2025-09-07T06:54:16.5231502Z DOCS=yes 2025-09-07T06:54:16.5231723Z SHLVL=1 2025-09-07T06:54:16.5231882Z MAX_JOBS=30 2025-09-07T06:54:16.5232045Z GITHUB_ACTOR_ID=97764156 2025-09-07T06:54:16.5232291Z GITHUB_WORKFLOW_SHA=93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T06:54:16.5232549Z GITHUB_REF_NAME=main 2025-09-07T06:54:16.5232826Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-09-07T06:54:16.5233125Z GITHUB_JOB=test 2025-09-07T06:54:16.5233306Z NO_TEST_TIMEOUT=False 2025-09-07T06:54:16.5233480Z TD_DISTRIBUTED=False 2025-09-07T06:54:16.5233673Z GITHUB_REPOSITORY=pytorch/pytorch 2025-09-07T06:54:16.5233888Z GITHUB_RETENTION_DAYS=90 2025-09-07T06:54:16.5234075Z OPENSSL_DIR=/opt/openssl 2025-09-07T06:54:16.5234289Z GITHUB_ACTION_REPOSITORY= 2025-09-07T06:54:16.5234793Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-09-07T06:54:16.5235298Z GITHUB_BASE_REF= 2025-09-07T06:54:16.5235458Z INSTALLED_ACL= 2025-09-07T06:54:16.5235768Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T06:54:16.5236102Z CI=true 2025-09-07T06:54:16.5236263Z GITHUB_REPOSITORY_OWNER=pytorch 2025-09-07T06:54:16.5236530Z RUST_LOG=sccache::server=error 2025-09-07T06:54:16.5236724Z JOB_ID=49774397867 2025-09-07T06:54:16.5236880Z GITHUB_HEAD_REF= 2025-09-07T06:54:16.5237046Z GITHUB_ACTION_REF= 2025-09-07T06:54:16.5237260Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-09-07T06:54:16.5237511Z TEST_SHOWLOCALS=False 2025-09-07T06:54:16.5237685Z GITHUB_WORKFLOW=inductor 2025-09-07T06:54:16.5237877Z DEBIAN_FRONTEND=noninteractive 2025-09-07T06:54:16.5238296Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_dc3ea66d-30d2-481d-8b1e-d71e31641f3e 2025-09-07T06:54:16.5238712Z NO_TD=False 2025-09-07T06:54:16.5238874Z SKIP_SCCACHE_INITIALIZATION=1 2025-09-07T06:54:16.5239089Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-09-07T06:54:16.5239303Z _=/usr/bin/env 2025-09-07T06:54:16.5239469Z + echo 'Testing pytorch' 2025-09-07T06:54:16.5239645Z Testing pytorch 2025-09-07T06:54:16.5239834Z + export LANG=C.UTF-8 2025-09-07T06:54:16.5240008Z + LANG=C.UTF-8 2025-09-07T06:54:16.5240166Z + PR_NUMBER= 2025-09-07T06:54:16.5240367Z + [[ dynamic_cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-09-07T06:54:16.5240674Z + [[ dynamic_cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-09-07T06:54:16.5241032Z + [[ dynamic_cpu_inductor_huggingface == \s\l\o\w ]] 2025-09-07T06:54:16.5241314Z + [[ linux-jammy-py3.9-gcc11-build == *slow-gradcheck* ]] 2025-09-07T06:54:16.5241601Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-09-07T06:54:16.5241853Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-09-07T06:54:16.5242100Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-09-07T06:54:16.5242358Z + [[ dynamic_cpu_inductor_huggingface == *crossref* ]] 2025-09-07T06:54:16.5242610Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-09-07T06:54:16.5242857Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-09-07T06:54:16.5243113Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-09-07T06:54:16.5243356Z + pip_install ninja==1.10.2 2025-09-07T06:54:16.5243610Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-09-07T06:54:16.5243920Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-09-07T06:54:16.8750624Z Collecting ninja==1.10.2 2025-09-07T06:54:16.8853233Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-09-07T06:54:16.8971450Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-09-07T06:54:17.6716710Z Installing collected packages: ninja 2025-09-07T06:54:17.6718565Z Attempting uninstall: ninja 2025-09-07T06:54:17.6727148Z Found existing installation: ninja 1.11.1.3 2025-09-07T06:54:17.6742732Z Uninstalling ninja-1.11.1.3: 2025-09-07T06:54:17.6802552Z Successfully uninstalled ninja-1.11.1.3 2025-09-07T06:54:18.2287959Z Successfully installed ninja-1.10.2 2025-09-07T06:54:18.3259373Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-09-07T06:54:18.3260414Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-09-07T06:54:18.3261106Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-09-07T06:54:18.3261390Z + [[ linux-jammy-py3.9-gcc11-build == *asan* ]] 2025-09-07T06:54:18.3261676Z + [[ linux-jammy-py3.9-gcc11-build == *-debug* ]] 2025-09-07T06:54:18.3261945Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-09-07T06:54:18.3262320Z + echo 'We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass' 2025-09-07T06:54:18.3262793Z We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass 2025-09-07T06:54:18.3263110Z + cd test 2025-09-07T06:54:18.3263369Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-09-07T06:54:18.6086508Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:54:18.6087457Z import pynvml # type: ignore[import] 2025-09-07T06:54:19.5555995Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-09-07T06:54:19.5560475Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-09-07T06:54:19.5563570Z + [[ dynamic_cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-09-07T06:54:19.5569533Z + DYNAMO_BENCHMARK_FLAGS=() 2025-09-07T06:54:19.5573734Z + [[ dynamic_cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-09-07T06:54:19.5575920Z + [[ dynamic_cpu_inductor_huggingface == *dynamo_eager* ]] 2025-09-07T06:54:19.5576348Z + [[ dynamic_cpu_inductor_huggingface == *aot_eager* ]] 2025-09-07T06:54:19.5580438Z + [[ dynamic_cpu_inductor_huggingface == *aot_inductor* ]] 2025-09-07T06:54:19.5580879Z + [[ dynamic_cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-09-07T06:54:19.5581218Z + [[ dynamic_cpu_inductor_huggingface == *inductor* ]] 2025-09-07T06:54:19.5581969Z + [[ dynamic_cpu_inductor_huggingface != *perf* ]] 2025-09-07T06:54:19.5582285Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-09-07T06:54:19.5582534Z + [[ dynamic_cpu_inductor_huggingface == *dynamic* ]] 2025-09-07T06:54:19.5582868Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2025-09-07T06:54:19.5583177Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-09-07T06:54:19.5583454Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-09-07T06:54:19.5807492Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-09-07T06:54:19.5807909Z + [[ linux-jammy-py3.9-gcc11-build == *-bazel-* ]] 2025-09-07T06:54:19.5810053Z + cd test 2025-09-07T06:54:19.5810293Z + python -c 'import torch; print(torch.__config__.show())' 2025-09-07T06:54:19.8866753Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:54:19.8867858Z import pynvml # type: ignore[import] 2025-09-07T06:54:20.5797015Z PyTorch built with: 2025-09-07T06:54:20.5797294Z - GCC 11.4 2025-09-07T06:54:20.5797474Z - C++ Version: 201703 2025-09-07T06:54:20.5797874Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-09-07T06:54:20.5798356Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-09-07T06:54:20.5799078Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-09-07T06:54:20.5799341Z - LAPACK is enabled (usually provided by MKL) 2025-09-07T06:54:20.5799566Z - NNPACK is enabled 2025-09-07T06:54:20.5799749Z - CPU capability usage: AVX512 2025-09-07T06:54:20.5802630Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=93fb23d6fae7c4e82c4239a1033e522088742634, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-09-07T06:54:20.5805621Z 2025-09-07T06:54:20.8031639Z + cd test 2025-09-07T06:54:20.8031974Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-09-07T06:54:21.1026169Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:54:21.1027103Z import pynvml # type: ignore[import] 2025-09-07T06:54:21.7857588Z ATen/Parallel: 2025-09-07T06:54:21.7857957Z at::get_num_threads() : 16 2025-09-07T06:54:21.7858245Z at::get_num_interop_threads() : 16 2025-09-07T06:54:21.7858513Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-09-07T06:54:21.7858755Z omp_get_max_threads() : 16 2025-09-07T06:54:21.7859218Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-09-07T06:54:21.7859676Z mkl_get_max_threads() : 16 2025-09-07T06:54:21.7859992Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-09-07T06:54:21.7860715Z std::thread::hardware_concurrency() : 32 2025-09-07T06:54:21.7860972Z Environment variables: 2025-09-07T06:54:21.7861175Z OMP_NUM_THREADS : [not set] 2025-09-07T06:54:21.7861380Z MKL_NUM_THREADS : [not set] 2025-09-07T06:54:21.7861589Z ATen parallel backend: OpenMP 2025-09-07T06:54:21.7861728Z 2025-09-07T06:54:22.0036558Z + [[ dynamic_cpu_inductor_huggingface == *numpy_2* ]] 2025-09-07T06:54:22.0042178Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-09-07T06:54:22.0044178Z + [[ dynamic_cpu_inductor_huggingface == *backward* ]] 2025-09-07T06:54:22.0049948Z + [[ dynamic_cpu_inductor_huggingface == *xla* ]] 2025-09-07T06:54:22.0052209Z + [[ dynamic_cpu_inductor_huggingface == *vllm* ]] 2025-09-07T06:54:22.0058186Z + [[ dynamic_cpu_inductor_huggingface == *executorch* ]] 2025-09-07T06:54:22.0059875Z + [[ dynamic_cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-09-07T06:54:22.0060235Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-09-07T06:54:22.0060579Z + [[ dynamic_cpu_inductor_huggingface == distributed ]] 2025-09-07T06:54:22.0060865Z + [[ dynamic_cpu_inductor_huggingface == *operator_benchmark* ]] 2025-09-07T06:54:22.0061180Z + [[ dynamic_cpu_inductor_huggingface == *inductor_distributed* ]] 2025-09-07T06:54:22.0061491Z + [[ dynamic_cpu_inductor_huggingface == *inductor-halide* ]] 2025-09-07T06:54:22.0061808Z + [[ dynamic_cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-09-07T06:54:22.0062148Z + [[ dynamic_cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-09-07T06:54:22.0062851Z + [[ dynamic_cpu_inductor_huggingface == *huggingface* ]] 2025-09-07T06:54:22.0063107Z + install_torchvision 2025-09-07T06:54:22.0063295Z + local orig_preload 2025-09-07T06:54:22.0063479Z + local commit 2025-09-07T06:54:22.0063668Z ++ get_pinned_commit vision 2025-09-07T06:54:22.0063880Z ++ cat .github/ci_commit_pins/vision.txt 2025-09-07T06:54:22.0518668Z + commit=966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-09-07T06:54:22.0519029Z + orig_preload= 2025-09-07T06:54:22.0519221Z + '[' -n '' ']' 2025-09-07T06:54:22.0519417Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-09-07T06:54:22.0520095Z + pip_build_and_install git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 dist/vision 2025-09-07T06:54:22.0520686Z + local build_target=git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-09-07T06:54:22.0521073Z + local wheel_dir=dist/vision 2025-09-07T06:54:22.0521288Z + local found_whl=0 2025-09-07T06:54:22.0521489Z + for file in "${wheel_dir}"/*.whl 2025-09-07T06:54:22.0521831Z + [[ -f dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl ]] 2025-09-07T06:54:22.0522147Z + found_whl=1 2025-09-07T06:54:22.0522309Z + break 2025-09-07T06:54:22.0522482Z + '[' 1 == 0 ']' 2025-09-07T06:54:22.0522664Z + for file in "${wheel_dir}"/*.whl 2025-09-07T06:54:22.0522988Z + pip_install_whl dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-09-07T06:54:22.0523424Z + args=('dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl') 2025-09-07T06:54:22.0523724Z + local args 2025-09-07T06:54:22.0523995Z + [[ dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl == *\ * ]] 2025-09-07T06:54:22.0524323Z + for path in "${args[@]}" 2025-09-07T06:54:22.0524659Z + echo 'Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl' 2025-09-07T06:54:22.0525107Z Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-09-07T06:54:22.0525598Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-09-07T06:54:22.3493079Z Processing ./dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-09-07T06:54:22.3576299Z Installing collected packages: torchvision 2025-09-07T06:54:22.7401917Z Successfully installed torchvision-0.22.0a0+966da7e 2025-09-07T06:54:22.7842275Z + '[' -n '' ']' 2025-09-07T06:54:22.7842964Z + id=0 2025-09-07T06:54:22.7843343Z + test_dynamo_benchmark huggingface 0 2025-09-07T06:54:22.7846298Z ++ pwd 2025-09-07T06:54:22.7846697Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-09-07T06:54:22.7847074Z + local suite=huggingface 2025-09-07T06:54:22.7847347Z + shift 2025-09-07T06:54:22.7847536Z + local shard_id=0 2025-09-07T06:54:22.7847772Z + shift 2025-09-07T06:54:22.7848055Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-09-07T06:54:22.7848393Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-09-07T06:54:22.7848759Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-09-07T06:54:22.7852982Z + local dt=float32 2025-09-07T06:54:22.7853209Z + [[ dynamic_cpu_inductor_huggingface == *amp* ]] 2025-09-07T06:54:22.7853538Z + [[ dynamic_cpu_inductor_huggingface == *freezing* ]] 2025-09-07T06:54:22.7853935Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-09-07T06:54:22.7854210Z ++ pwd 2025-09-07T06:54:22.7854497Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-09-07T06:54:22.7857405Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-09-07T06:54:22.7878865Z + local name=inference 2025-09-07T06:54:22.7881423Z + shift 2025-09-07T06:54:22.7881644Z + local suite=huggingface 2025-09-07T06:54:22.7881860Z + shift 2025-09-07T06:54:22.7882014Z + local shard_id=0 2025-09-07T06:54:22.7882179Z + shift 2025-09-07T06:54:22.7882323Z + partition_flags=() 2025-09-07T06:54:22.7882504Z + local partition_flags 2025-09-07T06:54:22.7882685Z + [[ -n 1 ]] 2025-09-07T06:54:22.7882843Z + [[ -n 0 ]] 2025-09-07T06:54:22.7883409Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-09-07T06:54:22.7883777Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-09-07T06:54:22.7884045Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-09-07T06:54:22.7884300Z + [[ dynamic_cpu_inductor_huggingface == *_avx2* ]] 2025-09-07T06:54:22.7884548Z + [[ dynamic_cpu_inductor_huggingface == *_avx512* ]] 2025-09-07T06:54:22.7885427Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --dynamic-shapes --dynamic-batch-only --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-09-07T06:54:23.5158501Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:54:23.5159345Z import pynvml # type: ignore[import] 2025-09-07T06:54:26.3507076Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T06:54:26.3508107Z from pkg_resources import resource_filename 2025-09-07T06:54:26.8563592Z 2025-09-07T06:54:26.8605174Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-09-07T06:56:49.6925437Z 2025-09-07T06:56:49.6925551Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6926112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6926643Z layer_outputs = layer_module( 2025-09-07T06:56:49.6927105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6927506Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6927951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6928406Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6928856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6929306Z self_outputs = self.self( 2025-09-07T06:56:49.6929738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.6930199Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.6930700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.6931289Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.6931536Z 2025-09-07T06:56:49.6931647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6932157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6932637Z layer_outputs = layer_module( 2025-09-07T06:56:49.6932987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6933352Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6933774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6934208Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6934636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6935087Z self_outputs = self.self( 2025-09-07T06:56:49.6935526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.6935984Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.6936535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.6937132Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.6937386Z 2025-09-07T06:56:49.6937494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6938026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6938537Z layer_outputs = layer_module( 2025-09-07T06:56:49.6938886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6939246Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6939677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6940115Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6940544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6940968Z self_outputs = self.self( 2025-09-07T06:56:49.6941377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.6941832Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.6942401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.6943008Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.6943254Z 2025-09-07T06:56:49.6943347Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.6943564Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.6943790Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.6944014Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.6944268Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6944827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6945366Z layer_outputs = layer_module( 2025-09-07T06:56:49.6945832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6946261Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6946732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6947201Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6947663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6948119Z self_outputs = self.self( 2025-09-07T06:56:49.6948556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.6949034Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.6949578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.6950174Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.6950744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.6951320Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.6951591Z 2025-09-07T06:56:49.6951686Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.6951934Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6952490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6953011Z layer_outputs = layer_module( 2025-09-07T06:56:49.6953385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6953779Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6954232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6954684Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6955140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6955593Z self_outputs = self.self( 2025-09-07T06:56:49.6956014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.6956436Z attn_scores += diagonal_mask 2025-09-07T06:56:49.6956569Z 2025-09-07T06:56:49.6956678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6957208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6957734Z layer_outputs = layer_module( 2025-09-07T06:56:49.6958089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6958463Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6958897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6959327Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6959751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6960178Z self_outputs = self.self( 2025-09-07T06:56:49.6960589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.6961023Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.6961160Z 2025-09-07T06:56:49.6961277Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6961803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6962304Z layer_outputs = layer_module( 2025-09-07T06:56:49.6962681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6963054Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6963484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6963921Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6964375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6964820Z self_outputs = self.self( 2025-09-07T06:56:49.6965254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.6965757Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.6966335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.6967022Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.6967496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.6967861Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.6968019Z 2025-09-07T06:56:49.6968132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6968663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6969160Z layer_outputs = layer_module( 2025-09-07T06:56:49.6969522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6969899Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6970326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6970751Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6971177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6971602Z self_outputs = self.self( 2025-09-07T06:56:49.6972008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.6972509Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.6988553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.6989202Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.6989769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.6990312Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.6990688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.6991082Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.6991263Z 2025-09-07T06:56:49.6991384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6991963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6992501Z layer_outputs = layer_module( 2025-09-07T06:56:49.6992884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6993291Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6993760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.6994228Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.6994688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.6995138Z self_outputs = self.self( 2025-09-07T06:56:49.6995557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.6996028Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.6996572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.6997157Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.6997368Z 2025-09-07T06:56:49.6997619Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.6998144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.6998646Z layer_outputs = layer_module( 2025-09-07T06:56:49.6999012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.6999391Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.6999828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7000261Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7000693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7001116Z self_outputs = self.self( 2025-09-07T06:56:49.7001535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7002000Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7002538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7003112Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7003329Z 2025-09-07T06:56:49.7003490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7004025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7004544Z layer_outputs = layer_module( 2025-09-07T06:56:49.7004917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7005314Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7005770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7006235Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7006715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7007141Z self_outputs = self.self( 2025-09-07T06:56:49.7007560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7008120Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7008379Z 2025-09-07T06:56:49.7008500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7009049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7009578Z layer_outputs = layer_module( 2025-09-07T06:56:49.7009954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7010329Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7010759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7011182Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7011617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7012108Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7012593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7013087Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7013239Z 2025-09-07T06:56:49.7013354Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7013910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7014438Z layer_outputs = layer_module( 2025-09-07T06:56:49.7014816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7015209Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7015653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7016122Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7016564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7016997Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7017442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7017938Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7018422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7018915Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7019067Z 2025-09-07T06:56:49.7019186Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7019943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7020477Z layer_outputs = layer_module( 2025-09-07T06:56:49.7020863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7021268Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7021729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7022194Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7022642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7023082Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7023546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7024078Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7024588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7025123Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7025557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7026034Z return self.act(input) 2025-09-07T06:56:49.7026221Z 2025-09-07T06:56:49.7026347Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7026926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7027484Z layer_outputs = layer_module( 2025-09-07T06:56:49.7027873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7028275Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7028749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7029344Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7029799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7030266Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7030740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7031273Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7031790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7032350Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7032531Z 2025-09-07T06:56:49.7032656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7033241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7033789Z layer_outputs = layer_module( 2025-09-07T06:56:49.7034175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7034573Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7035107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7035588Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7036067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7036551Z self_outputs = self.self( 2025-09-07T06:56:49.7036987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7037455Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7037605Z 2025-09-07T06:56:49.7037713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7038253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7038758Z layer_outputs = layer_module( 2025-09-07T06:56:49.7039117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7039499Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7039938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7040380Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7040805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7041241Z self_outputs = self.self( 2025-09-07T06:56:49.7041653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7042120Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7042648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7043260Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7043523Z 2025-09-07T06:56:49.7043630Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7044171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7044715Z layer_outputs = layer_module( 2025-09-07T06:56:49.7045069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7045433Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7045859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7046286Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7046713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7047160Z self_outputs = self.self( 2025-09-07T06:56:49.7047585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7048014Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7048160Z 2025-09-07T06:56:49.7048266Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7048788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7049285Z layer_outputs = layer_module( 2025-09-07T06:56:49.7049631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7050003Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7050467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7050902Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7051322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7051747Z self_outputs = self.self( 2025-09-07T06:56:49.7052165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7052633Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7053151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7053747Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7054013Z 2025-09-07T06:56:49.7054119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7054653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7055189Z layer_outputs = layer_module( 2025-09-07T06:56:49.7055568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7055963Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7056425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7056881Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7057311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7057739Z self_outputs = self.self( 2025-09-07T06:56:49.7058150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7058631Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7059176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7059861Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7060123Z 2025-09-07T06:56:49.7060243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7060799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7061326Z layer_outputs = layer_module( 2025-09-07T06:56:49.7061688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7062061Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7062520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7062974Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7063469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7063931Z self_outputs = self.self( 2025-09-07T06:56:49.7064365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7064845Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7065420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7066175Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7066464Z 2025-09-07T06:56:49.7066560Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7066815Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7067058Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7067270Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7067522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7068061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7068569Z layer_outputs = layer_module( 2025-09-07T06:56:49.7068925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7069305Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7069742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7070179Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7070611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7071034Z self_outputs = self.self( 2025-09-07T06:56:49.7071448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7071930Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7072458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7073021Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7073568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7074122Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7074330Z 2025-09-07T06:56:49.7074414Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7074666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7075275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7075805Z layer_outputs = layer_module( 2025-09-07T06:56:49.7076183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7076571Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7077024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7077456Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7077892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7078325Z self_outputs = self.self( 2025-09-07T06:56:49.7078774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7079202Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7079329Z 2025-09-07T06:56:49.7079445Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7079971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7080454Z layer_outputs = layer_module( 2025-09-07T06:56:49.7080849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7081226Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7081662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7082091Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7082515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7082938Z self_outputs = self.self( 2025-09-07T06:56:49.7083347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7083788Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7083934Z 2025-09-07T06:56:49.7084051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7084610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7085139Z layer_outputs = layer_module( 2025-09-07T06:56:49.7085517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7085911Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7086368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7086824Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7087271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7087724Z self_outputs = self.self( 2025-09-07T06:56:49.7088164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7088618Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7088779Z 2025-09-07T06:56:49.7088892Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7089448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7090029Z layer_outputs = layer_module( 2025-09-07T06:56:49.7090411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7090806Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7091269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7091729Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7092193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7092647Z self_outputs = self.self( 2025-09-07T06:56:49.7093079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7093636Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7094229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7094884Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7095358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7095745Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7095921Z 2025-09-07T06:56:49.7096035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7096644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7097176Z layer_outputs = layer_module( 2025-09-07T06:56:49.7097552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7097947Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7098406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7098862Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7099329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7099744Z self_outputs = self.self( 2025-09-07T06:56:49.7100159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7100684Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7101263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7101850Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7102417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7102903Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7103262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7103625Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7103791Z 2025-09-07T06:56:49.7103920Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7104493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7105052Z layer_outputs = layer_module( 2025-09-07T06:56:49.7105451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7106001Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7106486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7106962Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7107413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7107837Z self_outputs = self.self( 2025-09-07T06:56:49.7108259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7108734Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7109274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7109864Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7110084Z 2025-09-07T06:56:49.7110192Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7110725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7111228Z layer_outputs = layer_module( 2025-09-07T06:56:49.7111578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7111994Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7112427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7112855Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7113278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7113700Z self_outputs = self.self( 2025-09-07T06:56:49.7114112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7114579Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7115115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7115696Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7115906Z 2025-09-07T06:56:49.7116014Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7116537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7117028Z layer_outputs = layer_module( 2025-09-07T06:56:49.7117383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7117757Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7118179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7118607Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7119039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7119460Z self_outputs = self.self( 2025-09-07T06:56:49.7120035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7120586Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7120938Z 2025-09-07T06:56:49.7121049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7121580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7122077Z layer_outputs = layer_module( 2025-09-07T06:56:49.7122433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7122815Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7123271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7123704Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7124132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7124585Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7125049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7125477Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7125617Z 2025-09-07T06:56:49.7125727Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7126240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7126771Z layer_outputs = layer_module( 2025-09-07T06:56:49.7127121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7127489Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7127897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7128319Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7128706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7129097Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7129514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7129970Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7130422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7130844Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7130991Z 2025-09-07T06:56:49.7131093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7131604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7132120Z layer_outputs = layer_module( 2025-09-07T06:56:49.7132468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7132822Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7133240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7133718Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7134132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7134530Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7134956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7135472Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7135912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7136362Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7136738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7137092Z return self.act(input) 2025-09-07T06:56:49.7137214Z 2025-09-07T06:56:49.7137334Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7137848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7138335Z layer_outputs = layer_module( 2025-09-07T06:56:49.7138676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7139050Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7139480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7139917Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7140325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7140719Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7141182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7141673Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7142143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7142575Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7142729Z 2025-09-07T06:56:49.7142839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7143398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7143918Z layer_outputs = layer_module( 2025-09-07T06:56:49.7144289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7144701Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7145191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7145700Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7146165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7146612Z self_outputs = self.self( 2025-09-07T06:56:49.7147054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7147493Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7147637Z 2025-09-07T06:56:49.7147752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7148296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7148788Z layer_outputs = layer_module( 2025-09-07T06:56:49.7149145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7149520Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7149951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7150416Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7150838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7151263Z self_outputs = self.self( 2025-09-07T06:56:49.7151669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7152126Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7152640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7153335Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7153597Z 2025-09-07T06:56:49.7153703Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7154235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7154757Z layer_outputs = layer_module( 2025-09-07T06:56:49.7155133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7155500Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7155979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7156410Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7156839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7157254Z self_outputs = self.self( 2025-09-07T06:56:49.7157658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7158094Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7158235Z 2025-09-07T06:56:49.7158350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7158890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7159414Z layer_outputs = layer_module( 2025-09-07T06:56:49.7159772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7160143Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7160572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7161005Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7161425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7161857Z self_outputs = self.self( 2025-09-07T06:56:49.7162268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7162722Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7163239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7163850Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7164120Z 2025-09-07T06:56:49.7164232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7164786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7165356Z layer_outputs = layer_module( 2025-09-07T06:56:49.7165733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7166172Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7166602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7167032Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7167462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7167876Z self_outputs = self.self( 2025-09-07T06:56:49.7168270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7168717Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7169233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7169830Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7170078Z 2025-09-07T06:56:49.7170188Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7170706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7171226Z layer_outputs = layer_module( 2025-09-07T06:56:49.7171572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7171937Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7172368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7172810Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7173223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7173629Z self_outputs = self.self( 2025-09-07T06:56:49.7174034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7174478Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7175000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7175607Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7175873Z 2025-09-07T06:56:49.7175963Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7176199Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7176425Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7176639Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7176880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7177432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7177945Z layer_outputs = layer_module( 2025-09-07T06:56:49.7178286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7178650Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7179067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7179487Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7179893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7180344Z self_outputs = self.self( 2025-09-07T06:56:49.7180751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7181215Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7181735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7182291Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7182832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7183378Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7183590Z 2025-09-07T06:56:49.7183679Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7183926Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7184477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7185008Z layer_outputs = layer_module( 2025-09-07T06:56:49.7185391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7185983Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7186468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7186943Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7187424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7187908Z self_outputs = self.self( 2025-09-07T06:56:49.7188317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7188742Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7188868Z 2025-09-07T06:56:49.7188974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7189508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7190006Z layer_outputs = layer_module( 2025-09-07T06:56:49.7190368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7190734Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7191186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7191642Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7192093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7192530Z self_outputs = self.self( 2025-09-07T06:56:49.7192932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7193363Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7193504Z 2025-09-07T06:56:49.7193612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7194149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7194672Z layer_outputs = layer_module( 2025-09-07T06:56:49.7195036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7195499Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7195949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7196380Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7196805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7197220Z self_outputs = self.self( 2025-09-07T06:56:49.7197636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7198074Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7198218Z 2025-09-07T06:56:49.7198330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7198892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7199416Z layer_outputs = layer_module( 2025-09-07T06:56:49.7199796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7200193Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7200648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7201141Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7201568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7201988Z self_outputs = self.self( 2025-09-07T06:56:49.7202399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7202889Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7203452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7204089Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7204555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7204958Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7205126Z 2025-09-07T06:56:49.7205261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7205811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7206331Z layer_outputs = layer_module( 2025-09-07T06:56:49.7206710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7207104Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7207556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7208003Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7208452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7208987Z self_outputs = self.self( 2025-09-07T06:56:49.7209422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7209915Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7210474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7211107Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7211667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7212185Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7212555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7212933Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7213108Z 2025-09-07T06:56:49.7213215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7213757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7214283Z layer_outputs = layer_module( 2025-09-07T06:56:49.7214659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7215046Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7215541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7215973Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7216449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7216871Z self_outputs = self.self( 2025-09-07T06:56:49.7217282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7217744Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7218281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7218883Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7219107Z 2025-09-07T06:56:49.7219228Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7219978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7220515Z layer_outputs = layer_module( 2025-09-07T06:56:49.7220902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7221303Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7221754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7222219Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7222673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7223126Z self_outputs = self.self( 2025-09-07T06:56:49.7223566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7224059Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7224639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7225251Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7225473Z 2025-09-07T06:56:49.7225597Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7226212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7226838Z layer_outputs = layer_module( 2025-09-07T06:56:49.7227220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7227613Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7228044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7228480Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7228902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7229325Z self_outputs = self.self( 2025-09-07T06:56:49.7229735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7230278Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7230524Z 2025-09-07T06:56:49.7230640Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7231158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7231654Z layer_outputs = layer_module( 2025-09-07T06:56:49.7232067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7232444Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7232877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7233300Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7233727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7234211Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7234700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7235158Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7235319Z 2025-09-07T06:56:49.7235423Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7235948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7236446Z layer_outputs = layer_module( 2025-09-07T06:56:49.7236823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7237206Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7237661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7238120Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7238558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7238984Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7239433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7239935Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7240417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7240875Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7241022Z 2025-09-07T06:56:49.7241143Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7241742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7242243Z layer_outputs = layer_module( 2025-09-07T06:56:49.7242597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7242968Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7243391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7243810Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7244206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7244609Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7245033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7245502Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7245949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7246421Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7246819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7247212Z return self.act(input) 2025-09-07T06:56:49.7247329Z 2025-09-07T06:56:49.7247435Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7247962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7248468Z layer_outputs = layer_module( 2025-09-07T06:56:49.7248828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7249191Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7249598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7250033Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7250441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7250848Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7251270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7251747Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7252211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7252636Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7252772Z 2025-09-07T06:56:49.7252883Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7253402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7253891Z layer_outputs = layer_module( 2025-09-07T06:56:49.7254243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7254614Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7255039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7255464Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7255883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7256338Z self_outputs = self.self( 2025-09-07T06:56:49.7256747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7257186Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7257320Z 2025-09-07T06:56:49.7257428Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7257934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7258413Z layer_outputs = layer_module( 2025-09-07T06:56:49.7258758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7259131Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7259563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7260062Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7260514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7260980Z self_outputs = self.self( 2025-09-07T06:56:49.7261467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7261949Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7262501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7263143Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7263419Z 2025-09-07T06:56:49.7263544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7264110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7264633Z layer_outputs = layer_module( 2025-09-07T06:56:49.7265016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7265415Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7266020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7266525Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7267001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7267467Z self_outputs = self.self( 2025-09-07T06:56:49.7267890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7268314Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7268451Z 2025-09-07T06:56:49.7268562Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7269070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7269602Z layer_outputs = layer_module( 2025-09-07T06:56:49.7269979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7270373Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7270828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7271310Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7271724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7272130Z self_outputs = self.self( 2025-09-07T06:56:49.7272529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7272975Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7273493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7274095Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7274351Z 2025-09-07T06:56:49.7274458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7274989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7275520Z layer_outputs = layer_module( 2025-09-07T06:56:49.7275888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7276258Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7276684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7277142Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7277562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7277980Z self_outputs = self.self( 2025-09-07T06:56:49.7278386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7278839Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7279351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7279937Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7280191Z 2025-09-07T06:56:49.7280295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7280824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7281321Z layer_outputs = layer_module( 2025-09-07T06:56:49.7281677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7282043Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7282471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7282896Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7283322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7283743Z self_outputs = self.self( 2025-09-07T06:56:49.7284150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7284602Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7285134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7285751Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7286036Z 2025-09-07T06:56:49.7286129Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7286345Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7286562Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7286773Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7287013Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7287543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7288051Z layer_outputs = layer_module( 2025-09-07T06:56:49.7288412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7288786Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7289218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7289653Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7290086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7290516Z self_outputs = self.self( 2025-09-07T06:56:49.7290930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7291404Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7291967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7292532Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7293061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7293612Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7293824Z 2025-09-07T06:56:49.7293911Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7294147Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7294672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7295173Z layer_outputs = layer_module( 2025-09-07T06:56:49.7295552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7295943Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7296365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7296794Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7297218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7297637Z self_outputs = self.self( 2025-09-07T06:56:49.7298037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7298463Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7298596Z 2025-09-07T06:56:49.7298702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7299232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7299727Z layer_outputs = layer_module( 2025-09-07T06:56:49.7300074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7300445Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7300917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7301352Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7301767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7302169Z self_outputs = self.self( 2025-09-07T06:56:49.7302569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7302994Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7303133Z 2025-09-07T06:56:49.7303246Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7303793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7304312Z layer_outputs = layer_module( 2025-09-07T06:56:49.7304685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7305076Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7305535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7306057Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7306577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7307045Z self_outputs = self.self( 2025-09-07T06:56:49.7307497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7307936Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7308083Z 2025-09-07T06:56:49.7308196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7308731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7309230Z layer_outputs = layer_module( 2025-09-07T06:56:49.7309587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7309957Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7310382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7310813Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7311240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7311679Z self_outputs = self.self( 2025-09-07T06:56:49.7312092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7312554Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7313098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7313723Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7314193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7314578Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7314742Z 2025-09-07T06:56:49.7314856Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7315419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7315960Z layer_outputs = layer_module( 2025-09-07T06:56:49.7316315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7316684Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7317102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7317534Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7317962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7318387Z self_outputs = self.self( 2025-09-07T06:56:49.7318787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7319258Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7319964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7320543Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7321071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7321545Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7321975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7322338Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7322493Z 2025-09-07T06:56:49.7322609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7323140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7323635Z layer_outputs = layer_module( 2025-09-07T06:56:49.7324001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7324364Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7324789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7325211Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7325627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7326053Z self_outputs = self.self( 2025-09-07T06:56:49.7326472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7326926Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7327464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7328012Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7328229Z 2025-09-07T06:56:49.7328333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7328853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7329339Z layer_outputs = layer_module( 2025-09-07T06:56:49.7329685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7330046Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7330468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7330964Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7331386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7331788Z self_outputs = self.self( 2025-09-07T06:56:49.7332172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7332632Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7333160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7333722Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7333929Z 2025-09-07T06:56:49.7334042Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7334555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7335041Z layer_outputs = layer_module( 2025-09-07T06:56:49.7335392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7335766Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7336231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7336658Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7337120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7337530Z self_outputs = self.self( 2025-09-07T06:56:49.7337929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7338451Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7338704Z 2025-09-07T06:56:49.7338811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7339341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7339842Z layer_outputs = layer_module( 2025-09-07T06:56:49.7340196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7340583Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7341037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7341502Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7341929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7342411Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7342894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7343363Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7343520Z 2025-09-07T06:56:49.7343633Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7344190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7344720Z layer_outputs = layer_module( 2025-09-07T06:56:49.7345099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7345547Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7346070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7346564Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7347017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7347457Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7347910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7348406Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7348894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7349334Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7349475Z 2025-09-07T06:56:49.7349579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7350107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7350603Z layer_outputs = layer_module( 2025-09-07T06:56:49.7350958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7351363Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7351785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7352221Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7352634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7353040Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7353480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7353972Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7354457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7354952Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7355374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7355723Z return self.act(input) 2025-09-07T06:56:49.7355844Z 2025-09-07T06:56:49.7355949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7356470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7356971Z layer_outputs = layer_module( 2025-09-07T06:56:49.7357327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7357690Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7358119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7358556Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7358966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7359373Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7359792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7360272Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7360781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7361221Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7361362Z 2025-09-07T06:56:49.7361484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7361993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7362478Z layer_outputs = layer_module( 2025-09-07T06:56:49.7362822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7363191Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7363612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7364060Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7364512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7364959Z self_outputs = self.self( 2025-09-07T06:56:49.7365388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7365823Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7365978Z 2025-09-07T06:56:49.7366124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7366679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7367199Z layer_outputs = layer_module( 2025-09-07T06:56:49.7367574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7367963Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7368410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7368863Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7369310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7369755Z self_outputs = self.self( 2025-09-07T06:56:49.7370180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7370660Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7371200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7371836Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7372098Z 2025-09-07T06:56:49.7372218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7372768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7373289Z layer_outputs = layer_module( 2025-09-07T06:56:49.7373663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7374057Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7374508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7374956Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7375407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7375892Z self_outputs = self.self( 2025-09-07T06:56:49.7376324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7376769Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7376921Z 2025-09-07T06:56:49.7377032Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7377588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7378111Z layer_outputs = layer_module( 2025-09-07T06:56:49.7378467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7378837Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7379277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7379732Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7380180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7380622Z self_outputs = self.self( 2025-09-07T06:56:49.7381042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7381569Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7382118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7382747Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7383014Z 2025-09-07T06:56:49.7383135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7383686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7384209Z layer_outputs = layer_module( 2025-09-07T06:56:49.7384582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7384973Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7385424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7385941Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7386402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7386863Z self_outputs = self.self( 2025-09-07T06:56:49.7387338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7387822Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7388357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7388993Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7389265Z 2025-09-07T06:56:49.7389381Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7389939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7390436Z layer_outputs = layer_module( 2025-09-07T06:56:49.7390787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7391198Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7391627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7392058Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7392476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7392900Z self_outputs = self.self( 2025-09-07T06:56:49.7393312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7393767Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7394281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7394887Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7395153Z 2025-09-07T06:56:49.7395239Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7395468Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7395694Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7395915Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7396162Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7396744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7397244Z layer_outputs = layer_module( 2025-09-07T06:56:49.7397602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7397998Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7398453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7398918Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7399344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7399765Z self_outputs = self.self( 2025-09-07T06:56:49.7400171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7400638Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7401162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7401729Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7402278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7402829Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7403045Z 2025-09-07T06:56:49.7403128Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7403374Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7403934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7404460Z layer_outputs = layer_module( 2025-09-07T06:56:49.7404832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7405226Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7405674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7406144Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7406572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7406991Z self_outputs = self.self( 2025-09-07T06:56:49.7407402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7407828Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7407954Z 2025-09-07T06:56:49.7408068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7408601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7409128Z layer_outputs = layer_module( 2025-09-07T06:56:49.7409484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7409855Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7410286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7410709Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7411139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7411561Z self_outputs = self.self( 2025-09-07T06:56:49.7412005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7412435Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7412570Z 2025-09-07T06:56:49.7412675Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7413202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7413720Z layer_outputs = layer_module( 2025-09-07T06:56:49.7414103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7414513Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7414984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7415469Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7415920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7416365Z self_outputs = self.self( 2025-09-07T06:56:49.7416798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7417256Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7417413Z 2025-09-07T06:56:49.7417524Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7418074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7418596Z layer_outputs = layer_module( 2025-09-07T06:56:49.7418970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7419355Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7419919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7420383Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7420837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7421359Z self_outputs = self.self( 2025-09-07T06:56:49.7421795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7422295Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7422868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7423507Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7423963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7424349Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7424523Z 2025-09-07T06:56:49.7424637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7425212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7425947Z layer_outputs = layer_module( 2025-09-07T06:56:49.7426334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7426751Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7427288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7427778Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7428241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7428694Z self_outputs = self.self( 2025-09-07T06:56:49.7429133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7429643Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7430220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7430788Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7431324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7431804Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7432150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7432507Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7432663Z 2025-09-07T06:56:49.7432772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7433291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7433786Z layer_outputs = layer_module( 2025-09-07T06:56:49.7434135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7434511Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7434965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7435421Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7435882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7436342Z self_outputs = self.self( 2025-09-07T06:56:49.7436749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7437249Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7437771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7438329Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7438538Z 2025-09-07T06:56:49.7438641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7439154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7439633Z layer_outputs = layer_module( 2025-09-07T06:56:49.7439966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7440331Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7440751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7441167Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7441579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7441990Z self_outputs = self.self( 2025-09-07T06:56:49.7442441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7442893Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7443418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7443988Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7444210Z 2025-09-07T06:56:49.7444318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7444846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7445351Z layer_outputs = layer_module( 2025-09-07T06:56:49.7445700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7446067Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7446502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7446936Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7447366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7447801Z self_outputs = self.self( 2025-09-07T06:56:49.7448212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7448756Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7449009Z 2025-09-07T06:56:49.7449117Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7449643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7450134Z layer_outputs = layer_module( 2025-09-07T06:56:49.7450488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7450862Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7451296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7451762Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7452192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7452648Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7453115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7453554Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7453697Z 2025-09-07T06:56:49.7453811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7454341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7454836Z layer_outputs = layer_module( 2025-09-07T06:56:49.7455201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7455575Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7456003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7456432Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7456876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7457297Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7457733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7458210Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7458670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7459126Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7459273Z 2025-09-07T06:56:49.7459381Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7459915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7460425Z layer_outputs = layer_module( 2025-09-07T06:56:49.7460785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7461166Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7461603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7462054Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7462484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7462884Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7463312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7463784Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7464253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7464757Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7465172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7465556Z return self.act(input) 2025-09-07T06:56:49.7465748Z 2025-09-07T06:56:49.7465871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7466498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7467050Z layer_outputs = layer_module( 2025-09-07T06:56:49.7467427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7467837Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7468292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7468726Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7469127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7469527Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7469948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7470421Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7470890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7471314Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7471462Z 2025-09-07T06:56:49.7471568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7472122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7472610Z layer_outputs = layer_module( 2025-09-07T06:56:49.7472953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7473307Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7473729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7474155Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7474580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7474999Z self_outputs = self.self( 2025-09-07T06:56:49.7475399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7475837Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7475981Z 2025-09-07T06:56:49.7476084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7476591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7477070Z layer_outputs = layer_module( 2025-09-07T06:56:49.7477402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7477761Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7478174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7478594Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7479003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7479413Z self_outputs = self.self( 2025-09-07T06:56:49.7479699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7479805Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7480151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7481026Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7481030Z 2025-09-07T06:56:49.7481134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7481490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7481564Z layer_outputs = layer_module( 2025-09-07T06:56:49.7481791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7481870Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7482148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7482232Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7482508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7482584Z self_outputs = self.self( 2025-09-07T06:56:49.7482863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7482950Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7482953Z 2025-09-07T06:56:49.7483092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7483456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7483537Z layer_outputs = layer_module( 2025-09-07T06:56:49.7483764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7483857Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7484146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7484229Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7484518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7484588Z self_outputs = self.self( 2025-09-07T06:56:49.7484887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7484994Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7485348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7485541Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7485545Z 2025-09-07T06:56:49.7485650Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7486017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7486088Z layer_outputs = layer_module( 2025-09-07T06:56:49.7486324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7486402Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7486700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7486775Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7487061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7487183Z self_outputs = self.self( 2025-09-07T06:56:49.7487474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7487584Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7487938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7488134Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7488137Z 2025-09-07T06:56:49.7488242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7488610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7488694Z layer_outputs = layer_module( 2025-09-07T06:56:49.7488923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7489012Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7489303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7489379Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7489706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7489777Z self_outputs = self.self( 2025-09-07T06:56:49.7490150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7490286Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7490672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7490865Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7490869Z 2025-09-07T06:56:49.7490956Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7491049Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7491131Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7491220Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7491334Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7491730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7491817Z layer_outputs = layer_module( 2025-09-07T06:56:49.7492071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7492169Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7492486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7492572Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7492873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7492946Z self_outputs = self.self( 2025-09-07T06:56:49.7493258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7493376Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7493756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7493945Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7494279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7494439Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7494443Z 2025-09-07T06:56:49.7494524Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7494637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7495001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7495078Z layer_outputs = layer_module( 2025-09-07T06:56:49.7495306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7495386Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7495694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7495773Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7496086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7496160Z self_outputs = self.self( 2025-09-07T06:56:49.7496496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7496583Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7496587Z 2025-09-07T06:56:49.7496697Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7497093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7497171Z layer_outputs = layer_module( 2025-09-07T06:56:49.7497418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7497502Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7497804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7497891Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7498262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7498339Z self_outputs = self.self( 2025-09-07T06:56:49.7498625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7498705Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7498715Z 2025-09-07T06:56:49.7498823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7499182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7499260Z layer_outputs = layer_module( 2025-09-07T06:56:49.7499488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7499580Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7499886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7499966Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7500277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7500352Z self_outputs = self.self( 2025-09-07T06:56:49.7500698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7500789Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7500792Z 2025-09-07T06:56:49.7500908Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7501287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7501362Z layer_outputs = layer_module( 2025-09-07T06:56:49.7501608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7501691Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7502003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7502080Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7502383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7502465Z self_outputs = self.self( 2025-09-07T06:56:49.7502766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7502901Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7503316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7503508Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7503717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7503825Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7503831Z 2025-09-07T06:56:49.7503949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7504335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7504417Z layer_outputs = layer_module( 2025-09-07T06:56:49.7504655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7504746Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7505055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7505134Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7505443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7505519Z self_outputs = self.self( 2025-09-07T06:56:49.7505929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7506069Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7506463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7506625Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7507055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7507162Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7507372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7507487Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7507533Z 2025-09-07T06:56:49.7507645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7508027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7508111Z layer_outputs = layer_module( 2025-09-07T06:56:49.7508348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7508443Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7508750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7508831Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7509147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7509226Z self_outputs = self.self( 2025-09-07T06:56:49.7509539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7509663Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7510053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7510254Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7510259Z 2025-09-07T06:56:49.7510369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7510763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7510839Z layer_outputs = layer_module( 2025-09-07T06:56:49.7511090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7511176Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7511490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7511569Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7511874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7511957Z self_outputs = self.self( 2025-09-07T06:56:49.7512258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7512387Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7512770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7512937Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7512949Z 2025-09-07T06:56:49.7513059Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7513442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7513525Z layer_outputs = layer_module( 2025-09-07T06:56:49.7513767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7513856Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7514172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7514246Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7514582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7514652Z self_outputs = self.self( 2025-09-07T06:56:49.7514943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7515134Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7515137Z 2025-09-07T06:56:49.7515249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7515611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7515683Z layer_outputs = layer_module( 2025-09-07T06:56:49.7515914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7515996Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7516287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7516361Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7516645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7516763Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7517085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7517180Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7517184Z 2025-09-07T06:56:49.7517286Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7517653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7517728Z layer_outputs = layer_module( 2025-09-07T06:56:49.7517950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7518037Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7518320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7518415Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7518680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7518765Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7519052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7519166Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7519457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7519540Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7519543Z 2025-09-07T06:56:49.7519873Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7520244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7520319Z layer_outputs = layer_module( 2025-09-07T06:56:49.7520551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7520630Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7520925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7521082Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7521357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7521436Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7521725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7521842Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7522128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7522249Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7522467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7522541Z return self.act(input) 2025-09-07T06:56:49.7522551Z 2025-09-07T06:56:49.7522656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7523012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7523091Z layer_outputs = layer_module( 2025-09-07T06:56:49.7523311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7523444Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7523733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7523817Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7524087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7524167Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7524461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7524585Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7524880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7524964Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7524967Z 2025-09-07T06:56:49.7525073Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7525441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7525512Z layer_outputs = layer_module( 2025-09-07T06:56:49.7525757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7525833Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7526104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7526185Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7526464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7526540Z self_outputs = self.self( 2025-09-07T06:56:49.7526821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7526910Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7526914Z 2025-09-07T06:56:49.7527015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7527364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7527476Z layer_outputs = layer_module( 2025-09-07T06:56:49.7527694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7527777Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7528063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7528151Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7528437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7528502Z self_outputs = self.self( 2025-09-07T06:56:49.7528782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7528883Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7529227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7529408Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7529412Z 2025-09-07T06:56:49.7529510Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7529897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7529968Z layer_outputs = layer_module( 2025-09-07T06:56:49.7530198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7530274Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7530563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7530637Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7530915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7530990Z self_outputs = self.self( 2025-09-07T06:56:49.7531270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7531358Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7531362Z 2025-09-07T06:56:49.7531461Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7531815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7531891Z layer_outputs = layer_module( 2025-09-07T06:56:49.7532115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7532196Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7532479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7532559Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7532840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7532908Z self_outputs = self.self( 2025-09-07T06:56:49.7533193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7533293Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7533665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7533903Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7533907Z 2025-09-07T06:56:49.7534024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7534410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7534488Z layer_outputs = layer_module( 2025-09-07T06:56:49.7534741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7534818Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7535112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7535193Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7535477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7535555Z self_outputs = self.self( 2025-09-07T06:56:49.7535850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7535956Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7536333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7536525Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7536529Z 2025-09-07T06:56:49.7536630Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7536984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7537067Z layer_outputs = layer_module( 2025-09-07T06:56:49.7537290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7537376Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7537655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7537738Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7538023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7538094Z self_outputs = self.self( 2025-09-07T06:56:49.7538380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7538481Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7538826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7539005Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7539008Z 2025-09-07T06:56:49.7539098Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7539183Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7539261Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7539345Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7539447Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7539800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7539907Z layer_outputs = layer_module( 2025-09-07T06:56:49.7540126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7540210Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7540492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7540573Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7540855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7540926Z self_outputs = self.self( 2025-09-07T06:56:49.7541217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7541332Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7541692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7541843Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7542182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7542335Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7542339Z 2025-09-07T06:56:49.7542452Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7542564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7542935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7543012Z layer_outputs = layer_module( 2025-09-07T06:56:49.7543243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7543327Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7543631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7543708Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7544012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7544085Z self_outputs = self.self( 2025-09-07T06:56:49.7544400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7544478Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7544482Z 2025-09-07T06:56:49.7544590Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7544986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7545064Z layer_outputs = layer_module( 2025-09-07T06:56:49.7545315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7545400Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7545778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7545877Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7546254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7546340Z self_outputs = self.self( 2025-09-07T06:56:49.7546643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7546769Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7546774Z 2025-09-07T06:56:49.7546884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7547266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7547349Z layer_outputs = layer_module( 2025-09-07T06:56:49.7547590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7547675Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7547959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7548030Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7548319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7548390Z self_outputs = self.self( 2025-09-07T06:56:49.7548676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7548760Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7548764Z 2025-09-07T06:56:49.7548871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7549253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7549323Z layer_outputs = layer_module( 2025-09-07T06:56:49.7549548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7549624Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7549908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7549983Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7550268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7550344Z self_outputs = self.self( 2025-09-07T06:56:49.7550619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7550744Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7551101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7551282Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7551472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7551572Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7551575Z 2025-09-07T06:56:49.7551684Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7552038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7552116Z layer_outputs = layer_module( 2025-09-07T06:56:49.7552338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7552424Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7552702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7552776Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7553060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7553160Z self_outputs = self.self( 2025-09-07T06:56:49.7553448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7553565Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7553919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7554060Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7554377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7554476Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7554672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7554782Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7554786Z 2025-09-07T06:56:49.7554890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7555251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7555330Z layer_outputs = layer_module( 2025-09-07T06:56:49.7555587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7555675Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7555960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7556044Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7556334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7556404Z self_outputs = self.self( 2025-09-07T06:56:49.7556707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7556820Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7557181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7557333Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7557336Z 2025-09-07T06:56:49.7557446Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7557801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7557875Z layer_outputs = layer_module( 2025-09-07T06:56:49.7558108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7558187Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7558482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7558556Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7558844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7558921Z self_outputs = self.self( 2025-09-07T06:56:49.7559209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7559330Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7559780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7559938Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7559942Z 2025-09-07T06:56:49.7560045Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7560413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7560493Z layer_outputs = layer_module( 2025-09-07T06:56:49.7560723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7560811Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7561099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7561176Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7561468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7561538Z self_outputs = self.self( 2025-09-07T06:56:49.7561828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7562075Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7562079Z 2025-09-07T06:56:49.7562191Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7562555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7562626Z layer_outputs = layer_module( 2025-09-07T06:56:49.7562863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7562943Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7563239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7563313Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7563608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7563724Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7564012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7564104Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7564107Z 2025-09-07T06:56:49.7564208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7564577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7564647Z layer_outputs = layer_module( 2025-09-07T06:56:49.7564875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7564959Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7565249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7565341Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7565609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7565695Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7566015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7566125Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7566423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7566506Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7566509Z 2025-09-07T06:56:49.7566618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7566982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7567062Z layer_outputs = layer_module( 2025-09-07T06:56:49.7567285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7567364Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7567662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7567746Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7568019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7568095Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7568464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7568580Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7568852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7568969Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7569184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7569271Z return self.act(input) 2025-09-07T06:56:49.7569274Z 2025-09-07T06:56:49.7569383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7569762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7569848Z layer_outputs = layer_module( 2025-09-07T06:56:49.7570090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7570183Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7570487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7570576Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7570870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7570952Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7571271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7571395Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7571692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7571776Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7571779Z 2025-09-07T06:56:49.7571882Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7572247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7572354Z layer_outputs = layer_module( 2025-09-07T06:56:49.7572590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7572670Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7572959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7573055Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7573344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7573420Z self_outputs = self.self( 2025-09-07T06:56:49.7573704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7573792Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7573796Z 2025-09-07T06:56:49.7573900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7574260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7574340Z layer_outputs = layer_module( 2025-09-07T06:56:49.7574569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7574657Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7574979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7575064Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7575348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7575419Z self_outputs = self.self( 2025-09-07T06:56:49.7575714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7575817Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7576172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7576361Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7576364Z 2025-09-07T06:56:49.7576470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7576835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7576906Z layer_outputs = layer_module( 2025-09-07T06:56:49.7577137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7577219Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7577519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7577593Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7577869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7577946Z self_outputs = self.self( 2025-09-07T06:56:49.7578223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7578307Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7578310Z 2025-09-07T06:56:49.7578409Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7578764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7578864Z layer_outputs = layer_module( 2025-09-07T06:56:49.7579085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7579171Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7579459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7579543Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7579839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7579905Z self_outputs = self.self( 2025-09-07T06:56:49.7580195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7580298Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7580648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7580832Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7580835Z 2025-09-07T06:56:49.7580940Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7581327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7581399Z layer_outputs = layer_module( 2025-09-07T06:56:49.7581637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7581716Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7582013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7582091Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7582381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7582457Z self_outputs = self.self( 2025-09-07T06:56:49.7582747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7582859Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7583208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7583403Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7583407Z 2025-09-07T06:56:49.7583518Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7583903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7583985Z layer_outputs = layer_module( 2025-09-07T06:56:49.7584225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7584317Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7584625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7584711Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7585017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7585091Z self_outputs = self.self( 2025-09-07T06:56:49.7585442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7585548Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7585989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7586200Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7586204Z 2025-09-07T06:56:49.7586307Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7586398Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7586486Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7586579Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7586694Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7587109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7587191Z layer_outputs = layer_module( 2025-09-07T06:56:49.7587446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7587540Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7587846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7587972Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7588277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7588354Z self_outputs = self.self( 2025-09-07T06:56:49.7588669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7588791Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7589168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7589322Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7589682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7589848Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7589852Z 2025-09-07T06:56:49.7589936Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7590053Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7590434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7590520Z layer_outputs = layer_module( 2025-09-07T06:56:49.7590760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7590842Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7591155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7591236Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7591551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7591627Z self_outputs = self.self( 2025-09-07T06:56:49.7591938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7592016Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7592019Z 2025-09-07T06:56:49.7592164Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7592558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7592634Z layer_outputs = layer_module( 2025-09-07T06:56:49.7592881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7592966Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7593363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7593445Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7593750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7593833Z self_outputs = self.self( 2025-09-07T06:56:49.7594142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7594235Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7594239Z 2025-09-07T06:56:49.7594348Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7594728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7594807Z layer_outputs = layer_module( 2025-09-07T06:56:49.7595061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7595150Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7595452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7595539Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7595847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7595919Z self_outputs = self.self( 2025-09-07T06:56:49.7596225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7596316Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7596320Z 2025-09-07T06:56:49.7596434Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7596815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7596887Z layer_outputs = layer_module( 2025-09-07T06:56:49.7597117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7597199Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7597491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7597564Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7597856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7597925Z self_outputs = self.self( 2025-09-07T06:56:49.7598212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7598337Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7598701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7598885Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7599113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7599214Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7599225Z 2025-09-07T06:56:49.7599331Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7599689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7599771Z layer_outputs = layer_module( 2025-09-07T06:56:49.7599997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7600084Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7600368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7600447Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7600739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7600808Z self_outputs = self.self( 2025-09-07T06:56:49.7601100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7601218Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7601610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7601749Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7602073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7602177Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7602374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7602482Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7602486Z 2025-09-07T06:56:49.7602591Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7602956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7603036Z layer_outputs = layer_module( 2025-09-07T06:56:49.7603262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7603353Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7603659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7603750Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7604053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7604126Z self_outputs = self.self( 2025-09-07T06:56:49.7604434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7604556Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7604949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7605116Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7605119Z 2025-09-07T06:56:49.7605242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7605629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7605702Z layer_outputs = layer_module( 2025-09-07T06:56:49.7605936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7606014Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7606314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7606391Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7606675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7606753Z self_outputs = self.self( 2025-09-07T06:56:49.7607038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7607163Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7607523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7607681Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7607685Z 2025-09-07T06:56:49.7607794Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7608182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7608264Z layer_outputs = layer_module( 2025-09-07T06:56:49.7608489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7608578Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7608874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7608955Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7609229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7609296Z self_outputs = self.self( 2025-09-07T06:56:49.7609583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7609769Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7609773Z 2025-09-07T06:56:49.7609881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7610232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7610313Z layer_outputs = layer_module( 2025-09-07T06:56:49.7610530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7610607Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7610893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7610967Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7611256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7611366Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7611646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7611779Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7611783Z 2025-09-07T06:56:49.7611883Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7612239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7612309Z layer_outputs = layer_module( 2025-09-07T06:56:49.7612537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7612617Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7612897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7612992Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7613252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7613338Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7613621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7613731Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7614019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7614102Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7614138Z 2025-09-07T06:56:49.7614247Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7614629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7614709Z layer_outputs = layer_module( 2025-09-07T06:56:49.7614945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7615031Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7615340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7615429Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7615715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7615801Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7616107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7616232Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7616508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7616632Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7616845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7616919Z return self.act(input) 2025-09-07T06:56:49.7616922Z 2025-09-07T06:56:49.7617022Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7617369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7617446Z layer_outputs = layer_module( 2025-09-07T06:56:49.7617662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7617746Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7618025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7618143Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7618409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7618482Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7618771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7618890Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7619178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7619258Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7619261Z 2025-09-07T06:56:49.7619362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7619886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7619965Z layer_outputs = layer_module( 2025-09-07T06:56:49.7620193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7620269Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7620557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7620698Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7620979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7621056Z self_outputs = self.self( 2025-09-07T06:56:49.7621331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7621422Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7621425Z 2025-09-07T06:56:49.7621526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7621875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7621953Z layer_outputs = layer_module( 2025-09-07T06:56:49.7622170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7622259Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7622534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7622618Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7622901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7622974Z self_outputs = self.self( 2025-09-07T06:56:49.7623264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7623367Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7623726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7623917Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7623922Z 2025-09-07T06:56:49.7624033Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7624393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7624517Z layer_outputs = layer_module( 2025-09-07T06:56:49.7624750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7624828Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7625123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7625200Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7625490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7625566Z self_outputs = self.self( 2025-09-07T06:56:49.7625903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7625999Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7626003Z 2025-09-07T06:56:49.7626106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7626496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7626572Z layer_outputs = layer_module( 2025-09-07T06:56:49.7626817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7626915Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7627240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7627324Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7627615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7627684Z self_outputs = self.self( 2025-09-07T06:56:49.7628038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7628148Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7628506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7628693Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7628697Z 2025-09-07T06:56:49.7628810Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7629169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7629241Z layer_outputs = layer_module( 2025-09-07T06:56:49.7629473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7629554Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7629847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7629920Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7630212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7630280Z self_outputs = self.self( 2025-09-07T06:56:49.7630568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7630677Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7631023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7631213Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7631248Z 2025-09-07T06:56:49.7631354Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7631717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7631790Z layer_outputs = layer_module( 2025-09-07T06:56:49.7632018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7632109Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7632396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7632477Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7632765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7632838Z self_outputs = self.self( 2025-09-07T06:56:49.7633131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7633231Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7633582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7633806Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7633810Z 2025-09-07T06:56:49.7633903Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7633982Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7634059Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7634143Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7634246Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7634627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7634698Z layer_outputs = layer_module( 2025-09-07T06:56:49.7634929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7635018Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7635314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7635396Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7635693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7635763Z self_outputs = self.self( 2025-09-07T06:56:49.7636063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7636178Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7636543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7636687Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7637038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7637193Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7637197Z 2025-09-07T06:56:49.7637277Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7637387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7637755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7637868Z layer_outputs = layer_module( 2025-09-07T06:56:49.7638094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7638182Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7638468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7638546Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7638837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7638908Z self_outputs = self.self( 2025-09-07T06:56:49.7639203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7639280Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7639283Z 2025-09-07T06:56:49.7639387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7639755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7639828Z layer_outputs = layer_module( 2025-09-07T06:56:49.7640086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7640166Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7640460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7640535Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7640818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7640899Z self_outputs = self.self( 2025-09-07T06:56:49.7641180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7641266Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7641269Z 2025-09-07T06:56:49.7641370Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7641730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7641818Z layer_outputs = layer_module( 2025-09-07T06:56:49.7642035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7642119Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7642396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7642477Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7642751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7642819Z self_outputs = self.self( 2025-09-07T06:56:49.7643103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7643190Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7643193Z 2025-09-07T06:56:49.7643490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7643846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7643924Z layer_outputs = layer_module( 2025-09-07T06:56:49.7644186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7644264Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7644560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7644636Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7644936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7645012Z self_outputs = self.self( 2025-09-07T06:56:49.7645318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7645454Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7645829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7646017Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7646224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7646330Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7646334Z 2025-09-07T06:56:49.7646435Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7646823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7646904Z layer_outputs = layer_module( 2025-09-07T06:56:49.7647125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7647213Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7647498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7647580Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7647863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7647933Z self_outputs = self.self( 2025-09-07T06:56:49.7648223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7648342Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7648700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7648839Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7649158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7649257Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7649450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7649553Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7649557Z 2025-09-07T06:56:49.7649662Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7650022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7650095Z layer_outputs = layer_module( 2025-09-07T06:56:49.7650317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7650404Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7650721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7650801Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7651083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7651150Z self_outputs = self.self( 2025-09-07T06:56:49.7651437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7651550Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7651907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7652058Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7652064Z 2025-09-07T06:56:49.7652171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7652521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7652591Z layer_outputs = layer_module( 2025-09-07T06:56:49.7652817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7652895Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7653212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7653288Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7653577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7653648Z self_outputs = self.self( 2025-09-07T06:56:49.7653934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7654058Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7654418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7654578Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7654584Z 2025-09-07T06:56:49.7654687Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7655058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7655132Z layer_outputs = layer_module( 2025-09-07T06:56:49.7655372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7655468Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7655788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7655872Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7656163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7656235Z self_outputs = self.self( 2025-09-07T06:56:49.7656531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7656721Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7656724Z 2025-09-07T06:56:49.7656838Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7657231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7657308Z layer_outputs = layer_module( 2025-09-07T06:56:49.7657528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7657606Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7657900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7657974Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7658263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7658373Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7658666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7658750Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7658754Z 2025-09-07T06:56:49.7658856Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7659221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7659291Z layer_outputs = layer_module( 2025-09-07T06:56:49.7659561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7659640Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7659927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7660020Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7660288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7660372Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7660664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7660783Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7661076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7661161Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7661164Z 2025-09-07T06:56:49.7661275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7661631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7661711Z layer_outputs = layer_module( 2025-09-07T06:56:49.7661934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7662012Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7662305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7662388Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7662660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7662738Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7663032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7663141Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7663456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7663583Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7663811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7663893Z return self.act(input) 2025-09-07T06:56:49.7663897Z 2025-09-07T06:56:49.7664003Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7664387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7664463Z layer_outputs = layer_module( 2025-09-07T06:56:49.7664697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7664789Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7665097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7665193Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7665471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7665551Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7666011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7666152Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7666481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7666575Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7666579Z 2025-09-07T06:56:49.7666707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7667103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7667186Z layer_outputs = layer_module( 2025-09-07T06:56:49.7667448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7667536Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7667851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7667946Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7668233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7668315Z self_outputs = self.self( 2025-09-07T06:56:49.7668605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7668698Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7668701Z 2025-09-07T06:56:49.7668808Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7669176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7669249Z layer_outputs = layer_module( 2025-09-07T06:56:49.7669491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7669588Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7669897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7669987Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7670326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7670401Z self_outputs = self.self( 2025-09-07T06:56:49.7670708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7670819Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7671196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7671396Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7671399Z 2025-09-07T06:56:49.7671515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7671955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7672028Z layer_outputs = layer_module( 2025-09-07T06:56:49.7672260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7672338Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7672629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7672738Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7673038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7673110Z self_outputs = self.self( 2025-09-07T06:56:49.7673405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7673495Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7673499Z 2025-09-07T06:56:49.7673602Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7673973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7674044Z layer_outputs = layer_module( 2025-09-07T06:56:49.7674271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7674360Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7674648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7674731Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7675021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7675100Z self_outputs = self.self( 2025-09-07T06:56:49.7675389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7675492Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7675854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7676051Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7676054Z 2025-09-07T06:56:49.7676165Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7676529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7676605Z layer_outputs = layer_module( 2025-09-07T06:56:49.7676859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7676939Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7677231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7677307Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7677601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7677670Z self_outputs = self.self( 2025-09-07T06:56:49.7677954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7678065Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7678414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7678607Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7678610Z 2025-09-07T06:56:49.7678713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7679079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7679150Z layer_outputs = layer_module( 2025-09-07T06:56:49.7679403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7679493Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7679783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7679868Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7680161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7680239Z self_outputs = self.self( 2025-09-07T06:56:49.7680527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7680628Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7680993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7681181Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7681184Z 2025-09-07T06:56:49.7681275Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7681356Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7681435Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7681523Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7681631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7682002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7682074Z layer_outputs = layer_module( 2025-09-07T06:56:49.7682302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7682389Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7682679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7682761Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7683045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7683163Z self_outputs = self.self( 2025-09-07T06:56:49.7683448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7683560Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7683911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7684061Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7684398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7684553Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7684557Z 2025-09-07T06:56:49.7684643Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7684751Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7685111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7685190Z layer_outputs = layer_module( 2025-09-07T06:56:49.7685419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7685505Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7685821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7685900Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7686194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7686262Z self_outputs = self.self( 2025-09-07T06:56:49.7686559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7686632Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7686636Z 2025-09-07T06:56:49.7686747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7687105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7687177Z layer_outputs = layer_module( 2025-09-07T06:56:49.7687409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7687489Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7687784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7687861Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7688187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7688263Z self_outputs = self.self( 2025-09-07T06:56:49.7688550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7688636Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7688639Z 2025-09-07T06:56:49.7688740Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7689114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7689186Z layer_outputs = layer_module( 2025-09-07T06:56:49.7689411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7689528Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7689821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7689904Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7690191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7690259Z self_outputs = self.self( 2025-09-07T06:56:49.7690555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7690641Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7690645Z 2025-09-07T06:56:49.7690755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7691112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7691193Z layer_outputs = layer_module( 2025-09-07T06:56:49.7691422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7691500Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7691791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7691866Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7692187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7692258Z self_outputs = self.self( 2025-09-07T06:56:49.7692552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7692672Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7693044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7693224Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7693416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7693521Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7693524Z 2025-09-07T06:56:49.7693628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7693991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7694070Z layer_outputs = layer_module( 2025-09-07T06:56:49.7694292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7694381Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7694672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7694753Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7695044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7695113Z self_outputs = self.self( 2025-09-07T06:56:49.7695411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7695530Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7695901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7696074Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7696408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7696500Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7696695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7696801Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7696807Z 2025-09-07T06:56:49.7696910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7697276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7697348Z layer_outputs = layer_module( 2025-09-07T06:56:49.7697573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7697662Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7697947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7698037Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7698320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7698393Z self_outputs = self.self( 2025-09-07T06:56:49.7698703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7698822Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7699191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7699350Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7699354Z 2025-09-07T06:56:49.7699464Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7699826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7699903Z layer_outputs = layer_module( 2025-09-07T06:56:49.7700131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7700210Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7700511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7700583Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7700870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7700941Z self_outputs = self.self( 2025-09-07T06:56:49.7701236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7701364Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7701744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7701911Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7701915Z 2025-09-07T06:56:49.7702025Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7702414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7702525Z layer_outputs = layer_module( 2025-09-07T06:56:49.7702764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7702856Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7703162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7703247Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7703551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7703622Z self_outputs = self.self( 2025-09-07T06:56:49.7703933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7704133Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7704139Z 2025-09-07T06:56:49.7704257Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7704644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7704726Z layer_outputs = layer_module( 2025-09-07T06:56:49.7704968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7705084Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7705396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7705475Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7705855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7705985Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7706318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7706412Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7706416Z 2025-09-07T06:56:49.7706529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7706947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7707025Z layer_outputs = layer_module( 2025-09-07T06:56:49.7707283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7707369Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7707676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7707772Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7708030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7708122Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7708447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7708579Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7708910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7709001Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7709005Z 2025-09-07T06:56:49.7709125Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7709513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7709648Z layer_outputs = layer_module( 2025-09-07T06:56:49.7709908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7710001Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7710326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7710420Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7710720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7710804Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7711138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7711260Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7711583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7711715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7711952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7712035Z return self.act(input) 2025-09-07T06:56:49.7712069Z 2025-09-07T06:56:49.7712184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7712582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7712658Z layer_outputs = layer_module( 2025-09-07T06:56:49.7712922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7713021Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7713343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7713442Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7713732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7713819Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7714149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7714275Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7714572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7714658Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7714661Z 2025-09-07T06:56:49.7714770Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7715132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7715203Z layer_outputs = layer_module( 2025-09-07T06:56:49.7715447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7715524Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7715809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7715885Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7716169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7716269Z self_outputs = self.self( 2025-09-07T06:56:49.7716548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7716638Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7716641Z 2025-09-07T06:56:49.7716745Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7717115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7717187Z layer_outputs = layer_module( 2025-09-07T06:56:49.7717418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7717503Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7717779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7717863Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7718141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7718214Z self_outputs = self.self( 2025-09-07T06:56:49.7718493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7718626Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7718985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7719181Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7719185Z 2025-09-07T06:56:49.7719292Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7719812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7719897Z layer_outputs = layer_module( 2025-09-07T06:56:49.7720121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7720198Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7720490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7720564Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7720856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7720927Z self_outputs = self.self( 2025-09-07T06:56:49.7721202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7721290Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7721294Z 2025-09-07T06:56:49.7721396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7721755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7721825Z layer_outputs = layer_module( 2025-09-07T06:56:49.7722055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7722134Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7722413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7722494Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7722842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7722915Z self_outputs = self.self( 2025-09-07T06:56:49.7723192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7723293Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7723641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7723824Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7723827Z 2025-09-07T06:56:49.7723937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7724290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7724371Z layer_outputs = layer_module( 2025-09-07T06:56:49.7724594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7724673Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7724962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7725039Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7725388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7725461Z self_outputs = self.self( 2025-09-07T06:56:49.7725742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7725852Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7726198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7726388Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7726391Z 2025-09-07T06:56:49.7726495Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7726861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7726931Z layer_outputs = layer_module( 2025-09-07T06:56:49.7727159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7727245Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7727529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7727613Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7727898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7727972Z self_outputs = self.self( 2025-09-07T06:56:49.7728263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7728372Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7728816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7729012Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7729015Z 2025-09-07T06:56:49.7729145Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7729231Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7729313Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7729402Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7729513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7729906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7729980Z layer_outputs = layer_module( 2025-09-07T06:56:49.7730241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7730318Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7730602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7730683Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7730970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7731046Z self_outputs = self.self( 2025-09-07T06:56:49.7731325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7731434Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7731824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7731968Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7732319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7732479Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7732485Z 2025-09-07T06:56:49.7732577Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7732685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7733061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7733145Z layer_outputs = layer_module( 2025-09-07T06:56:49.7733386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7733480Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7733784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7733869Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7734174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7734250Z self_outputs = self.self( 2025-09-07T06:56:49.7734559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7734636Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7734639Z 2025-09-07T06:56:49.7734754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7735135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7735208Z layer_outputs = layer_module( 2025-09-07T06:56:49.7735454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7735536Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7735847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7736274Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7736587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7736660Z self_outputs = self.self( 2025-09-07T06:56:49.7736965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7737059Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7737062Z 2025-09-07T06:56:49.7737172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7737561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7737637Z layer_outputs = layer_module( 2025-09-07T06:56:49.7737876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7737969Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7738269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7738355Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7738656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7738771Z self_outputs = self.self( 2025-09-07T06:56:49.7739081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7739171Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7739175Z 2025-09-07T06:56:49.7739291Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7739689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7739771Z layer_outputs = layer_module( 2025-09-07T06:56:49.7740018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7740096Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7740392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7740466Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7740759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7740827Z self_outputs = self.self( 2025-09-07T06:56:49.7741120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7741243Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7741606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7741788Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7741987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7742097Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7742100Z 2025-09-07T06:56:49.7742201Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7742568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7742675Z layer_outputs = layer_module( 2025-09-07T06:56:49.7742896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7742980Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7743259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7743339Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7743637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7743710Z self_outputs = self.self( 2025-09-07T06:56:49.7744022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7744146Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7744535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7744687Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7745038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7745137Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7745376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7745503Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7745506Z 2025-09-07T06:56:49.7745616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7746056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7746142Z layer_outputs = layer_module( 2025-09-07T06:56:49.7746386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7746471Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7746776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7746866Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7747170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7747250Z self_outputs = self.self( 2025-09-07T06:56:49.7747556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7747679Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7748069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7748222Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7748225Z 2025-09-07T06:56:49.7748336Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7748726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7748810Z layer_outputs = layer_module( 2025-09-07T06:56:49.7749038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7749118Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7749409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7749526Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7749818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7749887Z self_outputs = self.self( 2025-09-07T06:56:49.7750173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7750295Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7750665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7750823Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7750826Z 2025-09-07T06:56:49.7750926Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7751290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7751362Z layer_outputs = layer_module( 2025-09-07T06:56:49.7751583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7751670Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7751987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7752070Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7752359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7752433Z self_outputs = self.self( 2025-09-07T06:56:49.7752721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7752917Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7752920Z 2025-09-07T06:56:49.7753032Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7753394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7753472Z layer_outputs = layer_module( 2025-09-07T06:56:49.7753714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7753803Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7754105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7754183Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7754496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7754616Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7754921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7755014Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7755017Z 2025-09-07T06:56:49.7755131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7755518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7755602Z layer_outputs = layer_module( 2025-09-07T06:56:49.7755832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7755946Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7756239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7756325Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7756595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7756680Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7756974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7757092Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7757384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7757469Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7757482Z 2025-09-07T06:56:49.7757585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7757943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7758022Z layer_outputs = layer_module( 2025-09-07T06:56:49.7758244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7758329Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7758650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7758741Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7759031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7759111Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7759426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7759541Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7759840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7759970Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7760201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7760284Z return self.act(input) 2025-09-07T06:56:49.7760287Z 2025-09-07T06:56:49.7760396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7760777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7760852Z layer_outputs = layer_module( 2025-09-07T06:56:49.7761074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7761162Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7761445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7761534Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7761797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7761880Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7762168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7762292Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7762619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7762701Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7762705Z 2025-09-07T06:56:49.7762815Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7763171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7763242Z layer_outputs = layer_module( 2025-09-07T06:56:49.7763476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7763557Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7763846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7763925Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7764222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7764295Z self_outputs = self.self( 2025-09-07T06:56:49.7764596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-09-07T06:56:49.7764694Z query_vectors = self.query(hidden_states) 2025-09-07T06:56:49.7764697Z 2025-09-07T06:56:49.7764834Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7765223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7765298Z layer_outputs = layer_module( 2025-09-07T06:56:49.7765534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7765626Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7765929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7766021Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7766324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7766405Z self_outputs = self.self( 2025-09-07T06:56:49.7766712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7766823Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7767200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7767399Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7767405Z 2025-09-07T06:56:49.7767523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7767904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7767985Z layer_outputs = layer_module( 2025-09-07T06:56:49.7768222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7768307Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7768619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7768699Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7769006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7769150Z self_outputs = self.self( 2025-09-07T06:56:49.7769453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-09-07T06:56:49.7769544Z key_vectors = self.key(hidden_states) 2025-09-07T06:56:49.7769548Z 2025-09-07T06:56:49.7769658Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7770052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7770127Z layer_outputs = layer_module( 2025-09-07T06:56:49.7770373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7770456Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7770758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7770849Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7771154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7771231Z self_outputs = self.self( 2025-09-07T06:56:49.7771517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7771669Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7772027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7772214Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7772218Z 2025-09-07T06:56:49.7772330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7772691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7772768Z layer_outputs = layer_module( 2025-09-07T06:56:49.7772992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7773070Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7773363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7773441Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7773748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7773821Z self_outputs = self.self( 2025-09-07T06:56:49.7774130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7774241Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7774610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7774811Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7774815Z 2025-09-07T06:56:49.7774928Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7775323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7775395Z layer_outputs = layer_module( 2025-09-07T06:56:49.7775629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7775740Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7776035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7776122Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7776426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7776507Z self_outputs = self.self( 2025-09-07T06:56:49.7776810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-09-07T06:56:49.7776919Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7777296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7777489Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-09-07T06:56:49.7777495Z 2025-09-07T06:56:49.7777593Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7777679Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7777768Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7777851Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7777960Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7778396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7778474Z layer_outputs = layer_module( 2025-09-07T06:56:49.7778718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7778802Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7779105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7779195Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7779499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7779578Z self_outputs = self.self( 2025-09-07T06:56:49.7779881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-09-07T06:56:49.7780001Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-09-07T06:56:49.7780379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-09-07T06:56:49.7780534Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-09-07T06:56:49.7780893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-09-07T06:56:49.7781059Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-09-07T06:56:49.7781063Z 2025-09-07T06:56:49.7781153Z cudagraph partition due to non gpu ops 2025-09-07T06:56:49.7781263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7781645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7781730Z layer_outputs = layer_module( 2025-09-07T06:56:49.7781967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7782060Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7782362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7782496Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7782802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7782874Z self_outputs = self.self( 2025-09-07T06:56:49.7783190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-09-07T06:56:49.7783268Z attn_scores += diagonal_mask 2025-09-07T06:56:49.7783272Z 2025-09-07T06:56:49.7783391Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7783776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7783852Z layer_outputs = layer_module( 2025-09-07T06:56:49.7784099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7784184Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7784500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7784577Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7784895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7784969Z self_outputs = self.self( 2025-09-07T06:56:49.7785309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-09-07T06:56:49.7785404Z attn_probs = nn.functional.softmax( 2025-09-07T06:56:49.7785407Z 2025-09-07T06:56:49.7785517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7785971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7786055Z layer_outputs = layer_module( 2025-09-07T06:56:49.7786302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7786385Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7786701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7786790Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7787111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7787203Z self_outputs = self.self( 2025-09-07T06:56:49.7787514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-09-07T06:56:49.7787605Z value_vectors = self.value(hidden_states) 2025-09-07T06:56:49.7787611Z 2025-09-07T06:56:49.7787731Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7788117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7788201Z layer_outputs = layer_module( 2025-09-07T06:56:49.7788441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7788532Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7788843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7788933Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7789231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7789336Z self_outputs = self.self( 2025-09-07T06:56:49.7789630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7789749Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7790112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7790300Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-09-07T06:56:49.7790499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7790607Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7790610Z 2025-09-07T06:56:49.7790715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7791083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7791157Z layer_outputs = layer_module( 2025-09-07T06:56:49.7791383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7791468Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7791758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7791872Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7792160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7792229Z self_outputs = self.self( 2025-09-07T06:56:49.7792524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7792648Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7793019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7793157Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-09-07T06:56:49.7793492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-09-07T06:56:49.7793588Z chunked_hidden_states = nn.functional.pad( 2025-09-07T06:56:49.7793783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-09-07T06:56:49.7793890Z return torch._C._nn.pad(input, pad, mode, value) 2025-09-07T06:56:49.7793894Z 2025-09-07T06:56:49.7793996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7794379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7794459Z layer_outputs = layer_module( 2025-09-07T06:56:49.7794707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7794790Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7795101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7795191Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7795497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7795579Z self_outputs = self.self( 2025-09-07T06:56:49.7795883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7796042Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7796417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7796572Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7796576Z 2025-09-07T06:56:49.7796688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7797053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7797133Z layer_outputs = layer_module( 2025-09-07T06:56:49.7797357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7797437Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7797733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7797810Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7798104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7798174Z self_outputs = self.self( 2025-09-07T06:56:49.7798494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-09-07T06:56:49.7798610Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-09-07T06:56:49.7798967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-09-07T06:56:49.7799125Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-09-07T06:56:49.7799131Z 2025-09-07T06:56:49.7799236Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7799599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7799672Z layer_outputs = layer_module( 2025-09-07T06:56:49.7799901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7799981Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7800271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7800356Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7800642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-09-07T06:56:49.7800722Z self_outputs = self.self( 2025-09-07T06:56:49.7801006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-09-07T06:56:49.7801197Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-09-07T06:56:49.7801200Z 2025-09-07T06:56:49.7801311Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7801674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7801755Z layer_outputs = layer_module( 2025-09-07T06:56:49.7801979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7802066Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7802351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-09-07T06:56:49.7802462Z self_attn_outputs = self.attention( 2025-09-07T06:56:49.7802760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-09-07T06:56:49.7802874Z attn_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:56:49.7803166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-09-07T06:56:49.7803254Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7803257Z 2025-09-07T06:56:49.7803359Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7803726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7803797Z layer_outputs = layer_module( 2025-09-07T06:56:49.7804036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7804114Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7804415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7804499Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7804762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7804872Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7805154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7805267Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7805544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-09-07T06:56:49.7805637Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7805640Z 2025-09-07T06:56:49.7805742Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7806089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7806167Z layer_outputs = layer_module( 2025-09-07T06:56:49.7806385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7806468Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7806748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7806829Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7807098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7807178Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7807474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-09-07T06:56:49.7807583Z intermediate_output = self.intermediate(attn_output) 2025-09-07T06:56:49.7807874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-09-07T06:56:49.7807991Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:56:49.7808210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:56:49.7808289Z return self.act(input) 2025-09-07T06:56:49.7808292Z 2025-09-07T06:56:49.7808394Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:56:49.7808756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-09-07T06:56:49.7808874Z layer_outputs = layer_module( 2025-09-07T06:56:49.7809097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:56:49.7809182Z return super().__call__(*args, **kwargs) 2025-09-07T06:56:49.7809465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-09-07T06:56:49.7809561Z layer_output = apply_chunking_to_forward( 2025-09-07T06:56:49.7809828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:56:49.7809911Z return forward_fn(*input_tensors) 2025-09-07T06:56:49.7810199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-09-07T06:56:49.7810325Z layer_output = self.output(intermediate_output, attn_output) 2025-09-07T06:56:49.7810621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-09-07T06:56:49.7810707Z hidden_states = self.dense(hidden_states) 2025-09-07T06:56:49.7810710Z 2025-09-07T06:58:00.1091370Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:00.1095465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-09-07T06:58:00.1098105Z prediction_scores = self.lm_head(sequence_output) 2025-09-07T06:58:00.1098646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward 2025-09-07T06:58:00.1099132Z x = self.dense(features) 2025-09-07T06:58:00.1099295Z 2025-09-07T06:58:00.1101861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:00.1102555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-09-07T06:58:00.1103183Z prediction_scores = self.lm_head(sequence_output) 2025-09-07T06:58:00.1103694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward 2025-09-07T06:58:00.1104180Z x = self.decoder(x) 2025-09-07T06:58:00.1104309Z 2025-09-07T06:58:00.1104481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:00.1105077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1723, in torch_dynamo_resume_in_forward_at_1703 2025-09-07T06:58:00.1106087Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T06:58:00.1106399Z 2025-09-07T06:58:01.7273932Z Compilation time (from dynamo_timed): 105.671507196 2025-09-07T06:58:01.7500151Z pass 2025-09-07T06:58:01.7500615Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:58:01.7501512Z TIMING: gc:0.00738 entire_frame_compile:105.67151 _recursive_pre_grad_passes:0.02163 _recursive_joint_graph_passes:1.02661 _recursive_post_grad_passes:1.89618 async_compile.wait:2.91664 code_gen:82.66559 inductor_compile:90.20625 backend_compile:100.1772 total_wall_time:105.67151 2025-09-07T06:58:01.7502732Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:57656 | FakeTensor.__torch_dispatch__:16284 | ProxyTorchDispatchMode.__torch_dispatch__:17446 2025-09-07T06:58:01.7503313Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-09-07T06:58:05.4866337Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:58:05.4867782Z import pynvml # type: ignore[import] 2025-09-07T06:58:08.2551008Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T06:58:08.2551990Z from pkg_resources import resource_filename 2025-09-07T06:58:08.9125320Z 2025-09-07T06:58:11.6204096Z loading model: 0it [00:00, ?it/s] 2025-09-07T06:58:11.6204412Z loading model: 0it [00:02, ?it/s] 2025-09-07T06:58:11.6224045Z cpu eval BartForCausalLM 2025-09-07T06:58:13.3455899Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:58:14.0223369Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:58:14.6770213Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:58:22.3345120Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3345472Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3345947Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3346198Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3346443Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3347760Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3348014Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3348245Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3348484Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3348715Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3348960Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3349208Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3349484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3349914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3350296Z return mod(**inputs) 2025-09-07T06:58:22.3350729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3351161Z outputs = self.model.decoder( 2025-09-07T06:58:22.3351597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3352028Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3352421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3352825Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3353255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3353716Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3354193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3354712Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3354980Z 2025-09-07T06:58:22.3355099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3355502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3355856Z return mod(**inputs) 2025-09-07T06:58:22.3356263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3356694Z outputs = self.model.decoder( 2025-09-07T06:58:22.3357083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3357639Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3358004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3358383Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3358779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3359205Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3359629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3360034Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3360178Z 2025-09-07T06:58:22.3360296Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3360662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3361006Z return mod(**inputs) 2025-09-07T06:58:22.3361379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3361778Z outputs = self.model.decoder( 2025-09-07T06:58:22.3362166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3362550Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3362949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3363329Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3363723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3364145Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3364660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3365069Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3365219Z 2025-09-07T06:58:22.3365305Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3365525Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3365733Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3365946Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3366188Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3366561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3366896Z return mod(**inputs) 2025-09-07T06:58:22.3367288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3367701Z outputs = self.model.decoder( 2025-09-07T06:58:22.3368089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3368489Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3368850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3369255Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3369666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3370084Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3370500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3370911Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3371371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3371903Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3372091Z 2025-09-07T06:58:22.3372203Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3372571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3372899Z return mod(**inputs) 2025-09-07T06:58:22.3373270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3373669Z outputs = self.model.decoder( 2025-09-07T06:58:22.3374062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3374457Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3374838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3375231Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3375629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3376048Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3376461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3376880Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3377367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3377840Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3378005Z 2025-09-07T06:58:22.3378146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3378513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3378864Z return mod(**inputs) 2025-09-07T06:58:22.3379258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3379680Z outputs = self.model.decoder( 2025-09-07T06:58:22.3380086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3380502Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3380876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3381279Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3381698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3382129Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3382566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3382993Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3383141Z 2025-09-07T06:58:22.3383263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3383652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3383995Z return mod(**inputs) 2025-09-07T06:58:22.3384385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3384804Z outputs = self.model.decoder( 2025-09-07T06:58:22.3385217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3385707Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3386104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3386495Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3386964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3387429Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3387617Z 2025-09-07T06:58:22.3387730Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3388125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3388482Z return mod(**inputs) 2025-09-07T06:58:22.3388874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3389290Z outputs = self.model.decoder( 2025-09-07T06:58:22.3389691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3390102Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3390486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3390884Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3391296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3391757Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3392177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3392580Z return self.act(input) 2025-09-07T06:58:22.3392699Z 2025-09-07T06:58:22.3392811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3393172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3393508Z return mod(**inputs) 2025-09-07T06:58:22.3393878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3394278Z outputs = self.model.decoder( 2025-09-07T06:58:22.3394657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3395048Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3395407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3395782Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3396186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3396627Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3396773Z 2025-09-07T06:58:22.3396878Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3397246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3397584Z return mod(**inputs) 2025-09-07T06:58:22.3397946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3398332Z outputs = self.model.decoder( 2025-09-07T06:58:22.3398727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3399142Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3399520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3399902Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3400321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3400764Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3401213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3401739Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3401954Z 2025-09-07T06:58:22.3402064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3402444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3402800Z return mod(**inputs) 2025-09-07T06:58:22.3403193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3403607Z outputs = self.model.decoder( 2025-09-07T06:58:22.3404008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3404437Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3404807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3405213Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3405621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3406058Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3406491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3406913Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3407054Z 2025-09-07T06:58:22.3407204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3407594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3407950Z return mod(**inputs) 2025-09-07T06:58:22.3408330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3408740Z outputs = self.model.decoder( 2025-09-07T06:58:22.3409134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3409521Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3409883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3410245Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3410632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3411032Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3411445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3411849Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3411993Z 2025-09-07T06:58:22.3412083Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3412305Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3412514Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3412728Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3412979Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3413338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3413662Z return mod(**inputs) 2025-09-07T06:58:22.3414031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3414427Z outputs = self.model.decoder( 2025-09-07T06:58:22.3414811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3415200Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3415556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3415980Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3416364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3416772Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3417168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3417574Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3418026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3418509Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3418692Z 2025-09-07T06:58:22.3418806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3419162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3419490Z return mod(**inputs) 2025-09-07T06:58:22.3420083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3420477Z outputs = self.model.decoder( 2025-09-07T06:58:22.3420868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3421295Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3421764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3422160Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3422578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3423012Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3423450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3423891Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3424376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3424874Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3425049Z 2025-09-07T06:58:22.3425162Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3425561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3425978Z return mod(**inputs) 2025-09-07T06:58:22.3426373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3426804Z outputs = self.model.decoder( 2025-09-07T06:58:22.3427218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3427645Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3428027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3428426Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3428837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3429303Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3429745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3430171Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3430317Z 2025-09-07T06:58:22.3430437Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3430817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3431244Z return mod(**inputs) 2025-09-07T06:58:22.3431649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3432080Z outputs = self.model.decoder( 2025-09-07T06:58:22.3432503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3432923Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3433304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3433702Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3434118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3434580Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3434782Z 2025-09-07T06:58:22.3434897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3435287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3435652Z return mod(**inputs) 2025-09-07T06:58:22.3436043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3436456Z outputs = self.model.decoder( 2025-09-07T06:58:22.3436898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3437309Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3437681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3438080Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3438479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3438924Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3439307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3439649Z return self.act(input) 2025-09-07T06:58:22.3439759Z 2025-09-07T06:58:22.3439860Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3440220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3440544Z return mod(**inputs) 2025-09-07T06:58:22.3440900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3441282Z outputs = self.model.decoder( 2025-09-07T06:58:22.3441649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3442038Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3442402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3442768Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3443144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3443533Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3443677Z 2025-09-07T06:58:22.3443784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3444146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3444473Z return mod(**inputs) 2025-09-07T06:58:22.3444825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3445210Z outputs = self.model.decoder( 2025-09-07T06:58:22.3445626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3446007Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3446350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3446702Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3447106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3447514Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3447916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3448363Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3448573Z 2025-09-07T06:58:22.3448676Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3449043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3449370Z return mod(**inputs) 2025-09-07T06:58:22.3449727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3450104Z outputs = self.model.decoder( 2025-09-07T06:58:22.3450482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3450892Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3451243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3451614Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3452004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3452433Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3452851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3453257Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3453394Z 2025-09-07T06:58:22.3453509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3453881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3454211Z return mod(**inputs) 2025-09-07T06:58:22.3454578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3454965Z outputs = self.model.decoder( 2025-09-07T06:58:22.3455339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3468041Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3468511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3468907Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3469317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3469754Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3470178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3470593Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3470743Z 2025-09-07T06:58:22.3470838Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3471053Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3471265Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3471474Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3471718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3472206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3472550Z return mod(**inputs) 2025-09-07T06:58:22.3472927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3473328Z outputs = self.model.decoder( 2025-09-07T06:58:22.3473717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3474097Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3474464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3474839Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3475245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3475673Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3476101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3476501Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3476951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3477435Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3477673Z 2025-09-07T06:58:22.3477790Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3478153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3478485Z return mod(**inputs) 2025-09-07T06:58:22.3478856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3479252Z outputs = self.model.decoder( 2025-09-07T06:58:22.3479635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3480005Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3480353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3480720Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3481112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3481518Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3481914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3482313Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3482763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3483224Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3483385Z 2025-09-07T06:58:22.3483497Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3483859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3484178Z return mod(**inputs) 2025-09-07T06:58:22.3484534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3484909Z outputs = self.model.decoder( 2025-09-07T06:58:22.3485273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3485654Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3485992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3486379Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3486755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3487140Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3487530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3487913Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3488049Z 2025-09-07T06:58:22.3488160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3488504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3488818Z return mod(**inputs) 2025-09-07T06:58:22.3489166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3489547Z outputs = self.model.decoder( 2025-09-07T06:58:22.3489913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3490275Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3490613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3490963Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3491363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3491785Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3491957Z 2025-09-07T06:58:22.3492060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3492415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3492743Z return mod(**inputs) 2025-09-07T06:58:22.3493099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3493470Z outputs = self.model.decoder( 2025-09-07T06:58:22.3493838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3494214Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3494569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3494940Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3495327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3495768Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3496156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3496496Z return self.act(input) 2025-09-07T06:58:22.3496605Z 2025-09-07T06:58:22.3496716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3497071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3497386Z return mod(**inputs) 2025-09-07T06:58:22.3497736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3498113Z outputs = self.model.decoder( 2025-09-07T06:58:22.3498470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3498842Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3499191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3499553Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3500016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3500391Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3500531Z 2025-09-07T06:58:22.3500631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3500982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3501319Z return mod(**inputs) 2025-09-07T06:58:22.3501688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3502087Z outputs = self.model.decoder( 2025-09-07T06:58:22.3502477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3502877Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3503236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3503608Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3504006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3504427Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3504860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3505397Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3505710Z 2025-09-07T06:58:22.3505835Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3506235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3506594Z return mod(**inputs) 2025-09-07T06:58:22.3507002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3507416Z outputs = self.model.decoder( 2025-09-07T06:58:22.3507805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3508200Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3508561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3508937Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3509326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3509743Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3510156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3510553Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3510698Z 2025-09-07T06:58:22.3510811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3511174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3511508Z return mod(**inputs) 2025-09-07T06:58:22.3511876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3512272Z outputs = self.model.decoder( 2025-09-07T06:58:22.3512652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3513041Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3513400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3513774Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3514170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3514631Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3515041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3515444Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3515587Z 2025-09-07T06:58:22.3515677Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3515909Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3516114Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3516320Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3516554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3516909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3517225Z return mod(**inputs) 2025-09-07T06:58:22.3517588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3517972Z outputs = self.model.decoder( 2025-09-07T06:58:22.3518351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3518724Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3519077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3519471Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3520014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3520427Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3520828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3521238Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3521697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3522177Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3522362Z 2025-09-07T06:58:22.3522475Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3522832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3523170Z return mod(**inputs) 2025-09-07T06:58:22.3523533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3523917Z outputs = self.model.decoder( 2025-09-07T06:58:22.3524288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3524671Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3525019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3525389Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3525808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3526238Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3526656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3527070Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3527536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3527995Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3528156Z 2025-09-07T06:58:22.3528352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3528717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3529041Z return mod(**inputs) 2025-09-07T06:58:22.3529404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3529788Z outputs = self.model.decoder( 2025-09-07T06:58:22.3530165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3530548Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3530896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3531258Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3531635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3532047Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3532444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3532842Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3532981Z 2025-09-07T06:58:22.3533094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3533453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3533843Z return mod(**inputs) 2025-09-07T06:58:22.3534234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3534656Z outputs = self.model.decoder( 2025-09-07T06:58:22.3535044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3535446Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3535805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3536177Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3536576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3537015Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3537195Z 2025-09-07T06:58:22.3537302Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3537661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3537987Z return mod(**inputs) 2025-09-07T06:58:22.3538352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3538731Z outputs = self.model.decoder( 2025-09-07T06:58:22.3539115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3539498Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3539849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3540214Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3540618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3541064Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3541471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3541829Z return self.act(input) 2025-09-07T06:58:22.3541948Z 2025-09-07T06:58:22.3542061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3542454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3542844Z return mod(**inputs) 2025-09-07T06:58:22.3543234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3543629Z outputs = self.model.decoder( 2025-09-07T06:58:22.3544005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3544396Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3544751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3545120Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3545506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3545978Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3546142Z 2025-09-07T06:58:22.3546258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3546662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3547025Z return mod(**inputs) 2025-09-07T06:58:22.3547417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3547811Z outputs = self.model.decoder( 2025-09-07T06:58:22.3548222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3548607Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3548953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3549309Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3549690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3550096Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3550494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3550974Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3551202Z 2025-09-07T06:58:22.3551314Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3551703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3552050Z return mod(**inputs) 2025-09-07T06:58:22.3552434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3552840Z outputs = self.model.decoder( 2025-09-07T06:58:22.3553222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3553617Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3553973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3554343Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3554739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3555176Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3555612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3556032Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3556177Z 2025-09-07T06:58:22.3556297Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3556676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3557072Z return mod(**inputs) 2025-09-07T06:58:22.3557462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3557885Z outputs = self.model.decoder( 2025-09-07T06:58:22.3558289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3558704Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3559086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3559478Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3559892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3560326Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3560764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3561195Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3561348Z 2025-09-07T06:58:22.3561441Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3561669Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3561901Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3562126Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3562382Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3562819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3563178Z return mod(**inputs) 2025-09-07T06:58:22.3563569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3563987Z outputs = self.model.decoder( 2025-09-07T06:58:22.3564401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3564818Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3565195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3565593Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3565981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3566389Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3566788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3567193Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3567644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3568132Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3568318Z 2025-09-07T06:58:22.3568432Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3568797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3569129Z return mod(**inputs) 2025-09-07T06:58:22.3569498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3569892Z outputs = self.model.decoder( 2025-09-07T06:58:22.3570279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3570665Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3571012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3571375Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3571800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3572195Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3572594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3572990Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3573437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3573898Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3574056Z 2025-09-07T06:58:22.3574159Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3574516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3574842Z return mod(**inputs) 2025-09-07T06:58:22.3575212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3575608Z outputs = self.model.decoder( 2025-09-07T06:58:22.3575985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3576368Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3576721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3577118Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3577494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3577896Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3578295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3578687Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3578823Z 2025-09-07T06:58:22.3578935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3579306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3579660Z return mod(**inputs) 2025-09-07T06:58:22.3580049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3580470Z outputs = self.model.decoder( 2025-09-07T06:58:22.3580874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3581297Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3581672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3582073Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3582488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3582940Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3583133Z 2025-09-07T06:58:22.3583245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3583634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3583982Z return mod(**inputs) 2025-09-07T06:58:22.3584363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3584776Z outputs = self.model.decoder( 2025-09-07T06:58:22.3585179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3585589Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3586058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3586492Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3586914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3587381Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3587810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3588189Z return self.act(input) 2025-09-07T06:58:22.3588316Z 2025-09-07T06:58:22.3588431Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3588827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3589182Z return mod(**inputs) 2025-09-07T06:58:22.3589572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3589998Z outputs = self.model.decoder( 2025-09-07T06:58:22.3590410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3590830Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3591221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3591626Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3592084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3592512Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3592670Z 2025-09-07T06:58:22.3592782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3593171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3593527Z return mod(**inputs) 2025-09-07T06:58:22.3593905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3594301Z outputs = self.model.decoder( 2025-09-07T06:58:22.3594709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3595127Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3595477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3595848Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3596245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3596664Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3597074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3597543Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3597758Z 2025-09-07T06:58:22.3597865Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3598234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3598570Z return mod(**inputs) 2025-09-07T06:58:22.3598933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3599335Z outputs = self.model.decoder( 2025-09-07T06:58:22.3599723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3600117Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3600475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3600882Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3601279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3601692Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3602103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3602501Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3602639Z 2025-09-07T06:58:22.3602747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3603113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3603445Z return mod(**inputs) 2025-09-07T06:58:22.3603809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3604194Z outputs = self.model.decoder( 2025-09-07T06:58:22.3604576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3604965Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3605321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3605686Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3606295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3606726Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3607138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3607540Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3607692Z 2025-09-07T06:58:22.3607773Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3607985Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3608188Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3608397Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3608797Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3609164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3609498Z return mod(**inputs) 2025-09-07T06:58:22.3609872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3610269Z outputs = self.model.decoder( 2025-09-07T06:58:22.3610649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3611036Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3611398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3611771Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3612153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3612566Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3612968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3613381Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3613841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3614325Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3614517Z 2025-09-07T06:58:22.3614621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3614982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3615353Z return mod(**inputs) 2025-09-07T06:58:22.3615719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3616104Z outputs = self.model.decoder( 2025-09-07T06:58:22.3616488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3616880Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3617237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3617603Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3617996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3618424Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3618841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3619251Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3619900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3620384Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3620564Z 2025-09-07T06:58:22.3620678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3621151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3621506Z return mod(**inputs) 2025-09-07T06:58:22.3621889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3622308Z outputs = self.model.decoder( 2025-09-07T06:58:22.3622720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3623151Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3623537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3623939Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3624358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3624804Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3625253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3625750Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3625914Z 2025-09-07T06:58:22.3626032Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3626432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3626805Z return mod(**inputs) 2025-09-07T06:58:22.3627204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3627617Z outputs = self.model.decoder( 2025-09-07T06:58:22.3628024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3628436Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3628816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3629201Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3629620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3630080Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3630328Z 2025-09-07T06:58:22.3630448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3630833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3631173Z return mod(**inputs) 2025-09-07T06:58:22.3631562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3631979Z outputs = self.model.decoder( 2025-09-07T06:58:22.3632393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3632807Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3633178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3633569Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3633992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3634454Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3634870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3635243Z return self.act(input) 2025-09-07T06:58:22.3635370Z 2025-09-07T06:58:22.3635480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3635868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3636263Z return mod(**inputs) 2025-09-07T06:58:22.3636654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3637070Z outputs = self.model.decoder( 2025-09-07T06:58:22.3637478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3637896Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3638278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3638668Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3639065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3639498Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3639657Z 2025-09-07T06:58:22.3639772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3640136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3640471Z return mod(**inputs) 2025-09-07T06:58:22.3640837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3641229Z outputs = self.model.decoder( 2025-09-07T06:58:22.3641617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3642000Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3642354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3642722Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3643115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3643530Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3643950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3644427Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3644636Z 2025-09-07T06:58:22.3644750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3645165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3645493Z return mod(**inputs) 2025-09-07T06:58:22.3645862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3646256Z outputs = self.model.decoder( 2025-09-07T06:58:22.3646644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3647035Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3647385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3647753Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3648143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3648558Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3648963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3649360Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3649505Z 2025-09-07T06:58:22.3649608Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3649971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3650300Z return mod(**inputs) 2025-09-07T06:58:22.3650695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3651087Z outputs = self.model.decoder( 2025-09-07T06:58:22.3651469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3651858Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3652219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3652593Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3652985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3653397Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3653809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3654209Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3654359Z 2025-09-07T06:58:22.3654442Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3654661Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3654876Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3655089Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3655325Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3655693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3656027Z return mod(**inputs) 2025-09-07T06:58:22.3656394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3656772Z outputs = self.model.decoder( 2025-09-07T06:58:22.3657149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3657529Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3657878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3658238Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3658612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3659051Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3659458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3659878Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3660330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3660838Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3661042Z 2025-09-07T06:58:22.3661158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3661541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3661874Z return mod(**inputs) 2025-09-07T06:58:22.3662232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3662630Z outputs = self.model.decoder( 2025-09-07T06:58:22.3663017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3663431Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3663816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3664209Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3665331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3665876Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3666323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3666754Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3667237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3667737Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3667920Z 2025-09-07T06:58:22.3668025Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3668381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3668699Z return mod(**inputs) 2025-09-07T06:58:22.3669060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3669446Z outputs = self.model.decoder( 2025-09-07T06:58:22.3669821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3670203Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3670543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3670910Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3671291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3671694Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3672088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3672478Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3672622Z 2025-09-07T06:58:22.3672726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3673090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3673408Z return mod(**inputs) 2025-09-07T06:58:22.3673749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3674183Z outputs = self.model.decoder( 2025-09-07T06:58:22.3674556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3674941Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3675290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3675650Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3676023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3676439Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3676608Z 2025-09-07T06:58:22.3676720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3677064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3677377Z return mod(**inputs) 2025-09-07T06:58:22.3677726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3678097Z outputs = self.model.decoder( 2025-09-07T06:58:22.3678461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3678821Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3679161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3679547Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3679925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3680342Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3680710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3681045Z return self.act(input) 2025-09-07T06:58:22.3681160Z 2025-09-07T06:58:22.3681261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3681612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3681923Z return mod(**inputs) 2025-09-07T06:58:22.3682272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3682642Z outputs = self.model.decoder( 2025-09-07T06:58:22.3683010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3683381Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3683714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3684068Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3684447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3684824Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3684960Z 2025-09-07T06:58:22.3685068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3685416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3685740Z return mod(**inputs) 2025-09-07T06:58:22.3686102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3686486Z outputs = self.model.decoder( 2025-09-07T06:58:22.3686865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3687254Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3687594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3687982Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3688357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3688754Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3689159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3689622Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3689832Z 2025-09-07T06:58:22.3689946Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3690320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3690646Z return mod(**inputs) 2025-09-07T06:58:22.3691022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3691416Z outputs = self.model.decoder( 2025-09-07T06:58:22.3691795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3692174Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3692522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3692884Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3693307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3693711Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3694109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3694501Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3694644Z 2025-09-07T06:58:22.3694747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3695102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3695418Z return mod(**inputs) 2025-09-07T06:58:22.3695779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3696162Z outputs = self.model.decoder( 2025-09-07T06:58:22.3696545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3696988Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3697328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3697689Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3698071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3698480Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3698885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3699274Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3699421Z 2025-09-07T06:58:22.3699503Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3699719Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3699932Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3700132Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3700365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3700724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3701058Z return mod(**inputs) 2025-09-07T06:58:22.3701415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3701843Z outputs = self.model.decoder( 2025-09-07T06:58:22.3702227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3702619Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3702989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3703386Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3703793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3704210Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3704626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3705048Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3705520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3706156Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3706376Z 2025-09-07T06:58:22.3706502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3706894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3707256Z return mod(**inputs) 2025-09-07T06:58:22.3707692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3708130Z outputs = self.model.decoder( 2025-09-07T06:58:22.3708556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3708985Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3709373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3709788Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3710218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3710673Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3711126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3711567Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3712062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3712577Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3712755Z 2025-09-07T06:58:22.3712882Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3713280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3713653Z return mod(**inputs) 2025-09-07T06:58:22.3714059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3714489Z outputs = self.model.decoder( 2025-09-07T06:58:22.3714908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3715332Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3715690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3716059Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3716451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3716941Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3717342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3717739Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3717881Z 2025-09-07T06:58:22.3717989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3718351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3718684Z return mod(**inputs) 2025-09-07T06:58:22.3719041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3719430Z outputs = self.model.decoder( 2025-09-07T06:58:22.3719937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3720338Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3720692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3721062Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3721458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3721894Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3722071Z 2025-09-07T06:58:22.3722184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3722649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3722970Z return mod(**inputs) 2025-09-07T06:58:22.3723323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3723698Z outputs = self.model.decoder( 2025-09-07T06:58:22.3724060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3724432Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3724774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3725128Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3725507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3725918Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3726294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3726628Z return self.act(input) 2025-09-07T06:58:22.3726734Z 2025-09-07T06:58:22.3726844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3727192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3727505Z return mod(**inputs) 2025-09-07T06:58:22.3727854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3728233Z outputs = self.model.decoder( 2025-09-07T06:58:22.3728598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3728962Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3729302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3729655Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3730027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3730409Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3730608Z 2025-09-07T06:58:22.3730708Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3731058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3731372Z return mod(**inputs) 2025-09-07T06:58:22.3731720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3732090Z outputs = self.model.decoder( 2025-09-07T06:58:22.3732449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3732820Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3733156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3733506Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3733867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3734259Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3734647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3735090Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3735292Z 2025-09-07T06:58:22.3735400Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3735779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3736104Z return mod(**inputs) 2025-09-07T06:58:22.3736468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3736842Z outputs = self.model.decoder( 2025-09-07T06:58:22.3737205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3737572Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3737911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3738260Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3738633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3739020Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3739414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3739791Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3739922Z 2025-09-07T06:58:22.3740031Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3740381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3740696Z return mod(**inputs) 2025-09-07T06:58:22.3741046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3741420Z outputs = self.model.decoder( 2025-09-07T06:58:22.3741788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3742151Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3742502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3742863Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3743245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3743650Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3744044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3744475Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3744646Z 2025-09-07T06:58:22.3744728Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3744945Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3745153Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3745351Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3745588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3746030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3746100Z return mod(**inputs) 2025-09-07T06:58:22.3746367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3746441Z outputs = self.model.decoder( 2025-09-07T06:58:22.3746695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3746783Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3747023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3747117Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3747394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3747504Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3747796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3747901Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3748212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3748348Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3748355Z 2025-09-07T06:58:22.3748469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3748676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3748744Z return mod(**inputs) 2025-09-07T06:58:22.3749015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3749087Z outputs = self.model.decoder( 2025-09-07T06:58:22.3749343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3749414Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3749647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3749727Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3749977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3750080Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3750323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3750426Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3750720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3750829Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3750832Z 2025-09-07T06:58:22.3750940Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3751184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3751257Z return mod(**inputs) 2025-09-07T06:58:22.3751540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3751758Z outputs = self.model.decoder( 2025-09-07T06:58:22.3752011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3752083Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3752310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3752394Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3752644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3752740Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3752981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3753076Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3753079Z 2025-09-07T06:58:22.3753182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3753387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3753453Z return mod(**inputs) 2025-09-07T06:58:22.3753701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3753781Z outputs = self.model.decoder( 2025-09-07T06:58:22.3754060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3754141Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3754360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3754448Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3754695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3754814Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3754817Z 2025-09-07T06:58:22.3754927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3755123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3755197Z return mod(**inputs) 2025-09-07T06:58:22.3755445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3755518Z outputs = self.model.decoder( 2025-09-07T06:58:22.3755773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3755844Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3756067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3756149Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3756395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3756520Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3756729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3756804Z return self.act(input) 2025-09-07T06:58:22.3756810Z 2025-09-07T06:58:22.3756910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3757114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3757179Z return mod(**inputs) 2025-09-07T06:58:22.3757422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3757533Z outputs = self.model.decoder( 2025-09-07T06:58:22.3757786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3757865Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3758088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3758166Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3758424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3758504Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3758508Z 2025-09-07T06:58:22.3758616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3758814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3758885Z return mod(**inputs) 2025-09-07T06:58:22.3759140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3759210Z outputs = self.model.decoder( 2025-09-07T06:58:22.3759467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3759536Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3759765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3759886Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3760138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3760242Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3760499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3760659Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3760662Z 2025-09-07T06:58:22.3760760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3760962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3761026Z return mod(**inputs) 2025-09-07T06:58:22.3761276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3761359Z outputs = self.model.decoder( 2025-09-07T06:58:22.3761611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3761687Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3761914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3761998Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3762260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3762364Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3762642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3762730Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3762734Z 2025-09-07T06:58:22.3762849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3763076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3763145Z return mod(**inputs) 2025-09-07T06:58:22.3763431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3763506Z outputs = self.model.decoder( 2025-09-07T06:58:22.3763821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3763896Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3764133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3764223Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3764501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3764615Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3764872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3764955Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3764958Z 2025-09-07T06:58:22.3765048Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3765129Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3765212Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3765288Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3765390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3765594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3765659Z return mod(**inputs) 2025-09-07T06:58:22.3765921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3766034Z outputs = self.model.decoder( 2025-09-07T06:58:22.3766306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3766387Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3766624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3766717Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3766986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3767099Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3767376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3767481Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3767816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3767952Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3767956Z 2025-09-07T06:58:22.3768066Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3768267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3768348Z return mod(**inputs) 2025-09-07T06:58:22.3768605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3768674Z outputs = self.model.decoder( 2025-09-07T06:58:22.3768925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3768996Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3769223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3769301Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3769551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3769656Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3769903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3770042Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3770341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3770453Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3770464Z 2025-09-07T06:58:22.3770568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3770773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3770847Z return mod(**inputs) 2025-09-07T06:58:22.3771101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3771182Z outputs = self.model.decoder( 2025-09-07T06:58:22.3771433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3771513Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3771766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3771854Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3772134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3772242Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3772549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3772650Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3772653Z 2025-09-07T06:58:22.3772767Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3772997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3773073Z return mod(**inputs) 2025-09-07T06:58:22.3773351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3773438Z outputs = self.model.decoder( 2025-09-07T06:58:22.3773716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3773803Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3774057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3774148Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3774418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3774547Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3774553Z 2025-09-07T06:58:22.3774670Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3774882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3774958Z return mod(**inputs) 2025-09-07T06:58:22.3775234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3775312Z outputs = self.model.decoder( 2025-09-07T06:58:22.3775592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3775668Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3775912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3775997Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3776271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3776431Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3776657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3776746Z return self.act(input) 2025-09-07T06:58:22.3776749Z 2025-09-07T06:58:22.3776853Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3777054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3777122Z return mod(**inputs) 2025-09-07T06:58:22.3777368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3777449Z outputs = self.model.decoder( 2025-09-07T06:58:22.3777692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3777776Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3778013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3778098Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3778370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3778458Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3778462Z 2025-09-07T06:58:22.3778579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3778824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3778904Z return mod(**inputs) 2025-09-07T06:58:22.3779169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3779246Z outputs = self.model.decoder( 2025-09-07T06:58:22.3779521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3779596Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3779837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3779922Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3780183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3780297Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3780559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3780726Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3780730Z 2025-09-07T06:58:22.3780840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3781062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3781130Z return mod(**inputs) 2025-09-07T06:58:22.3781396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3781480Z outputs = self.model.decoder( 2025-09-07T06:58:22.3781745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3781827Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3782064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3782149Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3782415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3782519Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3782826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3782913Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3782916Z 2025-09-07T06:58:22.3783026Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3783243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3783312Z return mod(**inputs) 2025-09-07T06:58:22.3783589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3783670Z outputs = self.model.decoder( 2025-09-07T06:58:22.3783950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3784028Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3784274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3784368Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3784638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3784750Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3785027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3785162Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3785166Z 2025-09-07T06:58:22.3785261Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3785347Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3785439Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3785520Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3785696Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3785938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3786009Z return mod(**inputs) 2025-09-07T06:58:22.3786293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3786372Z outputs = self.model.decoder( 2025-09-07T06:58:22.3786649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3786740Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3786986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3787084Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3787357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3787478Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3787755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3787858Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3788179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3788323Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3788327Z 2025-09-07T06:58:22.3788448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3788663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3788736Z return mod(**inputs) 2025-09-07T06:58:22.3789017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3789135Z outputs = self.model.decoder( 2025-09-07T06:58:22.3789407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3789481Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3789724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3789808Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3790076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3790189Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3790452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3790560Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3790874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3790996Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3791007Z 2025-09-07T06:58:22.3791115Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3791326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3791403Z return mod(**inputs) 2025-09-07T06:58:22.3791700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3791788Z outputs = self.model.decoder( 2025-09-07T06:58:22.3792059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3792135Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3792387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3792474Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3792746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3792848Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3793112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3793208Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3793212Z 2025-09-07T06:58:22.3793324Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3793548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3793619Z return mod(**inputs) 2025-09-07T06:58:22.3793871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3793955Z outputs = self.model.decoder( 2025-09-07T06:58:22.3794208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3794288Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3794509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3794599Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3794865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3794993Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3794996Z 2025-09-07T06:58:22.3795113Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3795324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3795401Z return mod(**inputs) 2025-09-07T06:58:22.3795704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3795780Z outputs = self.model.decoder( 2025-09-07T06:58:22.3796051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3796125Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3796368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3796465Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3796726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3796843Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3797057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3797136Z return self.act(input) 2025-09-07T06:58:22.3797139Z 2025-09-07T06:58:22.3797242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3797449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3797515Z return mod(**inputs) 2025-09-07T06:58:22.3797765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3797847Z outputs = self.model.decoder( 2025-09-07T06:58:22.3798127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3798210Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3798433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3798513Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3798770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3798852Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3798856Z 2025-09-07T06:58:22.3798968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3799167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3799241Z return mod(**inputs) 2025-09-07T06:58:22.3799495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3799569Z outputs = self.model.decoder( 2025-09-07T06:58:22.3799830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3799903Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3800135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3800217Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3800467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3800576Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3800822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:58:22.3800984Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:58:22.3800987Z 2025-09-07T06:58:22.3801089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3801297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3801363Z return mod(**inputs) 2025-09-07T06:58:22.3801616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3801729Z outputs = self.model.decoder( 2025-09-07T06:58:22.3801989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3802069Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3802298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3802378Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3802646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3802745Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3803007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:58:22.3803089Z key_states = self.k_proj(current_states) 2025-09-07T06:58:22.3803095Z 2025-09-07T06:58:22.3803206Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3803411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3803477Z return mod(**inputs) 2025-09-07T06:58:22.3803746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3803817Z outputs = self.model.decoder( 2025-09-07T06:58:22.3804115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3804189Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3804410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3804498Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3804747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3804856Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3805107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:58:22.3805192Z value_states = self.v_proj(current_states) 2025-09-07T06:58:22.3805196Z 2025-09-07T06:58:22.3805284Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3805363Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3805451Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3805528Z cudagraph partition due to non gpu ops 2025-09-07T06:58:22.3805630Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3805842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3805907Z return mod(**inputs) 2025-09-07T06:58:22.3806169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3806243Z outputs = self.model.decoder( 2025-09-07T06:58:22.3806497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3806577Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3806800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3806888Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3807144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3807251Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3807505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3807601Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3807958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:58:22.3808090Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:58:22.3808093Z 2025-09-07T06:58:22.3808199Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3808394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3808457Z return mod(**inputs) 2025-09-07T06:58:22.3808712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3808784Z outputs = self.model.decoder( 2025-09-07T06:58:22.3809035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3809104Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3809333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3809410Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3809651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3809755Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3809994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:58:22.3810129Z attn_output, attn_weights = attention_interface( 2025-09-07T06:58:22.3810418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:58:22.3810525Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:58:22.3810537Z 2025-09-07T06:58:22.3810637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3810835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3810907Z return mod(**inputs) 2025-09-07T06:58:22.3811150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3811229Z outputs = self.model.decoder( 2025-09-07T06:58:22.3811473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3811546Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3811776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3811853Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3812101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:58:22.3812200Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:58:22.3812443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:58:22.3812536Z attn_output = self.out_proj(attn_output) 2025-09-07T06:58:22.3812539Z 2025-09-07T06:58:22.3812641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3812849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3812915Z return mod(**inputs) 2025-09-07T06:58:22.3813181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3813254Z outputs = self.model.decoder( 2025-09-07T06:58:22.3813505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3813586Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3813846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3813932Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3814182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3814301Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3814305Z 2025-09-07T06:58:22.3814420Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3814634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3814712Z return mod(**inputs) 2025-09-07T06:58:22.3814978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3815055Z outputs = self.model.decoder( 2025-09-07T06:58:22.3815326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3815405Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3815654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3815732Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3815990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:58:22.3816141Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:58:22.3816358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:58:22.3816436Z return self.act(input) 2025-09-07T06:58:22.3816440Z 2025-09-07T06:58:22.3816543Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3816751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3816821Z return mod(**inputs) 2025-09-07T06:58:22.3817074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-09-07T06:58:22.3817156Z outputs = self.model.decoder( 2025-09-07T06:58:22.3817409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:58:22.3817487Z layer_outputs = decoder_layer( 2025-09-07T06:58:22.3817712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:58:22.3817792Z return super().__call__(*args, **kwargs) 2025-09-07T06:58:22.3818060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:58:22.3818140Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:58:22.3818143Z 2025-09-07T06:58:22.3818255Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3818452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3818522Z return mod(**inputs) 2025-09-07T06:58:22.3818772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1917, in forward 2025-09-07T06:58:22.3818851Z logits = self.lm_head(outputs[0]) 2025-09-07T06:58:22.3818854Z 2025-09-07T06:58:22.3818964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:58:22.3819167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:58:22.3819241Z return mod(**inputs) 2025-09-07T06:58:22.3819493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1923, in forward 2025-09-07T06:58:22.3819800Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T06:58:22.3819885Z 2025-09-07T06:58:34.6959516Z Compilation time (from dynamo_timed): 18.105278935 2025-09-07T06:58:34.7161500Z pass 2025-09-07T06:58:34.7161898Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:58:34.7162793Z TIMING: _recursive_pre_grad_passes:0.00781 _recursive_joint_graph_passes:0.3671 _recursive_post_grad_passes:0.0766 async_compile.wait:0.7952 code_gen:11.2294 inductor_compile:12.54363 backend_compile:15.81876 gc:0.00156 entire_frame_compile:18.10528 total_wall_time:18.10528 2025-09-07T06:58:34.7163862Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:13192 | FakeTensor.__torch_dispatch__:4538 | ProxyTorchDispatchMode.__torch_dispatch__:4813 2025-09-07T06:58:34.7164376Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2025-09-07T06:58:37.4790093Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:58:37.4790971Z import pynvml # type: ignore[import] 2025-09-07T06:58:40.2947174Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T06:58:40.2948450Z from pkg_resources import resource_filename 2025-09-07T06:58:40.9966280Z 2025-09-07T06:58:46.1136280Z loading model: 0it [00:00, ?it/s] 2025-09-07T06:58:46.1140581Z loading model: 0it [00:05, ?it/s] 2025-09-07T06:58:46.1163062Z cpu eval BartForConditionalGeneration 2025-09-07T06:58:49.5022714Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:58:50.7767587Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:58:52.0386596Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:59:09.3785376Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3791211Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3791555Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3792493Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3793225Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3793587Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3793833Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3794059Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3794293Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3794539Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3794764Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3794997Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3795269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3795705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3796090Z return mod(**inputs) 2025-09-07T06:59:09.3796534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3796994Z outputs = self.model( 2025-09-07T06:59:09.3797438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3797888Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3798326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3798764Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3799166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3800053Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3800499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3800971Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3801427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.3802106Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.3802350Z 2025-09-07T06:59:09.3802473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3802883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3803246Z return mod(**inputs) 2025-09-07T06:59:09.3803664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3804104Z outputs = self.model( 2025-09-07T06:59:09.3804532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3805030Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3805448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3805871Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3806384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3806801Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3807254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3807717Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3808178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.3808635Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.3808800Z 2025-09-07T06:59:09.3808937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3809350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3809712Z return mod(**inputs) 2025-09-07T06:59:09.3810159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3810592Z outputs = self.model( 2025-09-07T06:59:09.3810994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3811440Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3811875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3812326Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3812758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3813180Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3813614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3814078Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3814548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.3814992Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.3815158Z 2025-09-07T06:59:09.3815249Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3815488Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3815722Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3816004Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3816274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3816691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3817049Z return mod(**inputs) 2025-09-07T06:59:09.3817432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3817850Z outputs = self.model( 2025-09-07T06:59:09.3818248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3818685Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3819105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3819526Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3820145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3820564Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3820998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3821464Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3821906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.3822437Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.3822954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.3823498Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.3823711Z 2025-09-07T06:59:09.3823835Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3824238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3824611Z return mod(**inputs) 2025-09-07T06:59:09.3825049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3825551Z outputs = self.model( 2025-09-07T06:59:09.3826265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3826714Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3827153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3827574Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3827950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3828351Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3828770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3829214Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3829646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.3830098Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.3830595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.3831111Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.3831292Z 2025-09-07T06:59:09.3831416Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3831808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3832177Z return mod(**inputs) 2025-09-07T06:59:09.3832649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3833064Z outputs = self.model( 2025-09-07T06:59:09.3833460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3833884Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3834302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3834725Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3835109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3835511Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3835925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3836360Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3836795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.3837237Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.3837389Z 2025-09-07T06:59:09.3837516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3837917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3838288Z return mod(**inputs) 2025-09-07T06:59:09.3838735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3839169Z outputs = self.model( 2025-09-07T06:59:09.3839574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3840006Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3840434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3840861Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3841256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3841658Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3842090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.3842579Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.3842775Z 2025-09-07T06:59:09.3842904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3843308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3843665Z return mod(**inputs) 2025-09-07T06:59:09.3844076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3844509Z outputs = self.model( 2025-09-07T06:59:09.3844913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3845354Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3845768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3846196Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3846595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3847000Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3847421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.3847915Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.3848420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.3848803Z return self.act(input) 2025-09-07T06:59:09.3848929Z 2025-09-07T06:59:09.3849049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3849442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3849810Z return mod(**inputs) 2025-09-07T06:59:09.3850219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3850645Z outputs = self.model( 2025-09-07T06:59:09.3851042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3851481Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3851900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3852332Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3852717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3853119Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3853530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.3853949Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.3854135Z 2025-09-07T06:59:09.3854258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3854655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3855010Z return mod(**inputs) 2025-09-07T06:59:09.3855405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3855827Z outputs = self.model( 2025-09-07T06:59:09.3856222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3856649Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3857071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3857493Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3857888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3858307Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3858741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3859205Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3859668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.3860206Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.3860436Z 2025-09-07T06:59:09.3860561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3860977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3861336Z return mod(**inputs) 2025-09-07T06:59:09.3861779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3862221Z outputs = self.model( 2025-09-07T06:59:09.3862635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3863081Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3863511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3863977Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3864386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3864793Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3865235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3865861Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3866331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.3866783Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.3866946Z 2025-09-07T06:59:09.3867062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3867458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3867821Z return mod(**inputs) 2025-09-07T06:59:09.3868229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3868645Z outputs = self.model( 2025-09-07T06:59:09.3869046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3869476Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3870806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3871234Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3871616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3872020Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3872447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3872894Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3873403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.3873859Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.3874027Z 2025-09-07T06:59:09.3874117Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3874362Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3874598Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3874825Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3875089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3875497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3875878Z return mod(**inputs) 2025-09-07T06:59:09.3876298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3876737Z outputs = self.model( 2025-09-07T06:59:09.3877146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3877582Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3878007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3878427Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3878822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3879231Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3879659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3880122Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3880569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.3881078Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.3881589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.3882139Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.3882350Z 2025-09-07T06:59:09.3882467Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3882877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3883243Z return mod(**inputs) 2025-09-07T06:59:09.3883648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3884080Z outputs = self.model( 2025-09-07T06:59:09.3884495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3884930Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3885353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3885799Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3886188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3886589Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3887053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3887506Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3887950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.3888395Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.3888901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.3889419Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.3889600Z 2025-09-07T06:59:09.3889727Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3890134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3890490Z return mod(**inputs) 2025-09-07T06:59:09.3890898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3891327Z outputs = self.model( 2025-09-07T06:59:09.3891730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3892170Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3892585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3893021Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3893414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3893831Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3894259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3894712Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3895154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.3895602Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.3895755Z 2025-09-07T06:59:09.3895880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3896318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3896681Z return mod(**inputs) 2025-09-07T06:59:09.3897079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3897506Z outputs = self.model( 2025-09-07T06:59:09.3897912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3898333Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3898753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3899175Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3899562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3899966Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3900404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.3900883Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.3901078Z 2025-09-07T06:59:09.3901202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3901602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3901975Z return mod(**inputs) 2025-09-07T06:59:09.3902454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3902890Z outputs = self.model( 2025-09-07T06:59:09.3903310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3903751Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3904169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3904602Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3904988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3905401Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3905917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.3906424Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.3906859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.3907249Z return self.act(input) 2025-09-07T06:59:09.3907375Z 2025-09-07T06:59:09.3907503Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3907901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3908290Z return mod(**inputs) 2025-09-07T06:59:09.3908713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3909145Z outputs = self.model( 2025-09-07T06:59:09.3909542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3909968Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3910399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3910828Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3911218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3911620Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3912048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.3912530Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.3912682Z 2025-09-07T06:59:09.3912806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3913208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3913563Z return mod(**inputs) 2025-09-07T06:59:09.3913970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3914394Z outputs = self.model( 2025-09-07T06:59:09.3914794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3915230Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3915645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3916077Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3916464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3916881Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3917298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3917756Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3918236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.3918758Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.3918986Z 2025-09-07T06:59:09.3919110Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3919511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3920042Z return mod(**inputs) 2025-09-07T06:59:09.3920456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3920887Z outputs = self.model( 2025-09-07T06:59:09.3921282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3921721Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3922150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3922578Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3922971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3923377Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3923820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3924270Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3924714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.3925153Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.3925309Z 2025-09-07T06:59:09.3925426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3925838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3926208Z return mod(**inputs) 2025-09-07T06:59:09.3926611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3927031Z outputs = self.model( 2025-09-07T06:59:09.3927437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3927966Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3928477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3928929Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3929320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3929730Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3930165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3930616Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3931063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.3931512Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.3931683Z 2025-09-07T06:59:09.3931774Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3932024Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3932259Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3932485Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3932751Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3933156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3933527Z return mod(**inputs) 2025-09-07T06:59:09.3933999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3934432Z outputs = self.model( 2025-09-07T06:59:09.3934839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3935273Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3935698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3936119Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3936516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3936923Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3937359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3937802Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3938244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.3938698Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.3939207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.3939749Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.3939960Z 2025-09-07T06:59:09.3940082Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3940477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3940842Z return mod(**inputs) 2025-09-07T06:59:09.3941247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3941674Z outputs = self.model( 2025-09-07T06:59:09.3942074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3942508Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3942931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3943359Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3943760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3944203Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3944628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3945082Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3945529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.3946075Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.3946592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.3947115Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.3947298Z 2025-09-07T06:59:09.3947424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3947834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3948199Z return mod(**inputs) 2025-09-07T06:59:09.3948629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3949050Z outputs = self.model( 2025-09-07T06:59:09.3949452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3949880Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3950328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3950757Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3951144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3951561Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3951986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3952445Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3952891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.3953342Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.3953497Z 2025-09-07T06:59:09.3953621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3954030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3954385Z return mod(**inputs) 2025-09-07T06:59:09.3954775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3955187Z outputs = self.model( 2025-09-07T06:59:09.3955580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3955997Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3956405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3956815Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3957192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3957584Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3958001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.3958464Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.3958652Z 2025-09-07T06:59:09.3958774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3959161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3959591Z return mod(**inputs) 2025-09-07T06:59:09.3959980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3960390Z outputs = self.model( 2025-09-07T06:59:09.3960784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3961203Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3961609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3962024Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3962404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3962801Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3963211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.3963683Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.3964106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.3964481Z return self.act(input) 2025-09-07T06:59:09.3964602Z 2025-09-07T06:59:09.3964724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3965109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3965513Z return mod(**inputs) 2025-09-07T06:59:09.3965905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3966319Z outputs = self.model( 2025-09-07T06:59:09.3966701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3967121Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3967526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3967941Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3968316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3968710Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3969132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.3969556Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.3969702Z 2025-09-07T06:59:09.3969824Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3970212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3970559Z return mod(**inputs) 2025-09-07T06:59:09.3970948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3971359Z outputs = self.model( 2025-09-07T06:59:09.3971747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3972155Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3972564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3972971Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3973345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3973738Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3974145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3974619Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3975051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.3975558Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.3975777Z 2025-09-07T06:59:09.3975895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3976276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3976629Z return mod(**inputs) 2025-09-07T06:59:09.3977018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3977427Z outputs = self.model( 2025-09-07T06:59:09.3977809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3978223Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3978631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3979046Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3979421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3979822Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3980239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3980702Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3981142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.3981566Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.3981720Z 2025-09-07T06:59:09.3981832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3982223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3982580Z return mod(**inputs) 2025-09-07T06:59:09.3982970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3983375Z outputs = self.model( 2025-09-07T06:59:09.3983767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3984186Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3984597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3985016Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3985392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3985887Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3986323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3986779Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3987233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.3987671Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.3987837Z 2025-09-07T06:59:09.3987928Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3988167Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3989380Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3989610Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.3989881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3990288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3990716Z return mod(**inputs) 2025-09-07T06:59:09.3991112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3991540Z outputs = self.model( 2025-09-07T06:59:09.3991947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3992407Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.3992831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.3993253Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.3993642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.3994049Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.3994478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.3994921Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.3995370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.3995829Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.3996333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.3996873Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.3997121Z 2025-09-07T06:59:09.3997239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.3997645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.3998021Z return mod(**inputs) 2025-09-07T06:59:09.3998430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.3998858Z outputs = self.model( 2025-09-07T06:59:09.3999254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.3999688Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4000110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4000537Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4000921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4001326Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4001755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4002207Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4002638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4003078Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4003566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4004066Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4004238Z 2025-09-07T06:59:09.4004358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4004753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4005103Z return mod(**inputs) 2025-09-07T06:59:09.4005495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4005907Z outputs = self.model( 2025-09-07T06:59:09.4006297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4006744Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4007150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4007563Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4007942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4008337Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4008750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4009185Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4009613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4010036Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4010189Z 2025-09-07T06:59:09.4010310Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4010694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4011051Z return mod(**inputs) 2025-09-07T06:59:09.4011436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4011845Z outputs = self.model( 2025-09-07T06:59:09.4012259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4012680Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4013088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4013502Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4013881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4014270Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4014688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4015151Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4015340Z 2025-09-07T06:59:09.4015459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4015844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4016190Z return mod(**inputs) 2025-09-07T06:59:09.4016583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4016996Z outputs = self.model( 2025-09-07T06:59:09.4017389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4017802Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4018210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4018623Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4019002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4019407Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4020018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4020486Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4020913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4021302Z return self.act(input) 2025-09-07T06:59:09.4021423Z 2025-09-07T06:59:09.4021536Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4022041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4022396Z return mod(**inputs) 2025-09-07T06:59:09.4022791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4023209Z outputs = self.model( 2025-09-07T06:59:09.4023594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4024020Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4024437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4024859Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4025230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4025692Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4026143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4026595Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4026754Z 2025-09-07T06:59:09.4026872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4027253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4027607Z return mod(**inputs) 2025-09-07T06:59:09.4028071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4028483Z outputs = self.model( 2025-09-07T06:59:09.4028875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4029284Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4029694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4030105Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4030480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4030877Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4031315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4031761Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4032212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4032724Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4032949Z 2025-09-07T06:59:09.4033064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4033463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4033818Z return mod(**inputs) 2025-09-07T06:59:09.4034212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4034625Z outputs = self.model( 2025-09-07T06:59:09.4035010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4035427Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4035841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4036254Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4036621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4037029Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4037480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4037912Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4038341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4038764Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4038916Z 2025-09-07T06:59:09.4039029Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4039416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4039766Z return mod(**inputs) 2025-09-07T06:59:09.4040152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4040560Z outputs = self.model( 2025-09-07T06:59:09.4040927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4041321Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4041701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4042081Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4042435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4042808Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4043233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4043646Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4044046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4044468Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4044632Z 2025-09-07T06:59:09.4044720Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4044957Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4045180Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4045402Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4045654Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4046045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4046378Z return mod(**inputs) 2025-09-07T06:59:09.4046741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4047130Z outputs = self.model( 2025-09-07T06:59:09.4047498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4047900Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4048305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4048715Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4049095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4049464Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4049851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4050250Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4050659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4051082Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4051567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4052139Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4052340Z 2025-09-07T06:59:09.4052453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4052857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4053194Z return mod(**inputs) 2025-09-07T06:59:09.4053570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4053957Z outputs = self.model( 2025-09-07T06:59:09.4054330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4054725Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4055112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4055507Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4055859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4056230Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4056625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4057033Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4057467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4057888Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4058347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4058849Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4059032Z 2025-09-07T06:59:09.4059153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4059538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4059889Z return mod(**inputs) 2025-09-07T06:59:09.4060281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4060694Z outputs = self.model( 2025-09-07T06:59:09.4061089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4061501Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4061909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4062332Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4062721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4063121Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4063549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4063998Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4064444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4064879Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4065028Z 2025-09-07T06:59:09.4065143Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4065544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4065996Z return mod(**inputs) 2025-09-07T06:59:09.4066403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4066869Z outputs = self.model( 2025-09-07T06:59:09.4067267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4067682Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4068058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4068443Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4068790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4069158Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4069560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4070002Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4070178Z 2025-09-07T06:59:09.4070293Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4070657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4070991Z return mod(**inputs) 2025-09-07T06:59:09.4071363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4071751Z outputs = self.model( 2025-09-07T06:59:09.4072122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4072582Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4072987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4073370Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4073719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4074077Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4074462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4074900Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4075305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4075662Z return self.act(input) 2025-09-07T06:59:09.4075779Z 2025-09-07T06:59:09.4075892Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4076288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4076638Z return mod(**inputs) 2025-09-07T06:59:09.4077013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4077393Z outputs = self.model( 2025-09-07T06:59:09.4077781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4078168Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4078555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4078950Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4079300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4079679Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4080076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4080481Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4080621Z 2025-09-07T06:59:09.4080735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4081096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4081486Z return mod(**inputs) 2025-09-07T06:59:09.4081856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4082245Z outputs = self.model( 2025-09-07T06:59:09.4082605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4082999Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4083388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4083777Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4084134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4084498Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4084894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4085306Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4085711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4086181Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4086392Z 2025-09-07T06:59:09.4086498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4086914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4087250Z return mod(**inputs) 2025-09-07T06:59:09.4087623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4088006Z outputs = self.model( 2025-09-07T06:59:09.4088378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4088778Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4089166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4089555Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4089908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4090341Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4090742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4091153Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4091554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4091952Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4092109Z 2025-09-07T06:59:09.4092214Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4092584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4092944Z return mod(**inputs) 2025-09-07T06:59:09.4093333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4093724Z outputs = self.model( 2025-09-07T06:59:09.4094096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4094490Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4094876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4095246Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4095591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4095991Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4096391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4096826Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4097257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4097684Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4097839Z 2025-09-07T06:59:09.4097933Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4098164Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4098382Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4098592Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4098831Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4099199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4099520Z return mod(**inputs) 2025-09-07T06:59:09.4099888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4100276Z outputs = self.model( 2025-09-07T06:59:09.4100652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4101079Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4101514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4101923Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4102299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4102692Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4103104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4103549Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4103977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4104426Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4104924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4105445Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4105724Z 2025-09-07T06:59:09.4105850Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4106255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4106622Z return mod(**inputs) 2025-09-07T06:59:09.4107040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4107458Z outputs = self.model( 2025-09-07T06:59:09.4107861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4108285Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4108694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4109096Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4109446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4109809Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4110210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4110680Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4111103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4111538Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4112016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4112513Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4112685Z 2025-09-07T06:59:09.4112806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4113186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4113536Z return mod(**inputs) 2025-09-07T06:59:09.4113924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4114344Z outputs = self.model( 2025-09-07T06:59:09.4114725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4115137Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4115540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4115955Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4116317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4116740Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4117158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4117597Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4118029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4118454Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4118594Z 2025-09-07T06:59:09.4118701Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4119070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4119406Z return mod(**inputs) 2025-09-07T06:59:09.4119979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4120376Z outputs = self.model( 2025-09-07T06:59:09.4120761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4121180Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4121581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4121977Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4122326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4122711Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4123138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4123586Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4123766Z 2025-09-07T06:59:09.4123879Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4124244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4124580Z return mod(**inputs) 2025-09-07T06:59:09.4124953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4125346Z outputs = self.model( 2025-09-07T06:59:09.4125807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4126204Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4126611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4127037Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4127417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4127803Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4128226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4128722Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4129146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4129511Z return self.act(input) 2025-09-07T06:59:09.4129641Z 2025-09-07T06:59:09.4129751Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4130139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4130506Z return mod(**inputs) 2025-09-07T06:59:09.4130888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4131300Z outputs = self.model( 2025-09-07T06:59:09.4131780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4132218Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4132631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4133042Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4133416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4133815Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4134231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4134653Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4134803Z 2025-09-07T06:59:09.4134916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4135310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4135679Z return mod(**inputs) 2025-09-07T06:59:09.4136070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4136505Z outputs = self.model( 2025-09-07T06:59:09.4136911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4137333Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4137739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4138151Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4138520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4138915Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4139334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4139770Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4140205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4140719Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4141011Z 2025-09-07T06:59:09.4141124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4141518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4141869Z return mod(**inputs) 2025-09-07T06:59:09.4142259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4142664Z outputs = self.model( 2025-09-07T06:59:09.4143066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4143481Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4143887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4144304Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4144690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4145099Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4145525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4146041Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4146483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4146930Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4147130Z 2025-09-07T06:59:09.4147249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4147663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4148024Z return mod(**inputs) 2025-09-07T06:59:09.4148415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4148838Z outputs = self.model( 2025-09-07T06:59:09.4149237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4149660Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4150067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4150490Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4150875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4151273Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4151696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4152128Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4152568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4153008Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4153154Z 2025-09-07T06:59:09.4153244Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4153460Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4153685Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4153908Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4154161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4154559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4154912Z return mod(**inputs) 2025-09-07T06:59:09.4155310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4155728Z outputs = self.model( 2025-09-07T06:59:09.4156135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4156570Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4156958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4157356Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4157734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4158127Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4158541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4158975Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4159403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4159834Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4160292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4160782Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4160982Z 2025-09-07T06:59:09.4161090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4161465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4161799Z return mod(**inputs) 2025-09-07T06:59:09.4162195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4162586Z outputs = self.model( 2025-09-07T06:59:09.4162959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4163369Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4163791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4164183Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4164553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4164951Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4165354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4165766Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4166194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4166637Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4167120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4167623Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4167798Z 2025-09-07T06:59:09.4167912Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4168305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4168658Z return mod(**inputs) 2025-09-07T06:59:09.4169049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4169473Z outputs = self.model( 2025-09-07T06:59:09.4169840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4170247Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4170655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4171077Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4171489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4171886Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4172308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4172745Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4173183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4173605Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4173757Z 2025-09-07T06:59:09.4173872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4174258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4174608Z return mod(**inputs) 2025-09-07T06:59:09.4174993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4175398Z outputs = self.model( 2025-09-07T06:59:09.4175786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4176197Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4176601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4177011Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4177429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4177830Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4178257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4178734Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4178924Z 2025-09-07T06:59:09.4179038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4179431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4179786Z return mod(**inputs) 2025-09-07T06:59:09.4180219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4180637Z outputs = self.model( 2025-09-07T06:59:09.4181032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4181462Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4181882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4182319Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4182699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4183111Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4183543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4184031Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4184471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4184850Z return self.act(input) 2025-09-07T06:59:09.4184982Z 2025-09-07T06:59:09.4185097Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4185501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4185965Z return mod(**inputs) 2025-09-07T06:59:09.4186365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4186844Z outputs = self.model( 2025-09-07T06:59:09.4187249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4187685Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4188105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4188521Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4188913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4189318Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4189745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4190181Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4190333Z 2025-09-07T06:59:09.4190454Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4190857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4191227Z return mod(**inputs) 2025-09-07T06:59:09.4191626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4192040Z outputs = self.model( 2025-09-07T06:59:09.4192441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4192948Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4193371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4193790Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4194169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4194576Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4195001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4195457Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4195858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4196343Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4196575Z 2025-09-07T06:59:09.4196692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4197085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4197440Z return mod(**inputs) 2025-09-07T06:59:09.4197826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4198242Z outputs = self.model( 2025-09-07T06:59:09.4198614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4199007Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4199392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4199772Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4200131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4200502Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4200895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4201298Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4201703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4202141Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4202278Z 2025-09-07T06:59:09.4202393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4202759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4203090Z return mod(**inputs) 2025-09-07T06:59:09.4203446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4203823Z outputs = self.model( 2025-09-07T06:59:09.4204185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4204570Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4204939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4205329Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4205684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4205766Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4206027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4206121Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4206412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4206503Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4206507Z 2025-09-07T06:59:09.4206589Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4206679Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4206757Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4206840Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4206950Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4207152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4207227Z return mod(**inputs) 2025-09-07T06:59:09.4207481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4207555Z outputs = self.model( 2025-09-07T06:59:09.4207811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4207885Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4208141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4208217Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4208445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4208530Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4208777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4208878Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4209127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4209234Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4209538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4209680Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4209683Z 2025-09-07T06:59:09.4209788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4209993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4210102Z return mod(**inputs) 2025-09-07T06:59:09.4210357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4210433Z outputs = self.model( 2025-09-07T06:59:09.4210690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4210766Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4211025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4211100Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4211332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4211412Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4211672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4211768Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4212019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4212125Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4212427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4212574Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4212578Z 2025-09-07T06:59:09.4212684Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4212891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4212966Z return mod(**inputs) 2025-09-07T06:59:09.4213222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4213301Z outputs = self.model( 2025-09-07T06:59:09.4213558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4213634Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4213896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4213969Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4214219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4214305Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4214595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4214692Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4214983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4215081Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4215084Z 2025-09-07T06:59:09.4215194Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4215416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4215485Z return mod(**inputs) 2025-09-07T06:59:09.4215751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4215830Z outputs = self.model( 2025-09-07T06:59:09.4216085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4216165Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4216419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4216530Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4216755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4216834Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4217090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4217216Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4217220Z 2025-09-07T06:59:09.4217339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4217552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4217622Z return mod(**inputs) 2025-09-07T06:59:09.4217896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4217972Z outputs = self.model( 2025-09-07T06:59:09.4218246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4218324Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4218598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4218681Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4218954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4219049Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4219329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4219464Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4219902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4219988Z return self.act(input) 2025-09-07T06:59:09.4219992Z 2025-09-07T06:59:09.4220115Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4220333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4220414Z return mod(**inputs) 2025-09-07T06:59:09.4220696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4220772Z outputs = self.model( 2025-09-07T06:59:09.4221070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4221151Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4221426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4221508Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4221748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4221841Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4222108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4222207Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4222211Z 2025-09-07T06:59:09.4222324Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4222541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4222611Z return mod(**inputs) 2025-09-07T06:59:09.4222879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4222957Z outputs = self.model( 2025-09-07T06:59:09.4223319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4223401Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4223666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4223742Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4223984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4224077Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4224348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4224445Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4224718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4224884Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4224888Z 2025-09-07T06:59:09.4224997Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4225217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4225287Z return mod(**inputs) 2025-09-07T06:59:09.4225573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4225754Z outputs = self.model( 2025-09-07T06:59:09.4226040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4226128Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4226400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4226490Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4226737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4226825Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4227093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4227185Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4227447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4227530Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4227534Z 2025-09-07T06:59:09.4227647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4227847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4227914Z return mod(**inputs) 2025-09-07T06:59:09.4228179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4228249Z outputs = self.model( 2025-09-07T06:59:09.4228510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4228584Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4228832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4228917Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4229142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4229230Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4229478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4229607Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4229865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4229953Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4229957Z 2025-09-07T06:59:09.4230047Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4230127Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4230211Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4230288Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4230394Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4230605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4230672Z return mod(**inputs) 2025-09-07T06:59:09.4230931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4231003Z outputs = self.model( 2025-09-07T06:59:09.4231256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4231339Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4231601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4231682Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4231939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4232020Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4232279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4232370Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4232632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4232735Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4233049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4233184Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4233187Z 2025-09-07T06:59:09.4233291Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4233502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4233568Z return mod(**inputs) 2025-09-07T06:59:09.4233829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4233898Z outputs = self.model( 2025-09-07T06:59:09.4234150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4234235Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4234485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4234562Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4234788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4234867Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4235129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4235221Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4235478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4235576Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4235922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4236033Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4236037Z 2025-09-07T06:59:09.4236140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4236349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4236416Z return mod(**inputs) 2025-09-07T06:59:09.4236679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4236748Z outputs = self.model( 2025-09-07T06:59:09.4237006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4237086Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4237337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4237419Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4237646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4237731Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4237985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4238077Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4238371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4238456Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4238459Z 2025-09-07T06:59:09.4238571Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4238773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4238844Z return mod(**inputs) 2025-09-07T06:59:09.4239105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4239173Z outputs = self.model( 2025-09-07T06:59:09.4239432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4239505Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4239759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4239839Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4240106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4240193Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4240444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4240572Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4240575Z 2025-09-07T06:59:09.4240679Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4240880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4240956Z return mod(**inputs) 2025-09-07T06:59:09.4241213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4241287Z outputs = self.model( 2025-09-07T06:59:09.4241540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4241613Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4241872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4241986Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4242218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4242297Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4242556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4242675Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4242896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4242977Z return self.act(input) 2025-09-07T06:59:09.4242980Z 2025-09-07T06:59:09.4243084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4243294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4243362Z return mod(**inputs) 2025-09-07T06:59:09.4243617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4243692Z outputs = self.model( 2025-09-07T06:59:09.4243946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4244029Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4244281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4244390Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4244622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4244702Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4244960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4245045Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4245049Z 2025-09-07T06:59:09.4245159Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4245361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4245440Z return mod(**inputs) 2025-09-07T06:59:09.4245693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4245759Z outputs = self.model( 2025-09-07T06:59:09.4246014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4246087Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4246340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4246423Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4246655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4246742Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4246991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4247085Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4247339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4247496Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4247500Z 2025-09-07T06:59:09.4247613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4247814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4247886Z return mod(**inputs) 2025-09-07T06:59:09.4248185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4248252Z outputs = self.model( 2025-09-07T06:59:09.4248513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4248588Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4248846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4248921Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4249151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4249238Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4249485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4249585Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4249836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4249924Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4249928Z 2025-09-07T06:59:09.4250030Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4250233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4250308Z return mod(**inputs) 2025-09-07T06:59:09.4250591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4250669Z outputs = self.model( 2025-09-07T06:59:09.4250926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4251000Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4251263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4251336Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4251565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4251643Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4251890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4251994Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4252243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4252338Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4252341Z 2025-09-07T06:59:09.4252421Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4252508Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4252590Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4252668Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4252778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4252981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4253052Z return mod(**inputs) 2025-09-07T06:59:09.4253305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4253372Z outputs = self.model( 2025-09-07T06:59:09.4253632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4253705Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4253966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4254047Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4254297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4254382Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4254623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4254719Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4254964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4255060Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4255354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4255486Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4255489Z 2025-09-07T06:59:09.4255600Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4255795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4255865Z return mod(**inputs) 2025-09-07T06:59:09.4256118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4256187Z outputs = self.model( 2025-09-07T06:59:09.4256445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4256558Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4256816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4256889Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4257113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4257204Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4257451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4257550Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4257800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4257904Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4258205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4258316Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4258320Z 2025-09-07T06:59:09.4258432Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4258632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4258710Z return mod(**inputs) 2025-09-07T06:59:09.4258969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4259037Z outputs = self.model( 2025-09-07T06:59:09.4259287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4259360Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4259611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4259682Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4259899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4259984Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4260229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4260359Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4260608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4260698Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4260702Z 2025-09-07T06:59:09.4260803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4261003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4261080Z return mod(**inputs) 2025-09-07T06:59:09.4261332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4261407Z outputs = self.model( 2025-09-07T06:59:09.4261661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4261740Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4261996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4262069Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4262301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4262379Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4262665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4262787Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4262790Z 2025-09-07T06:59:09.4262895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4263106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4263172Z return mod(**inputs) 2025-09-07T06:59:09.4263435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4263503Z outputs = self.model( 2025-09-07T06:59:09.4263755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4263839Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4264089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4264169Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4264395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4264476Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4264736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4264857Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4265085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4265157Z return self.act(input) 2025-09-07T06:59:09.4265161Z 2025-09-07T06:59:09.4265271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4265478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4265548Z return mod(**inputs) 2025-09-07T06:59:09.4265908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4265987Z outputs = self.model( 2025-09-07T06:59:09.4266260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4266338Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4266644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4266732Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4266977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4267065Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4267318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4267404Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4267416Z 2025-09-07T06:59:09.4267522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4267724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4267799Z return mod(**inputs) 2025-09-07T06:59:09.4268051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4268129Z outputs = self.model( 2025-09-07T06:59:09.4268381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4268453Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4268707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4268778Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4269046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4269129Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4269376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4269479Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4269729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4269889Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4269892Z 2025-09-07T06:59:09.4269995Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4270200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4270267Z return mod(**inputs) 2025-09-07T06:59:09.4270518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4270594Z outputs = self.model( 2025-09-07T06:59:09.4270848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4270928Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4271177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4271252Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4271485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4271565Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4271822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4271916Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4272167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4272257Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4272261Z 2025-09-07T06:59:09.4272364Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4272574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4272677Z return mod(**inputs) 2025-09-07T06:59:09.4272941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4273010Z outputs = self.model( 2025-09-07T06:59:09.4273264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4273346Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4273600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4273679Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4273903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4273983Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4274240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4274335Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4274592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4274678Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4274682Z 2025-09-07T06:59:09.4274770Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4274851Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4274961Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4275047Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4275152Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4275357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4275431Z return mod(**inputs) 2025-09-07T06:59:09.4275686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4275764Z outputs = self.model( 2025-09-07T06:59:09.4276019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4276100Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4276352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4276423Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4276661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4276742Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4277002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4277093Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4277349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4277460Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4277761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4277905Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4277908Z 2025-09-07T06:59:09.4278016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4278228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4278295Z return mod(**inputs) 2025-09-07T06:59:09.4278558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4278637Z outputs = self.model( 2025-09-07T06:59:09.4278919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4278999Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4279242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4279312Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4279548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4279630Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4279883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4279973Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4280222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4280327Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4280619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4280737Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4280740Z 2025-09-07T06:59:09.4280841Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4281043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4281141Z return mod(**inputs) 2025-09-07T06:59:09.4281387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4281461Z outputs = self.model( 2025-09-07T06:59:09.4281712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4281796Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4282045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4282118Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4282349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4282427Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4282688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4282783Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4283064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4283152Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4283156Z 2025-09-07T06:59:09.4283264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4283494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4283560Z return mod(**inputs) 2025-09-07T06:59:09.4283820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4283888Z outputs = self.model( 2025-09-07T06:59:09.4284137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4284220Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4284470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4284549Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4284772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4284886Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4285141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4285262Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4285265Z 2025-09-07T06:59:09.4285377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4285578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4285654Z return mod(**inputs) 2025-09-07T06:59:09.4285920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4285991Z outputs = self.model( 2025-09-07T06:59:09.4286267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4286344Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4286620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4286697Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4286936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4287031Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4287295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4287462Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4287695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4287772Z return self.act(input) 2025-09-07T06:59:09.4287793Z 2025-09-07T06:59:09.4287896Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4288100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4288180Z return mod(**inputs) 2025-09-07T06:59:09.4288436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4288511Z outputs = self.model( 2025-09-07T06:59:09.4288761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4288834Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4289103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4289181Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4289424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4289509Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4289779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4289875Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4289879Z 2025-09-07T06:59:09.4289993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4290213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4290282Z return mod(**inputs) 2025-09-07T06:59:09.4290562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4290633Z outputs = self.model( 2025-09-07T06:59:09.4290901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4290987Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4291250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4291372Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4291612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4291697Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4291976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4292076Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4292352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4292515Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4292518Z 2025-09-07T06:59:09.4292635Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4292850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4292922Z return mod(**inputs) 2025-09-07T06:59:09.4293200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4293270Z outputs = self.model( 2025-09-07T06:59:09.4293546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4293626Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4293933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4294020Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4294259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4294352Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4294626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4294727Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4295010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4295097Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4295101Z 2025-09-07T06:59:09.4295218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4295433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4295514Z return mod(**inputs) 2025-09-07T06:59:09.4295781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4295854Z outputs = self.model( 2025-09-07T06:59:09.4296137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4296219Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4296506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4296580Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4296816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4296906Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4297181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4297286Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4297565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4297659Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4297708Z 2025-09-07T06:59:09.4297795Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4317038Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4317299Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4317395Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4317528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4317785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4317867Z return mod(**inputs) 2025-09-07T06:59:09.4318217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4318302Z outputs = self.model( 2025-09-07T06:59:09.4318589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4318683Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4318965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4319065Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4319318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4319420Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4319920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4320034Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4320515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4320630Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4320973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4321117Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4321123Z 2025-09-07T06:59:09.4321245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4321462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4321535Z return mod(**inputs) 2025-09-07T06:59:09.4321810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4321885Z outputs = self.model( 2025-09-07T06:59:09.4322154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4322236Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4322496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4322585Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4322821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4322919Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4323172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4323270Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4323531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4323635Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4323947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4324063Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4324068Z 2025-09-07T06:59:09.4324184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4324458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4324539Z return mod(**inputs) 2025-09-07T06:59:09.4324797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4324867Z outputs = self.model( 2025-09-07T06:59:09.4325129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4325209Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4325468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4325543Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4325770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4325862Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4326129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-09-07T06:59:09.4326232Z hidden_states, attn_weights = self.self_attn( 2025-09-07T06:59:09.4326496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4326593Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4326597Z 2025-09-07T06:59:09.4326706Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4326957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4327041Z return mod(**inputs) 2025-09-07T06:59:09.4327314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4327393Z outputs = self.model( 2025-09-07T06:59:09.4327663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4327742Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4328015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4328088Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4328319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4328402Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4328650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4328780Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4328784Z 2025-09-07T06:59:09.4328887Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4329100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4329166Z return mod(**inputs) 2025-09-07T06:59:09.4329425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4329494Z outputs = self.model( 2025-09-07T06:59:09.4329744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4329824Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4330076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4330157Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4330382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4330463Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4330768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-09-07T06:59:09.4330884Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4331107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4331179Z return self.act(input) 2025-09-07T06:59:09.4331182Z 2025-09-07T06:59:09.4331293Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4331498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4331564Z return mod(**inputs) 2025-09-07T06:59:09.4331828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4331896Z outputs = self.model( 2025-09-07T06:59:09.4332153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-09-07T06:59:09.4332230Z encoder_outputs = self.encoder( 2025-09-07T06:59:09.4332476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-09-07T06:59:09.4332557Z layer_outputs = encoder_layer( 2025-09-07T06:59:09.4332781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4332869Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4333843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-09-07T06:59:09.4333944Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4333957Z 2025-09-07T06:59:09.4334061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4334267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4334345Z return mod(**inputs) 2025-09-07T06:59:09.4334599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4334673Z outputs = self.model( 2025-09-07T06:59:09.4334928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4335002Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4335269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4335342Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4335572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4335653Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4335909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4336028Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4336292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4336468Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4336472Z 2025-09-07T06:59:09.4336585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4336810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4336880Z return mod(**inputs) 2025-09-07T06:59:09.4337153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4337232Z outputs = self.model( 2025-09-07T06:59:09.4337500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4337626Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4337897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4337974Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4338223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4338302Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4338565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4338668Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4338918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4339008Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4339012Z 2025-09-07T06:59:09.4339118Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4339338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4339407Z return mod(**inputs) 2025-09-07T06:59:09.4339681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4339753Z outputs = self.model( 2025-09-07T06:59:09.4340057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4340148Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4340415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4340500Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4340737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4340824Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4341108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4341215Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4341497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4341589Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4341593Z 2025-09-07T06:59:09.4341684Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4341777Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4341860Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4341949Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4342061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4342273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4342355Z return mod(**inputs) 2025-09-07T06:59:09.4342624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4342703Z outputs = self.model( 2025-09-07T06:59:09.4342985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4343070Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4343352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4343428Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4343678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4343762Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4344050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4344185Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4344463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4344576Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4344894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4345051Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4345055Z 2025-09-07T06:59:09.4345163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4345384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4345454Z return mod(**inputs) 2025-09-07T06:59:09.4345977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4346070Z outputs = self.model( 2025-09-07T06:59:09.4346338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4346425Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4346695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4346774Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4347064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4347155Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4347429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4347536Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4347808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4347922Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4348245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4348372Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4348376Z 2025-09-07T06:59:09.4348492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4348717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4348789Z return mod(**inputs) 2025-09-07T06:59:09.4349058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4349140Z outputs = self.model( 2025-09-07T06:59:09.4349414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4349498Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4349765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4349843Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4350089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4350178Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4350451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4350555Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4350832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4350959Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4350963Z 2025-09-07T06:59:09.4351074Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4351298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4351369Z return mod(**inputs) 2025-09-07T06:59:09.4351642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4351714Z outputs = self.model( 2025-09-07T06:59:09.4351982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4352077Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4352328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4352409Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4352635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4352715Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4352989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4353106Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4353432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4353588Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4353592Z 2025-09-07T06:59:09.4353702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4353904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4353970Z return mod(**inputs) 2025-09-07T06:59:09.4354244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4354315Z outputs = self.model( 2025-09-07T06:59:09.4354586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4354664Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4354927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4355011Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4355232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4355320Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4355571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4355683Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4355941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4356023Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4356026Z 2025-09-07T06:59:09.4356136Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4356337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4356408Z return mod(**inputs) 2025-09-07T06:59:09.4356664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4356732Z outputs = self.model( 2025-09-07T06:59:09.4356992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4357063Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4357360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4357433Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4357659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4357749Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4357998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4358116Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4358375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4358472Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4358476Z 2025-09-07T06:59:09.4358559Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4358640Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4358728Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4358805Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4358915Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4359118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4359185Z return mod(**inputs) 2025-09-07T06:59:09.4359444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4359545Z outputs = self.model( 2025-09-07T06:59:09.4359808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4359881Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4360136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4360218Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4360444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4360529Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4360784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4360890Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4361153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4361252Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4361560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4361698Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4361705Z 2025-09-07T06:59:09.4361812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4362016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4362081Z return mod(**inputs) 2025-09-07T06:59:09.4362343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4362410Z outputs = self.model( 2025-09-07T06:59:09.4362671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4362745Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4362998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4363078Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4363301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4363420Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4363670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4363783Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4364033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4364130Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4364437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4364544Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4364548Z 2025-09-07T06:59:09.4364657Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4364858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4364928Z return mod(**inputs) 2025-09-07T06:59:09.4365187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4365256Z outputs = self.model( 2025-09-07T06:59:09.4365516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4365590Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4365872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4365953Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4366177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4366264Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4366531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4366651Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4366914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4367004Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4367008Z 2025-09-07T06:59:09.4367124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4367340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4367420Z return mod(**inputs) 2025-09-07T06:59:09.4367688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4367759Z outputs = self.model( 2025-09-07T06:59:09.4368029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4368110Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4368391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4368467Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4368711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4368794Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4369064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4369192Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4369195Z 2025-09-07T06:59:09.4369298Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4369505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4369635Z return mod(**inputs) 2025-09-07T06:59:09.4369888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4369964Z outputs = self.model( 2025-09-07T06:59:09.4370218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4370297Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4370553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4370625Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4370856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4370935Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4371190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4371316Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4371555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4371630Z return self.act(input) 2025-09-07T06:59:09.4371633Z 2025-09-07T06:59:09.4371743Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4371966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4372072Z return mod(**inputs) 2025-09-07T06:59:09.4372351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4372425Z outputs = self.model( 2025-09-07T06:59:09.4372695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4372783Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4373050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4373133Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4373384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4373472Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4373774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4373862Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4373866Z 2025-09-07T06:59:09.4373983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4374198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4374273Z return mod(**inputs) 2025-09-07T06:59:09.4374540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4374611Z outputs = self.model( 2025-09-07T06:59:09.4374887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4374964Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4375236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4375316Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4375551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4375642Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4375907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4376055Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4376322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4376496Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4376500Z 2025-09-07T06:59:09.4376610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4376843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4376926Z return mod(**inputs) 2025-09-07T06:59:09.4377218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4377297Z outputs = self.model( 2025-09-07T06:59:09.4377572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4377652Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4377940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4378016Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4378261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4378344Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4378668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4378783Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4379111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4379201Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4379205Z 2025-09-07T06:59:09.4379308Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4379518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4379584Z return mod(**inputs) 2025-09-07T06:59:09.4379837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4379912Z outputs = self.model( 2025-09-07T06:59:09.4380167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4380251Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4380542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4380620Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4380876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4380967Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4381257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4381363Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4381653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4381750Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4381754Z 2025-09-07T06:59:09.4381842Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4381941Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4382027Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4382119Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4382231Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4382451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4382566Z return mod(**inputs) 2025-09-07T06:59:09.4382880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4382960Z outputs = self.model( 2025-09-07T06:59:09.4383230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4383307Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4383609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4383687Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4383932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4384018Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4384293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4384411Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4384684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4384795Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4385112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4385300Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4385305Z 2025-09-07T06:59:09.4385417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4385723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4385816Z return mod(**inputs) 2025-09-07T06:59:09.4386094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4386187Z outputs = self.model( 2025-09-07T06:59:09.4386464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4386552Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4386843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4386922Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4387178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4387263Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4387534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4387639Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4387908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4388023Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4388339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4388463Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4388467Z 2025-09-07T06:59:09.4388576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4388797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4388868Z return mod(**inputs) 2025-09-07T06:59:09.4389137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4389218Z outputs = self.model( 2025-09-07T06:59:09.4389483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4389606Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4389872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4389949Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4390198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4390282Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4390559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4390665Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4390933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4391030Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4391036Z 2025-09-07T06:59:09.4391145Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4391369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4391439Z return mod(**inputs) 2025-09-07T06:59:09.4391729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4391800Z outputs = self.model( 2025-09-07T06:59:09.4392099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4392185Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4392452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4392537Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4392773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4392860Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4393132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4393248Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4393519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4393681Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4393685Z 2025-09-07T06:59:09.4393801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4394017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4394086Z return mod(**inputs) 2025-09-07T06:59:09.4394361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4394435Z outputs = self.model( 2025-09-07T06:59:09.4394713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4394789Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4398967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4399053Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4399291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4399383Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4399637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4399751Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4400039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4400122Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4400127Z 2025-09-07T06:59:09.4400244Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4400460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4400574Z return mod(**inputs) 2025-09-07T06:59:09.4400854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4400927Z outputs = self.model( 2025-09-07T06:59:09.4401192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4401279Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4401544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4401626Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4401849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4401937Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4402188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4402332Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4402592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4402679Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4402683Z 2025-09-07T06:59:09.4402770Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4402849Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4402928Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4403013Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4403116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4403325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4403391Z return mod(**inputs) 2025-09-07T06:59:09.4403646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4403725Z outputs = self.model( 2025-09-07T06:59:09.4403981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4404063Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4404316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4404388Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4404621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4404700Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4404957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4405143Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4405405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4405503Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4405813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4405962Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4405966Z 2025-09-07T06:59:09.4406108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4406330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4406401Z return mod(**inputs) 2025-09-07T06:59:09.4406679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4406760Z outputs = self.model( 2025-09-07T06:59:09.4407033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4407120Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4407391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4407472Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4407715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4407803Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4408078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4408191Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4408464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4408570Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4408933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4409051Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4409055Z 2025-09-07T06:59:09.4409156Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4409412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4409489Z return mod(**inputs) 2025-09-07T06:59:09.4409749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4409815Z outputs = self.model( 2025-09-07T06:59:09.4410070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4410152Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4410406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4410486Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4410725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4410810Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4411083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4411199Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4411469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4411558Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4411605Z 2025-09-07T06:59:09.4411715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4411940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4412011Z return mod(**inputs) 2025-09-07T06:59:09.4412288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4412361Z outputs = self.model( 2025-09-07T06:59:09.4412639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4412741Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4413012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4413096Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4413348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4413444Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4413722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4413853Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4413856Z 2025-09-07T06:59:09.4413978Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4414199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4414279Z return mod(**inputs) 2025-09-07T06:59:09.4414555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4414630Z outputs = self.model( 2025-09-07T06:59:09.4414911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4414991Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4415307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4415385Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4415636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4415724Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4415995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4416131Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4416371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4416454Z return self.act(input) 2025-09-07T06:59:09.4416457Z 2025-09-07T06:59:09.4416569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4416792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4416882Z return mod(**inputs) 2025-09-07T06:59:09.4417158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4417237Z outputs = self.model( 2025-09-07T06:59:09.4417508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4417586Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4417866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4417943Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4418342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4418492Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4418768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4418859Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4418863Z 2025-09-07T06:59:09.4418974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4419213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4419283Z return mod(**inputs) 2025-09-07T06:59:09.4419779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4419903Z outputs = self.model( 2025-09-07T06:59:09.4420171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4420259Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4420531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4420617Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4420878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4420973Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4421246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4421357Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4421651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4421818Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4421823Z 2025-09-07T06:59:09.4421943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4422165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4422237Z return mod(**inputs) 2025-09-07T06:59:09.4422578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4422654Z outputs = self.model( 2025-09-07T06:59:09.4422946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4423026Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4423318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4423403Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4423647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4423744Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4424031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4424149Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4424436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4424525Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4424529Z 2025-09-07T06:59:09.4424649Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4424868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4424944Z return mod(**inputs) 2025-09-07T06:59:09.4425218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4425292Z outputs = self.model( 2025-09-07T06:59:09.4425586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4425809Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4426106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4426185Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4426450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4426537Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4426841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4426960Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4427244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4427347Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4427351Z 2025-09-07T06:59:09.4427436Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4427524Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4427618Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4427699Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4427817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4428048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4428117Z return mod(**inputs) 2025-09-07T06:59:09.4428396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4428471Z outputs = self.model( 2025-09-07T06:59:09.4428756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4428834Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4429104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4429221Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4429462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4429554Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4429822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4429938Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4430202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4430307Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4430631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4430779Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4430783Z 2025-09-07T06:59:09.4430904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4431119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4431188Z return mod(**inputs) 2025-09-07T06:59:09.4431469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4431543Z outputs = self.model( 2025-09-07T06:59:09.4431817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4431894Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4432161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4432277Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4432523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4432616Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4432882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4432994Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4433258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4433379Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4433701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4433817Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4433823Z 2025-09-07T06:59:09.4433936Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4434153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4434223Z return mod(**inputs) 2025-09-07T06:59:09.4434497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4434568Z outputs = self.model( 2025-09-07T06:59:09.4434841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4434921Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4435194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4435271Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4435508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4435605Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4435906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4436026Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4436275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4436360Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4436365Z 2025-09-07T06:59:09.4436476Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4436680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4436755Z return mod(**inputs) 2025-09-07T06:59:09.4437021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4437101Z outputs = self.model( 2025-09-07T06:59:09.4437368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4437445Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4437717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4437793Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4438038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4438123Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4438386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4438510Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4438773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4438969Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4438973Z 2025-09-07T06:59:09.4439082Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4439295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4439371Z return mod(**inputs) 2025-09-07T06:59:09.4439639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4439740Z outputs = self.model( 2025-09-07T06:59:09.4440009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4440091Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4440360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4440437Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4440684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4440766Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4441040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4441155Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4441422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4441514Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4441518Z 2025-09-07T06:59:09.4441627Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4441849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4441920Z return mod(**inputs) 2025-09-07T06:59:09.4442239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4442312Z outputs = self.model( 2025-09-07T06:59:09.4442580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4442664Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4442930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4443015Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4443252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4443335Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4443611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4443734Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4443993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4444081Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4444085Z 2025-09-07T06:59:09.4444166Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4444253Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4444331Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4444418Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4444521Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4444721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4444795Z return mod(**inputs) 2025-09-07T06:59:09.4445048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4445151Z outputs = self.model( 2025-09-07T06:59:09.4445425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4445502Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4445777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4445854Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4446101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4446204Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4446469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4446591Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4446856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4446969Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4447289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4447440Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4447443Z 2025-09-07T06:59:09.4447551Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4447766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4447845Z return mod(**inputs) 2025-09-07T06:59:09.4448114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4448192Z outputs = self.model( 2025-09-07T06:59:09.4448463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4448578Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4448859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4448935Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4449178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4449264Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4449541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4449657Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4449919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4450031Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4450345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4450466Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4450470Z 2025-09-07T06:59:09.4450577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4450790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4450868Z return mod(**inputs) 2025-09-07T06:59:09.4451135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4451214Z outputs = self.model( 2025-09-07T06:59:09.4451480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4451587Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4451853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4451933Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4452179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4452262Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4452532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4452666Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4452929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4453024Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4453027Z 2025-09-07T06:59:09.4453135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4453352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4453424Z return mod(**inputs) 2025-09-07T06:59:09.4453691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4453771Z outputs = self.model( 2025-09-07T06:59:09.4454035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4454122Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4454391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4454474Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4454710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4454796Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4455111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4455242Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4455247Z 2025-09-07T06:59:09.4455361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4455574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4455645Z return mod(**inputs) 2025-09-07T06:59:09.4455920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4455993Z outputs = self.model( 2025-09-07T06:59:09.4456268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4456345Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4456613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4456699Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4456933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4457025Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4457288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4457423Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4457651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4457726Z return self.act(input) 2025-09-07T06:59:09.4457730Z 2025-09-07T06:59:09.4457849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4458085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4458166Z return mod(**inputs) 2025-09-07T06:59:09.4458437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4458513Z outputs = self.model( 2025-09-07T06:59:09.4458792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4458871Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4459148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4459246Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4459484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4459575Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4459842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4459942Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4459946Z 2025-09-07T06:59:09.4460056Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4460275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4460344Z return mod(**inputs) 2025-09-07T06:59:09.4460610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4460689Z outputs = self.model( 2025-09-07T06:59:09.4460958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4461041Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4461308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4461386Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4461672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4461758Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4462031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4462138Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4462419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4462582Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4462585Z 2025-09-07T06:59:09.4462693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4462915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4462986Z return mod(**inputs) 2025-09-07T06:59:09.4463263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4463334Z outputs = self.model( 2025-09-07T06:59:09.4463602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4463687Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4463969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4464056Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4464293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4464380Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4464666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4464797Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4465079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4465167Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4465171Z 2025-09-07T06:59:09.4465288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4465502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4465594Z return mod(**inputs) 2025-09-07T06:59:09.4465974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4466049Z outputs = self.model( 2025-09-07T06:59:09.4466334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4466417Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4466694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4466789Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4467029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4467121Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4467388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4467503Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4467772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4467864Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4467869Z 2025-09-07T06:59:09.4467963Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4468047Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4468182Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4468266Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4468374Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4468597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4468667Z return mod(**inputs) 2025-09-07T06:59:09.4468942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4469016Z outputs = self.model( 2025-09-07T06:59:09.4469284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4469369Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4469638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4469725Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4469964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4470049Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4470322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4470426Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4470699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4470796Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4471103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4471270Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4471274Z 2025-09-07T06:59:09.4471386Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4471606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4471677Z return mod(**inputs) 2025-09-07T06:59:09.4471951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4472023Z outputs = self.model( 2025-09-07T06:59:09.4472309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4472403Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4472655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4472736Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4472964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4473048Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4473308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4473407Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4473678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4473781Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4474105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4474222Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4474225Z 2025-09-07T06:59:09.4474338Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4474561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4474664Z return mod(**inputs) 2025-09-07T06:59:09.4474942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4475015Z outputs = self.model( 2025-09-07T06:59:09.4475282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4475366Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4475618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4475699Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4475922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4476011Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4476267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4476369Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4476627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4476710Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4476714Z 2025-09-07T06:59:09.4476823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4477026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4477092Z return mod(**inputs) 2025-09-07T06:59:09.4477353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4477442Z outputs = self.model( 2025-09-07T06:59:09.4477715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4477794Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4478071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4478148Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4478389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4478500Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4478764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4478888Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4479153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4479316Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4479320Z 2025-09-07T06:59:09.4479441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4479655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4479733Z return mod(**inputs) 2025-09-07T06:59:09.4479999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4480074Z outputs = self.model( 2025-09-07T06:59:09.4480348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4480427Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4480706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4480780Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4481053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4481133Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4481381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4481499Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4481751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4481843Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4481846Z 2025-09-07T06:59:09.4481951Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4482152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4482227Z return mod(**inputs) 2025-09-07T06:59:09.4482480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4482558Z outputs = self.model( 2025-09-07T06:59:09.4482809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4482888Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4483145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4483218Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4483450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4483530Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4483786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4483918Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4484170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4484264Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4484267Z 2025-09-07T06:59:09.4484347Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4484432Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4484511Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4484588Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4484717Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4484919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4484992Z return mod(**inputs) 2025-09-07T06:59:09.4485244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4485315Z outputs = self.model( 2025-09-07T06:59:09.4485575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4485649Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4485930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4486006Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4486257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4486345Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4486609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4486729Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4486992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4487106Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4487454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4487601Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4487604Z 2025-09-07T06:59:09.4487720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4487938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4488020Z return mod(**inputs) 2025-09-07T06:59:09.4488294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4488376Z outputs = self.model( 2025-09-07T06:59:09.4488650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4488728Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4489010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4489087Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4489336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4489421Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4489692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4489812Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4490081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4490192Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4490530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4490645Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4490658Z 2025-09-07T06:59:09.4490766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4490979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4491058Z return mod(**inputs) 2025-09-07T06:59:09.4491348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4491428Z outputs = self.model( 2025-09-07T06:59:09.4491696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4491774Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4492050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4492128Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4492372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4492457Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4492720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4492843Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4493107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4493203Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4493207Z 2025-09-07T06:59:09.4493318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4493540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4493610Z return mod(**inputs) 2025-09-07T06:59:09.4493916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4493997Z outputs = self.model( 2025-09-07T06:59:09.4494270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4494354Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4494624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4494699Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4494945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4495030Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4495306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4495436Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4495440Z 2025-09-07T06:59:09.4495550Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4495769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4495839Z return mod(**inputs) 2025-09-07T06:59:09.4496115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4496189Z outputs = self.model( 2025-09-07T06:59:09.4496463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4496540Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4496807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4496917Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4497154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4497247Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4497510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4497636Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4497892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4497966Z return self.act(input) 2025-09-07T06:59:09.4497970Z 2025-09-07T06:59:09.4498085Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4498299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4498372Z return mod(**inputs) 2025-09-07T06:59:09.4498649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4498724Z outputs = self.model( 2025-09-07T06:59:09.4498996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4499073Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4499350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4499428Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4499666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4499758Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4500032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4500128Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4500132Z 2025-09-07T06:59:09.4500279Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4500500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4500579Z return mod(**inputs) 2025-09-07T06:59:09.4500859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4500942Z outputs = self.model( 2025-09-07T06:59:09.4501218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4501298Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4501595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4501711Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4502058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4502153Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4502438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4502550Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4502832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4503010Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4503014Z 2025-09-07T06:59:09.4503127Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4503352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4503455Z return mod(**inputs) 2025-09-07T06:59:09.4503732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4503815Z outputs = self.model( 2025-09-07T06:59:09.4504090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4504175Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4504450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4504565Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4504807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4504893Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4505175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4505285Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4505566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4505726Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4505733Z 2025-09-07T06:59:09.4505855Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4506082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4506156Z return mod(**inputs) 2025-09-07T06:59:09.4506438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4506512Z outputs = self.model( 2025-09-07T06:59:09.4506788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4506879Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4507198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4507290Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4507545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4507645Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4507919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4508032Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4508310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4508407Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4508411Z 2025-09-07T06:59:09.4508507Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4508594Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4508679Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4508773Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4508885Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4509124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4509196Z return mod(**inputs) 2025-09-07T06:59:09.4509474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4509560Z outputs = self.model( 2025-09-07T06:59:09.4509838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4509924Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4510200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4510337Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4510592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4510680Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4510960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4511066Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4511355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4511482Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4511812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4511972Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4511977Z 2025-09-07T06:59:09.4512094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4512330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4512406Z return mod(**inputs) 2025-09-07T06:59:09.4512688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4512773Z outputs = self.model( 2025-09-07T06:59:09.4513057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4513150Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4513431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4513520Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4513771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4513861Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4514181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4514289Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4514572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4514676Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4515003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4515131Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4515135Z 2025-09-07T06:59:09.4515246Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4515473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4515544Z return mod(**inputs) 2025-09-07T06:59:09.4515831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4515905Z outputs = self.model( 2025-09-07T06:59:09.4516183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4516270Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4516557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4516640Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4516876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4516958Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4517261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4517371Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4517652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4517753Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4517757Z 2025-09-07T06:59:09.4517875Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4518113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4518182Z return mod(**inputs) 2025-09-07T06:59:09.4518459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4518532Z outputs = self.model( 2025-09-07T06:59:09.4518824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4518902Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4519190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4519276Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4519513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4519770Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4520049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4520168Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4520444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4520611Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4520615Z 2025-09-07T06:59:09.4520837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4521053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4521129Z return mod(**inputs) 2025-09-07T06:59:09.4521397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4521468Z outputs = self.model( 2025-09-07T06:59:09.4521748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4521827Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4522112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4522192Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4522436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4522536Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4522811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4522947Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4523213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4523302Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4523314Z 2025-09-07T06:59:09.4523424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4523638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4523718Z return mod(**inputs) 2025-09-07T06:59:09.4524019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4524098Z outputs = self.model( 2025-09-07T06:59:09.4524370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4524448Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4524724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4524799Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4525075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4525157Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4525423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4525546Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4525809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4525912Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4525916Z 2025-09-07T06:59:09.4526000Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4526091Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4526174Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4526255Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4526375Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4526591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4526662Z return mod(**inputs) 2025-09-07T06:59:09.4526946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4527023Z outputs = self.model( 2025-09-07T06:59:09.4527308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4527420Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4527706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4527786Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4528031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4528128Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4528403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4528527Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4528807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4528912Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4529246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4529390Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4529394Z 2025-09-07T06:59:09.4529509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4529721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4529798Z return mod(**inputs) 2025-09-07T06:59:09.4530074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4530146Z outputs = self.model( 2025-09-07T06:59:09.4530431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4530539Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4530820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4530896Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4531140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4531233Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4531503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4531647Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4531928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4532030Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4532354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4532470Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4532474Z 2025-09-07T06:59:09.4532595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4532813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4532890Z return mod(**inputs) 2025-09-07T06:59:09.4533177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4533250Z outputs = self.model( 2025-09-07T06:59:09.4533527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4533604Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4533879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4533958Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4534243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4534337Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4534602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4534721Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4534990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4535090Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4535094Z 2025-09-07T06:59:09.4535207Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4535425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4535507Z return mod(**inputs) 2025-09-07T06:59:09.4535789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4535871Z outputs = self.model( 2025-09-07T06:59:09.4536147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4536228Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4536517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4536597Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4536851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4536937Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4537208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4537370Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4537377Z 2025-09-07T06:59:09.4537492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4537726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4537797Z return mod(**inputs) 2025-09-07T06:59:09.4538084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4538179Z outputs = self.model( 2025-09-07T06:59:09.4538455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4538545Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4538822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4538910Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4539155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4539243Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4539525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4539655Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4539895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4539972Z return self.act(input) 2025-09-07T06:59:09.4539976Z 2025-09-07T06:59:09.4540097Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4540313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4540389Z return mod(**inputs) 2025-09-07T06:59:09.4540670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4540780Z outputs = self.model( 2025-09-07T06:59:09.4541062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4541143Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4541417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4541504Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4541746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4541839Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4542110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4542200Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4542212Z 2025-09-07T06:59:09.4542328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4542545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4542623Z return mod(**inputs) 2025-09-07T06:59:09.4542897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4542978Z outputs = self.model( 2025-09-07T06:59:09.4543252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4543332Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4543614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4543693Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4543966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4544056Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4544328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4544445Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4544731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4544924Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4544927Z 2025-09-07T06:59:09.4545038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4545277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4545348Z return mod(**inputs) 2025-09-07T06:59:09.4545697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4545791Z outputs = self.model( 2025-09-07T06:59:09.4546081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4546166Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4546451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4546532Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4546790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4546877Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4547160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4547270Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4547600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4547700Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4547705Z 2025-09-07T06:59:09.4547817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4548045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4548116Z return mod(**inputs) 2025-09-07T06:59:09.4548402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4548479Z outputs = self.model( 2025-09-07T06:59:09.4548756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4548843Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4549135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4549222Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4549469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4549555Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4549837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4549946Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4550242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4550338Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4550342Z 2025-09-07T06:59:09.4550448Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4550559Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4550642Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4550731Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4550846Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4551057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4551133Z return mod(**inputs) 2025-09-07T06:59:09.4551401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4551502Z outputs = self.model( 2025-09-07T06:59:09.4551782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4551869Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4552148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4552226Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4552474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4552559Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4552829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4552934Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4553211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4553325Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4553640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4553789Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4553795Z 2025-09-07T06:59:09.4553904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4554164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4554236Z return mod(**inputs) 2025-09-07T06:59:09.4554509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4554589Z outputs = self.model( 2025-09-07T06:59:09.4554865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4554951Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4555222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4555299Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4555545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4555632Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4555906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4556012Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4556276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4556384Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4556702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4556824Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4556828Z 2025-09-07T06:59:09.4556937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4557158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4557254Z return mod(**inputs) 2025-09-07T06:59:09.4557523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4557603Z outputs = self.model( 2025-09-07T06:59:09.4557869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4557955Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4558221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4558320Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4558566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4558651Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4558931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4559037Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4559314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4559403Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4559406Z 2025-09-07T06:59:09.4559517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4559741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4559814Z return mod(**inputs) 2025-09-07T06:59:09.4560090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4560162Z outputs = self.model( 2025-09-07T06:59:09.4560429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4560519Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4560821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4560907Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4561144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4561227Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4561501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4561619Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4561893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4562059Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4562064Z 2025-09-07T06:59:09.4562184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4562407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4562478Z return mod(**inputs) 2025-09-07T06:59:09.4562762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4562837Z outputs = self.model( 2025-09-07T06:59:09.4563123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4563200Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4563465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4563548Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4563804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4563896Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4564162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4564281Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4564554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4564640Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4564661Z 2025-09-07T06:59:09.4564778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4565002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4565079Z return mod(**inputs) 2025-09-07T06:59:09.4565348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4565424Z outputs = self.model( 2025-09-07T06:59:09.4565718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4565797Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4566078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4566155Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4566401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4566496Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4566769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4566893Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4567184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4567324Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4567329Z 2025-09-07T06:59:09.4567418Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4567514Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4567604Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4567685Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4567799Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4568016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4568086Z return mod(**inputs) 2025-09-07T06:59:09.4568366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4568440Z outputs = self.model( 2025-09-07T06:59:09.4568722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4568803Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4569093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4569181Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4569428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4569524Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4569802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4569921Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4570202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4570329Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4570668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4570825Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4570829Z 2025-09-07T06:59:09.4570950Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4571170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4571263Z return mod(**inputs) 2025-09-07T06:59:09.4571548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4571621Z outputs = self.model( 2025-09-07T06:59:09.4571908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4571991Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4572279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4572369Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4572615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4572711Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4572989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4573115Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4573400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4573505Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4573837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4573958Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4574007Z 2025-09-07T06:59:09.4574129Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4574348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4574421Z return mod(**inputs) 2025-09-07T06:59:09.4574707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4574784Z outputs = self.model( 2025-09-07T06:59:09.4575068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4575149Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4575455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4575538Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4575786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4575883Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4576156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4576280Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4576552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4576645Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4576648Z 2025-09-07T06:59:09.4576770Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4576988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4577095Z return mod(**inputs) 2025-09-07T06:59:09.4577375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4577448Z outputs = self.model( 2025-09-07T06:59:09.4577730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4577807Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4578088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4578182Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4578438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4578526Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4578805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4578943Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4578947Z 2025-09-07T06:59:09.4579060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4579288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4579359Z return mod(**inputs) 2025-09-07T06:59:09.4579639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4579722Z outputs = self.model( 2025-09-07T06:59:09.4580001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4580086Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4580368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4580448Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4580707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4580844Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4581125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4581254Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4581496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4581573Z return self.act(input) 2025-09-07T06:59:09.4581577Z 2025-09-07T06:59:09.4581688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4581914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4581985Z return mod(**inputs) 2025-09-07T06:59:09.4582274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4582346Z outputs = self.model( 2025-09-07T06:59:09.4582619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4582704Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4582971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4583056Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4583295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4583382Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4583667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4583779Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4583782Z 2025-09-07T06:59:09.4583904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4584128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4584207Z return mod(**inputs) 2025-09-07T06:59:09.4584487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4584562Z outputs = self.model( 2025-09-07T06:59:09.4584847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4584945Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4585227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4585304Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4585553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4585720Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4586009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4586128Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4586415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4586590Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4586596Z 2025-09-07T06:59:09.4586707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4586928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4587021Z return mod(**inputs) 2025-09-07T06:59:09.4587290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4587375Z outputs = self.model( 2025-09-07T06:59:09.4587707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4587787Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4588064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4588140Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4588388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4588474Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4588742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4588856Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4589124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4589224Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4589227Z 2025-09-07T06:59:09.4589336Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4589558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4589629Z return mod(**inputs) 2025-09-07T06:59:09.4589898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4589980Z outputs = self.model( 2025-09-07T06:59:09.4590249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4590336Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4590604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4590707Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4590958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4591043Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4591316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4591422Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4591730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4591823Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4591826Z 2025-09-07T06:59:09.4591911Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4592004Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4592088Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4592175Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4592287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4592501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4592576Z return mod(**inputs) 2025-09-07T06:59:09.4592849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4592927Z outputs = self.model( 2025-09-07T06:59:09.4593196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4593274Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4593550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4593625Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4593870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4593994Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4594259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4594368Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4594630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4594740Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4595054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4595203Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4595207Z 2025-09-07T06:59:09.4595318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4595529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4595609Z return mod(**inputs) 2025-09-07T06:59:09.4595876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4595956Z outputs = self.model( 2025-09-07T06:59:09.4596223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4596303Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4596578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4596656Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4596897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4597003Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4597271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4597383Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4597648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4597759Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4598096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4598245Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4598249Z 2025-09-07T06:59:09.4598360Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4598577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4598661Z return mod(**inputs) 2025-09-07T06:59:09.4598942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4599025Z outputs = self.model( 2025-09-07T06:59:09.4599304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4599380Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4599656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4599736Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4599986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4600074Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4600358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4600468Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4600779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4600880Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4600883Z 2025-09-07T06:59:09.4600995Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4601222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4601296Z return mod(**inputs) 2025-09-07T06:59:09.4601572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4601656Z outputs = self.model( 2025-09-07T06:59:09.4601933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4602019Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4602299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4602378Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4602631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4602717Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4602997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4603120Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4603400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4603566Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4603596Z 2025-09-07T06:59:09.4603710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4603938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4604010Z return mod(**inputs) 2025-09-07T06:59:09.4604293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4604367Z outputs = self.model( 2025-09-07T06:59:09.4604643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4604760Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4605040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4605126Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4605374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4605471Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4605753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4605870Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4606157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4606244Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4606249Z 2025-09-07T06:59:09.4606367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4606586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4606657Z return mod(**inputs) 2025-09-07T06:59:09.4606941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4607018Z outputs = self.model( 2025-09-07T06:59:09.4607335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4607415Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4607697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4607775Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4608020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4608116Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4608388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4608510Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4608782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4608878Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4608882Z 2025-09-07T06:59:09.4608979Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4609065Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4609156Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4609239Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4609351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4609575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4609647Z return mod(**inputs) 2025-09-07T06:59:09.4609928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4610002Z outputs = self.model( 2025-09-07T06:59:09.4610276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4610386Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4610669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4610752Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4610999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4611093Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4611368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4611502Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4611783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4611887Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4612219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4612370Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4612374Z 2025-09-07T06:59:09.4612484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4612710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4612782Z return mod(**inputs) 2025-09-07T06:59:09.4613068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4613142Z outputs = self.model( 2025-09-07T06:59:09.4613418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4613504Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4613782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4613902Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4614149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4614244Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4614517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4614636Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4614924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4615029Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4615365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4615485Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4615489Z 2025-09-07T06:59:09.4615612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4615833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4615905Z return mod(**inputs) 2025-09-07T06:59:09.4616194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4616279Z outputs = self.model( 2025-09-07T06:59:09.4616559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4616636Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4616907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4617026Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4617270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4617365Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4617633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4617747Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4618029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4618136Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4618139Z 2025-09-07T06:59:09.4618256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4618471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4618548Z return mod(**inputs) 2025-09-07T06:59:09.4618817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4618893Z outputs = self.model( 2025-09-07T06:59:09.4619169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4619246Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4619522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4619811Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4620056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4620148Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4620413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4620554Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4620558Z 2025-09-07T06:59:09.4620667Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4620977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4621057Z return mod(**inputs) 2025-09-07T06:59:09.4621326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4621407Z outputs = self.model( 2025-09-07T06:59:09.4621675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4621760Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4622027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4622104Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4622353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4622440Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4622714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4622842Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4623070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4623153Z return self.act(input) 2025-09-07T06:59:09.4623157Z 2025-09-07T06:59:09.4623267Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4623488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4623557Z return mod(**inputs) 2025-09-07T06:59:09.4623825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4623937Z outputs = self.model( 2025-09-07T06:59:09.4624207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4624290Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4624553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4624635Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4624870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4624985Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4625256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4625343Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4625348Z 2025-09-07T06:59:09.4625465Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4625739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4625816Z return mod(**inputs) 2025-09-07T06:59:09.4626098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4626173Z outputs = self.model( 2025-09-07T06:59:09.4626455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4626536Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4626811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4626897Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4627139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4627239Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4627593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4627713Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4628045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4628224Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4628230Z 2025-09-07T06:59:09.4628348Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4628563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4628641Z return mod(**inputs) 2025-09-07T06:59:09.4628910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4628985Z outputs = self.model( 2025-09-07T06:59:09.4629316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4629394Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4629673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4629749Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4630006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4630093Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4630368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4630482Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4630797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4630894Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4630901Z 2025-09-07T06:59:09.4631010Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4631223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4631323Z return mod(**inputs) 2025-09-07T06:59:09.4631593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4631695Z outputs = self.model( 2025-09-07T06:59:09.4631963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4632040Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4632330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4632410Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4632664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4632747Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4633021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4633125Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4633402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4633506Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4633509Z 2025-09-07T06:59:09.4633596Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4633689Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4633772Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4633855Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4633971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4634233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4634313Z return mod(**inputs) 2025-09-07T06:59:09.4634587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4634658Z outputs = self.model( 2025-09-07T06:59:09.4634945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4635025Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4635309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4635393Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4635621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4635708Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4635962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4636069Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4636330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4636439Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4636767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4636909Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4636913Z 2025-09-07T06:59:09.4637030Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4637264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4637342Z return mod(**inputs) 2025-09-07T06:59:09.4637614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4637686Z outputs = self.model( 2025-09-07T06:59:09.4637962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4638040Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4638329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4638406Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4638659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4638739Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4638990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4639097Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4639348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4639452Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4639750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4639859Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4639872Z 2025-09-07T06:59:09.4639975Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4640174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4640247Z return mod(**inputs) 2025-09-07T06:59:09.4640498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4640608Z outputs = self.model( 2025-09-07T06:59:09.4640865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4640938Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4641205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4641282Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4641526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4641610Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4641873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4641988Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4642256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4642350Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4642353Z 2025-09-07T06:59:09.4642467Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4642669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4642742Z return mod(**inputs) 2025-09-07T06:59:09.4643000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4643076Z outputs = self.model( 2025-09-07T06:59:09.4643327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4643406Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4643686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4643766Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4644011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4644095Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4644366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4644500Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4644765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4644935Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4644939Z 2025-09-07T06:59:09.4645048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4645265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4645338Z return mod(**inputs) 2025-09-07T06:59:09.4645610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4645682Z outputs = self.model( 2025-09-07T06:59:09.4645950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4646038Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4646306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4646391Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4646628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4646714Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4647016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4647134Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4647409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4647497Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4647501Z 2025-09-07T06:59:09.4647619Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4647833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4647903Z return mod(**inputs) 2025-09-07T06:59:09.4648173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4648243Z outputs = self.model( 2025-09-07T06:59:09.4648503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4648577Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4648829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4648908Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4649131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4649223Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4649489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4649602Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4649884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4649993Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4649996Z 2025-09-07T06:59:09.4650087Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4650169Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4650249Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4650335Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4650441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4650661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4650748Z return mod(**inputs) 2025-09-07T06:59:09.4651025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4651103Z outputs = self.model( 2025-09-07T06:59:09.4651379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4651468Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4651743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4651827Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4652069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4652153Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4652435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4652551Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4652833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4652936Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4653261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4653455Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4653459Z 2025-09-07T06:59:09.4653568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4653789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4653858Z return mod(**inputs) 2025-09-07T06:59:09.4654134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4654208Z outputs = self.model( 2025-09-07T06:59:09.4654477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4654562Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4654828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4654914Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4655154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4655239Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4655511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4655625Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4655896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4655999Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4656317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4656457Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4656461Z 2025-09-07T06:59:09.4656569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4656794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4656863Z return mod(**inputs) 2025-09-07T06:59:09.4657140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4657211Z outputs = self.model( 2025-09-07T06:59:09.4657500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4657582Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4657869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4657953Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4658196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4658284Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4658562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4658677Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4658955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4659046Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4659049Z 2025-09-07T06:59:09.4659166Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4659383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4659451Z return mod(**inputs) 2025-09-07T06:59:09.4659734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4659807Z outputs = self.model( 2025-09-07T06:59:09.4660125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4660203Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4660468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4660554Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4660791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4660885Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4661148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4661275Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4661288Z 2025-09-07T06:59:09.4661398Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4661612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4661690Z return mod(**inputs) 2025-09-07T06:59:09.4661958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4662038Z outputs = self.model( 2025-09-07T06:59:09.4662304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4662381Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4662659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4662734Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4662980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4663088Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4663356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4663488Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4663718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4663800Z return self.act(input) 2025-09-07T06:59:09.4663822Z 2025-09-07T06:59:09.4663932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4664152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4664222Z return mod(**inputs) 2025-09-07T06:59:09.4664486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4664567Z outputs = self.model( 2025-09-07T06:59:09.4664839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4664924Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4665190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4665266Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4665519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4665678Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4665973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4666064Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4666069Z 2025-09-07T06:59:09.4666185Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4666416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4666529Z return mod(**inputs) 2025-09-07T06:59:09.4666814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4666888Z outputs = self.model( 2025-09-07T06:59:09.4667175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4667257Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4667539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4667625Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4667864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4667959Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4668226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4668335Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4668624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4668777Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4668781Z 2025-09-07T06:59:09.4668894Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4669095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4669170Z return mod(**inputs) 2025-09-07T06:59:09.4669421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4669507Z outputs = self.model( 2025-09-07T06:59:09.4669773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4669849Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4670123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4670199Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4670436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4670550Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4670818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4670931Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4671196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4671285Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4671296Z 2025-09-07T06:59:09.4671408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4671622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4671698Z return mod(**inputs) 2025-09-07T06:59:09.4671963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4672044Z outputs = self.model( 2025-09-07T06:59:09.4672308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4672383Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4672655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4672734Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4672986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4673109Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4673363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4673468Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4673715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4673810Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4673813Z 2025-09-07T06:59:09.4673894Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4673974Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4674059Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4674141Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4674259Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4674476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4674546Z return mod(**inputs) 2025-09-07T06:59:09.4674820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4674892Z outputs = self.model( 2025-09-07T06:59:09.4675163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4675243Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4675515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4675593Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4675833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4675948Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4676220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4676331Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4676592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4676695Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4677040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4677183Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4677187Z 2025-09-07T06:59:09.4677305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4677517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4677596Z return mod(**inputs) 2025-09-07T06:59:09.4677870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4677953Z outputs = self.model( 2025-09-07T06:59:09.4678215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4678290Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4678550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4678625Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4678849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4678936Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4679197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4679358Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4679624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4679728Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4680047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4680157Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4680161Z 2025-09-07T06:59:09.4680273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4680470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4680541Z return mod(**inputs) 2025-09-07T06:59:09.4680798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4680866Z outputs = self.model( 2025-09-07T06:59:09.4681137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4681214Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4681488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4681565Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4681803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4681894Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4682157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4682268Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4682556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4682656Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4682660Z 2025-09-07T06:59:09.4682769Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4682989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4683063Z return mod(**inputs) 2025-09-07T06:59:09.4683311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4683411Z outputs = self.model( 2025-09-07T06:59:09.4683677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4683754Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4684035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4684110Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4684363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4684451Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4684725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4684853Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4685132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4685306Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4685310Z 2025-09-07T06:59:09.4685423Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4685651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4685723Z return mod(**inputs) 2025-09-07T06:59:09.4686043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4686123Z outputs = self.model( 2025-09-07T06:59:09.4686395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4686477Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4686758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4686835Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4687088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4687176Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4687461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4687580Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4687865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4687955Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4687959Z 2025-09-07T06:59:09.4688072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4688305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4688377Z return mod(**inputs) 2025-09-07T06:59:09.4688659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4688732Z outputs = self.model( 2025-09-07T06:59:09.4689030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4689115Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4689394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4689478Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4689722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4689808Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4690106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4690224Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4690503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4690598Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4690602Z 2025-09-07T06:59:09.4690697Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4690788Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4690874Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4690965Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4691078Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4691307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4691380Z return mod(**inputs) 2025-09-07T06:59:09.4691660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4691742Z outputs = self.model( 2025-09-07T06:59:09.4692018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4692106Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4692383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4692498Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4692753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4692842Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4693125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4693243Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4693514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4693627Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4693949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4694113Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4694119Z 2025-09-07T06:59:09.4694227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4694447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4694517Z return mod(**inputs) 2025-09-07T06:59:09.4694785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4694869Z outputs = self.model( 2025-09-07T06:59:09.4695143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4695227Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4695502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4695594Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4695835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4695927Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4696190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4696297Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4696562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4696675Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4696973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4697088Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4697093Z 2025-09-07T06:59:09.4697195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4697407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4697473Z return mod(**inputs) 2025-09-07T06:59:09.4697727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4697802Z outputs = self.model( 2025-09-07T06:59:09.4698054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4698135Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4698391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4698469Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4698692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4698774Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4699065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4699176Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4699436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4699519Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4699524Z 2025-09-07T06:59:09.4699627Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4699846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4699917Z return mod(**inputs) 2025-09-07T06:59:09.4700196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4700270Z outputs = self.model( 2025-09-07T06:59:09.4700543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4700628Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4700899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4700983Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4701221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4701312Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4701579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4701706Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4701731Z 2025-09-07T06:59:09.4701849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4702065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4702142Z return mod(**inputs) 2025-09-07T06:59:09.4702410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4702481Z outputs = self.model( 2025-09-07T06:59:09.4702759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4702865Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4703137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4703212Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4703450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4703541Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4703807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4703938Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4704164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4704245Z return self.act(input) 2025-09-07T06:59:09.4704248Z 2025-09-07T06:59:09.4704358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4704566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4704644Z return mod(**inputs) 2025-09-07T06:59:09.4704909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4704989Z outputs = self.model( 2025-09-07T06:59:09.4705256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4705376Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4705748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4705837Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4706093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4706183Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4706467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4706557Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4706561Z 2025-09-07T06:59:09.4706684Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4706905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4706975Z return mod(**inputs) 2025-09-07T06:59:09.4707251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4707335Z outputs = self.model( 2025-09-07T06:59:09.4707588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4707669Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4707924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4708003Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4708224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4708305Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4708591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4708694Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4708954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4709106Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4709110Z 2025-09-07T06:59:09.4709220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4709441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4709507Z return mod(**inputs) 2025-09-07T06:59:09.4709774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4709843Z outputs = self.model( 2025-09-07T06:59:09.4710103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4710180Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4710432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4710515Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4710739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4710825Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4711073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4711196Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4711451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4711535Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4711538Z 2025-09-07T06:59:09.4711685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4711888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4711961Z return mod(**inputs) 2025-09-07T06:59:09.4712214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4712282Z outputs = self.model( 2025-09-07T06:59:09.4712544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4712615Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4712873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4712945Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4713174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4713264Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4713517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4713624Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4713876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4713972Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4713976Z 2025-09-07T06:59:09.4714056Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4714137Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4714224Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4714303Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4714437Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4714638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4714707Z return mod(**inputs) 2025-09-07T06:59:09.4714972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4715039Z outputs = self.model( 2025-09-07T06:59:09.4715300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4715394Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4715646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4715727Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4715951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4716039Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4716290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4716390Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4716647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4716745Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4717065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4717208Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4717212Z 2025-09-07T06:59:09.4717328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4717539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4717609Z return mod(**inputs) 2025-09-07T06:59:09.4717919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4717993Z outputs = self.model( 2025-09-07T06:59:09.4718266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4718352Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4718604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4718685Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4718908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4718995Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4719245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4719350Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4719787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4719891Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4720194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4720303Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4720309Z 2025-09-07T06:59:09.4720421Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4720623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4720692Z return mod(**inputs) 2025-09-07T06:59:09.4720951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4721091Z outputs = self.model( 2025-09-07T06:59:09.4721354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4721427Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4721679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4721760Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4721987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4722100Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4722351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4722460Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4722713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4722799Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4722803Z 2025-09-07T06:59:09.4722915Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4723120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4723196Z return mod(**inputs) 2025-09-07T06:59:09.4723449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4723519Z outputs = self.model( 2025-09-07T06:59:09.4723781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4723854Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4724115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4724190Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4724471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4724554Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4724810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4724925Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4725184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4725342Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4725346Z 2025-09-07T06:59:09.4725448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4725657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4725737Z return mod(**inputs) 2025-09-07T06:59:09.4726015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4726096Z outputs = self.model( 2025-09-07T06:59:09.4726367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4726443Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4726719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4726796Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4727049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4727131Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4727420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4727530Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4727788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4727880Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4727883Z 2025-09-07T06:59:09.4727987Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4728203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4728324Z return mod(**inputs) 2025-09-07T06:59:09.4728589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4728669Z outputs = self.model( 2025-09-07T06:59:09.4728934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4729022Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4729289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4729374Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4729610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4729693Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4729969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4730085Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4730357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4730449Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4730454Z 2025-09-07T06:59:09.4730539Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4730632Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4730750Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4730844Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4730956Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4731172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4731251Z return mod(**inputs) 2025-09-07T06:59:09.4731528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4731610Z outputs = self.model( 2025-09-07T06:59:09.4731888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4731964Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4732247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4732324Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4732576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4732659Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4732939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4733053Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4733329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4733439Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4733761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4733933Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4733937Z 2025-09-07T06:59:09.4734049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4734264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4734341Z return mod(**inputs) 2025-09-07T06:59:09.4734610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4734690Z outputs = self.model( 2025-09-07T06:59:09.4734978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4735060Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4735325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4735402Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4735645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4735732Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4736005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4736120Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4736384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4736498Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4736809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4736930Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4736936Z 2025-09-07T06:59:09.4737053Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4737296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4737365Z return mod(**inputs) 2025-09-07T06:59:09.4737617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4737693Z outputs = self.model( 2025-09-07T06:59:09.4737946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4738030Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4738284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4738356Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4738587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4738669Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4738929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4739037Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4739287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4739378Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4739381Z 2025-09-07T06:59:09.4739486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4739693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4739759Z return mod(**inputs) 2025-09-07T06:59:09.4740019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4740108Z outputs = self.model( 2025-09-07T06:59:09.4740365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4740451Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4740712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4740791Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4741021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4741119Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4741377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4741498Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4741502Z 2025-09-07T06:59:09.4741612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4741817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4741885Z return mod(**inputs) 2025-09-07T06:59:09.4742146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4742214Z outputs = self.model( 2025-09-07T06:59:09.4742491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4742570Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4742847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4742923Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4743162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4743257Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4743565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4743700Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4743929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4744002Z return self.act(input) 2025-09-07T06:59:09.4744006Z 2025-09-07T06:59:09.4744123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4744339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4744416Z return mod(**inputs) 2025-09-07T06:59:09.4744683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4744754Z outputs = self.model( 2025-09-07T06:59:09.4745028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4745106Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4745382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4745457Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4745760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4745850Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4746127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4746225Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4746229Z 2025-09-07T06:59:09.4746343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4746581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4746678Z return mod(**inputs) 2025-09-07T06:59:09.4746950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4747032Z outputs = self.model( 2025-09-07T06:59:09.4747299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4747386Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4747655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4747752Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4747997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4748083Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4748360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4748468Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4748744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4748908Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4748912Z 2025-09-07T06:59:09.4749023Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4749242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4749315Z return mod(**inputs) 2025-09-07T06:59:09.4749590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4749661Z outputs = self.model( 2025-09-07T06:59:09.4749926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4750012Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4750319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4750404Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4750644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4750735Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4750999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4751116Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4751372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4751456Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4751459Z 2025-09-07T06:59:09.4751571Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4751782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4751847Z return mod(**inputs) 2025-09-07T06:59:09.4752097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4752162Z outputs = self.model( 2025-09-07T06:59:09.4752413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4752486Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4752727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4752806Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4753027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4753136Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4753394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4753501Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4753755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4753843Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4753864Z 2025-09-07T06:59:09.4753955Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4754037Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4754123Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4754200Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4754304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4754515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4754581Z return mod(**inputs) 2025-09-07T06:59:09.4754846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4754916Z outputs = self.model( 2025-09-07T06:59:09.4755172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4755255Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4755513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4755592Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4755819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4755900Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4756159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4756300Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4756568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4756672Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4756995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4757139Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4757143Z 2025-09-07T06:59:09.4757252Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4757475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4757548Z return mod(**inputs) 2025-09-07T06:59:09.4757825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4757899Z outputs = self.model( 2025-09-07T06:59:09.4758170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4758255Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4758523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4758610Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4758847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4758948Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4759204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4759323Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4759587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4759684Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4759988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4760098Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4760118Z 2025-09-07T06:59:09.4760223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4760443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4760513Z return mod(**inputs) 2025-09-07T06:59:09.4760793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4760867Z outputs = self.model( 2025-09-07T06:59:09.4761149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4761227Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4761497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4761585Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4761827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4761921Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4762190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4762294Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4762581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4762670Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4762673Z 2025-09-07T06:59:09.4762826Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4763028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4763094Z return mod(**inputs) 2025-09-07T06:59:09.4763354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4763423Z outputs = self.model( 2025-09-07T06:59:09.4763680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4763752Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4764022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4764098Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4764348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4764438Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4764708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4764825Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4765080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4765236Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4765247Z 2025-09-07T06:59:09.4765350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4765549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4765641Z return mod(**inputs) 2025-09-07T06:59:09.4765913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4765992Z outputs = self.model( 2025-09-07T06:59:09.4766262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4766338Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4766612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4766710Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4766955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4767038Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4767315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4767440Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4767718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4767811Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4767815Z 2025-09-07T06:59:09.4767924Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4768144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4768216Z return mod(**inputs) 2025-09-07T06:59:09.4768483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4768563Z outputs = self.model( 2025-09-07T06:59:09.4768832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4768917Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4769217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4769295Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4769559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4769643Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4769926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4770041Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4770316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4770416Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4770420Z 2025-09-07T06:59:09.4770506Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4770597Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4770679Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4770763Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4770878Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4771089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4771166Z return mod(**inputs) 2025-09-07T06:59:09.4771431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4771505Z outputs = self.model( 2025-09-07T06:59:09.4771779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4771857Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4772126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4772223Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4772471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4772556Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4772821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4772940Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4773230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4773339Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4773656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4773799Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4773809Z 2025-09-07T06:59:09.4773917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4774132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4774210Z return mod(**inputs) 2025-09-07T06:59:09.4774478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4774557Z outputs = self.model( 2025-09-07T06:59:09.4774826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4774904Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4775178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4775253Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4775498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4775631Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4775908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4776028Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4776295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4776406Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4776728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4776850Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4776854Z 2025-09-07T06:59:09.4776963Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4777178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4777262Z return mod(**inputs) 2025-09-07T06:59:09.4777531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4777609Z outputs = self.model( 2025-09-07T06:59:09.4777876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4777952Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4778236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4778313Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4778569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4778677Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4778953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4779077Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4779358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4779454Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4779458Z 2025-09-07T06:59:09.4779570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4779824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4779896Z return mod(**inputs) 2025-09-07T06:59:09.4780171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4780254Z outputs = self.model( 2025-09-07T06:59:09.4780531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4780620Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4780894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4780972Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4781231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4781318Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4781609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4781742Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4781745Z 2025-09-07T06:59:09.4781863Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4782085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4782157Z return mod(**inputs) 2025-09-07T06:59:09.4782475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4782551Z outputs = self.model( 2025-09-07T06:59:09.4782844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4782922Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4783205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4783291Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4783532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4783628Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4783901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4784034Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4784279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4784355Z return self.act(input) 2025-09-07T06:59:09.4784359Z 2025-09-07T06:59:09.4784477Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4784696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4784778Z return mod(**inputs) 2025-09-07T06:59:09.4785054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4785128Z outputs = self.model( 2025-09-07T06:59:09.4785412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4785512Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4785870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4785956Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4786199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4786297Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4786571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4786695Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4786699Z 2025-09-07T06:59:09.4786812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4787030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4787115Z return mod(**inputs) 2025-09-07T06:59:09.4787393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4787477Z outputs = self.model( 2025-09-07T06:59:09.4787759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4787844Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4788109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4788189Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4788437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4788523Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4788794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4788901Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4789201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4789374Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4789378Z 2025-09-07T06:59:09.4789485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4789703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4789774Z return mod(**inputs) 2025-09-07T06:59:09.4790049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4790120Z outputs = self.model( 2025-09-07T06:59:09.4790386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4790472Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4790743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4790826Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4791063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4791146Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4791420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4791525Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4791798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4791883Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4791905Z 2025-09-07T06:59:09.4792015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4792239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4792310Z return mod(**inputs) 2025-09-07T06:59:09.4792590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4792662Z outputs = self.model( 2025-09-07T06:59:09.4792939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4793056Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4793323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4793407Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4793644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4793735Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4794013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4794116Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4794401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4794493Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4794498Z 2025-09-07T06:59:09.4794591Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4794674Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4794756Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4794842Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4794949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4795181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4795251Z return mod(**inputs) 2025-09-07T06:59:09.4795553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4795634Z outputs = self.model( 2025-09-07T06:59:09.4795901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4795985Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4796252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4796337Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4796572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4796656Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4796928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4797034Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4797318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4797422Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4797736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4797886Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4797890Z 2025-09-07T06:59:09.4797997Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4798218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4798288Z return mod(**inputs) 2025-09-07T06:59:09.4798583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4798655Z outputs = self.model( 2025-09-07T06:59:09.4798926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4799011Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4799277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4799363Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4799649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4799734Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4800019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4800126Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4800407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4800512Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4800834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4800960Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4800964Z 2025-09-07T06:59:09.4801072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4801296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4801366Z return mod(**inputs) 2025-09-07T06:59:09.4801640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4801712Z outputs = self.model( 2025-09-07T06:59:09.4801980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4802100Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4802368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4802455Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4802699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4802786Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4803060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-09-07T06:59:09.4803162Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T06:59:09.4803435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4803525Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4803528Z 2025-09-07T06:59:09.4803645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4803860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4803928Z return mod(**inputs) 2025-09-07T06:59:09.4804203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4804276Z outputs = self.model( 2025-09-07T06:59:09.4804553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4804629Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4804897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4804980Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4805235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4805330Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4805597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4805713Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4805986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-09-07T06:59:09.4806167Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T06:59:09.4806171Z 2025-09-07T06:59:09.4806288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4806503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4806582Z return mod(**inputs) 2025-09-07T06:59:09.4806851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4806925Z outputs = self.model( 2025-09-07T06:59:09.4807205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4807283Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4807558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4807636Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4807876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4807969Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4808235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4808361Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4808669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-09-07T06:59:09.4808765Z key_states = self.k_proj(current_states) 2025-09-07T06:59:09.4808769Z 2025-09-07T06:59:09.4808881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4809099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4809179Z return mod(**inputs) 2025-09-07T06:59:09.4809458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4809543Z outputs = self.model( 2025-09-07T06:59:09.4809811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4809888Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4810164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4810243Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4810493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4810579Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4810863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4810986Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4811249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-09-07T06:59:09.4811351Z value_states = self.v_proj(current_states) 2025-09-07T06:59:09.4811354Z 2025-09-07T06:59:09.4811440Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4811555Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4811639Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4811721Z cudagraph partition due to non gpu ops 2025-09-07T06:59:09.4811842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4812054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4812130Z return mod(**inputs) 2025-09-07T06:59:09.4812401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4812496Z outputs = self.model( 2025-09-07T06:59:09.4812778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4812854Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4813133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4813211Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4813459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4813555Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4813825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4813945Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4814224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4814328Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4814670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T06:59:09.4814811Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:09.4814817Z 2025-09-07T06:59:09.4814937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4815186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4815264Z return mod(**inputs) 2025-09-07T06:59:09.4815533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4815604Z outputs = self.model( 2025-09-07T06:59:09.4815876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4815957Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4816232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4816308Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4816544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4816639Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4816918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4817038Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4817315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-09-07T06:59:09.4817425Z attn_output, attn_weights = attention_interface( 2025-09-07T06:59:09.4817746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T06:59:09.4817859Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T06:59:09.4817863Z 2025-09-07T06:59:09.4817980Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4818216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4818292Z return mod(**inputs) 2025-09-07T06:59:09.4818565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4818643Z outputs = self.model( 2025-09-07T06:59:09.4818922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4818999Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4819292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4819368Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4819793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4819892Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4820169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-09-07T06:59:09.4820301Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T06:59:09.4820589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-09-07T06:59:09.4820691Z attn_output = self.out_proj(attn_output) 2025-09-07T06:59:09.4820695Z 2025-09-07T06:59:09.4820806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4821033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4821116Z return mod(**inputs) 2025-09-07T06:59:09.4821386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4821467Z outputs = self.model( 2025-09-07T06:59:09.4821750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4821832Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4822203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4822283Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4822539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4822626Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4822924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4823054Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4823058Z 2025-09-07T06:59:09.4823170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4823398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4823472Z return mod(**inputs) 2025-09-07T06:59:09.4823758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4823831Z outputs = self.model( 2025-09-07T06:59:09.4824105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4824193Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4824465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4824556Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4824799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4824885Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4825166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-09-07T06:59:09.4825399Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T06:59:09.4825695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:09.4825781Z return self.act(input) 2025-09-07T06:59:09.4825785Z 2025-09-07T06:59:09.4825907Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4826126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4826235Z return mod(**inputs) 2025-09-07T06:59:09.4826523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-09-07T06:59:09.4826596Z outputs = self.model( 2025-09-07T06:59:09.4826877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-09-07T06:59:09.4826958Z decoder_outputs = self.decoder( 2025-09-07T06:59:09.4827233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-09-07T06:59:09.4827317Z layer_outputs = decoder_layer( 2025-09-07T06:59:09.4827561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:09.4827657Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:09.4827926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-09-07T06:59:09.4828028Z hidden_states = self.fc2(hidden_states) 2025-09-07T06:59:09.4828031Z 2025-09-07T06:59:09.4828143Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4828361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4828443Z return mod(**inputs) 2025-09-07T06:59:09.4828716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1490, in forward 2025-09-07T06:59:09.4828873Z lm_logits = self.lm_head(outputs[0]) 2025-09-07T06:59:09.4828878Z 2025-09-07T06:59:09.4828994Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:09.4829212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:09.4829294Z return mod(**inputs) 2025-09-07T06:59:09.4829574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1497, in forward 2025-09-07T06:59:09.4829775Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T06:59:09.4829780Z 2025-09-07T06:59:25.3726188Z Compilation time (from dynamo_timed): 30.756174948 2025-09-07T06:59:25.3820706Z pass 2025-09-07T06:59:25.3821146Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:59:25.3822270Z TIMING: _recursive_pre_grad_passes:0.01536 _recursive_joint_graph_passes:0.8086 _recursive_post_grad_passes:0.18257 async_compile.wait:0.89499 code_gen:14.07288 inductor_compile:17.35835 backend_compile:24.80697 gc:0.00098 entire_frame_compile:30.75617 total_wall_time:30.75617 2025-09-07T06:59:25.3823481Z STATS: call_* op count: 980 | FakeTensorMode.__torch_dispatch__:33505 | FakeTensor.__torch_dispatch__:11174 | ProxyTorchDispatchMode.__torch_dispatch__:12370 2025-09-07T06:59:25.3824189Z Dynamo produced 1 graphs covering 980 ops with 0 graph breaks (0 unique) 2025-09-07T06:59:28.5688305Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:59:28.5689216Z import pynvml # type: ignore[import] 2025-09-07T06:59:31.4446079Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T06:59:31.4449358Z from pkg_resources import resource_filename 2025-09-07T06:59:32.1408814Z 2025-09-07T06:59:33.4447121Z loading model: 0it [00:00, ?it/s] 2025-09-07T06:59:33.4447464Z loading model: 0it [00:01, ?it/s] 2025-09-07T06:59:33.4462737Z cpu eval BertForMaskedLM 2025-09-07T06:59:33.9824954Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:59:34.2284872Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:59:34.4681054Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:59:42.3563277Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3568779Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3569429Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3569660Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3570006Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3570239Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3570461Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3570800Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3571044Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3571367Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3571595Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3571921Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3572222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3572765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3573134Z return mod(**inputs) 2025-09-07T06:59:42.3573958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3574366Z outputs = self.bert( 2025-09-07T06:59:42.3574739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3575144Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3575577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3575979Z layer_outputs = layer_module( 2025-09-07T06:59:42.3576353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3576759Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3577335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3577908Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3582799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3587281Z return func(*args, **kwargs) 2025-09-07T06:59:42.3593530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3594714Z self_outputs = self.self( 2025-09-07T06:59:42.3595660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3599326Z return func(*args, **kwargs) 2025-09-07T06:59:42.3604642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.3606571Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.3607144Z 2025-09-07T06:59:42.3607284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3607708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3608084Z return mod(**inputs) 2025-09-07T06:59:42.3608514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3608934Z outputs = self.bert( 2025-09-07T06:59:42.3609337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3609840Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3610265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3610684Z layer_outputs = layer_module( 2025-09-07T06:59:42.3611079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3611484Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3611931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3612363Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3612981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3613393Z return func(*args, **kwargs) 2025-09-07T06:59:42.3613791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3614202Z self_outputs = self.self( 2025-09-07T06:59:42.3614598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3614997Z return func(*args, **kwargs) 2025-09-07T06:59:42.3615393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.3615856Z self.key(current_states) 2025-09-07T06:59:42.3615993Z 2025-09-07T06:59:42.3616110Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3616504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3616848Z return mod(**inputs) 2025-09-07T06:59:42.3617240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3617654Z outputs = self.bert( 2025-09-07T06:59:42.3618052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3618480Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3618898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3619315Z layer_outputs = layer_module( 2025-09-07T06:59:42.3619910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3620318Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3620740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3621180Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3621596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3622012Z return func(*args, **kwargs) 2025-09-07T06:59:42.3622426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3622847Z self_outputs = self.self( 2025-09-07T06:59:42.3623282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3623696Z return func(*args, **kwargs) 2025-09-07T06:59:42.3624107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.3624533Z self.value(current_states) 2025-09-07T06:59:42.3624663Z 2025-09-07T06:59:42.3624763Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3625023Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3625455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3626104Z return mod(**inputs) 2025-09-07T06:59:42.3626511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3626939Z outputs = self.bert( 2025-09-07T06:59:42.3627359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3627773Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3628173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3628567Z layer_outputs = layer_module( 2025-09-07T06:59:42.3628920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3629298Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3629697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3630102Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3630511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3630916Z return func(*args, **kwargs) 2025-09-07T06:59:42.3631319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3631823Z self_outputs = self.self( 2025-09-07T06:59:42.3632215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3632614Z return func(*args, **kwargs) 2025-09-07T06:59:42.3633037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.3633523Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.3633727Z 2025-09-07T06:59:42.3633852Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3634247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3634594Z return mod(**inputs) 2025-09-07T06:59:42.3634986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3635393Z outputs = self.bert( 2025-09-07T06:59:42.3635783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3636189Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3636601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3637012Z layer_outputs = layer_module( 2025-09-07T06:59:42.3637392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3637791Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3638195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3638645Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3639059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3639467Z return func(*args, **kwargs) 2025-09-07T06:59:42.3639867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.3640341Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.3640817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.3641282Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3641435Z 2025-09-07T06:59:42.3641558Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3641941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3642291Z return mod(**inputs) 2025-09-07T06:59:42.3642680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3643088Z outputs = self.bert( 2025-09-07T06:59:42.3643481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3643891Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3644298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3644710Z layer_outputs = layer_module( 2025-09-07T06:59:42.3645090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3645481Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3645945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3646380Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3646858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3647297Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3647738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3648240Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3648709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.3649140Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3649291Z 2025-09-07T06:59:42.3649413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3649810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3650152Z return mod(**inputs) 2025-09-07T06:59:42.3650523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3650918Z outputs = self.bert( 2025-09-07T06:59:42.3651285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3651680Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3652092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3652510Z layer_outputs = layer_module( 2025-09-07T06:59:42.3652884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3653274Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3653694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3654120Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3654542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3654953Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3655371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3655843Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3656280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.3656732Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.3657126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.3657474Z return self.act(input) 2025-09-07T06:59:42.3657599Z 2025-09-07T06:59:42.3657707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3658083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3658423Z return mod(**inputs) 2025-09-07T06:59:42.3658806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3659214Z outputs = self.bert( 2025-09-07T06:59:42.3659602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3660027Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3660441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3660849Z layer_outputs = layer_module( 2025-09-07T06:59:42.3661229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3661847Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3662319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3662743Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3663184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3663618Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3664070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.3664589Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.3665056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.3665486Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3665726Z 2025-09-07T06:59:42.3665866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3666282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3666688Z return mod(**inputs) 2025-09-07T06:59:42.3667086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3667523Z outputs = self.bert( 2025-09-07T06:59:42.3667935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3668369Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3668770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3669189Z layer_outputs = layer_module( 2025-09-07T06:59:42.3669567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3669998Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3670475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3670909Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3671336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3671754Z return func(*args, **kwargs) 2025-09-07T06:59:42.3672174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3672592Z self_outputs = self.self( 2025-09-07T06:59:42.3672973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3673386Z return func(*args, **kwargs) 2025-09-07T06:59:42.3673789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.3674361Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.3674644Z 2025-09-07T06:59:42.3674762Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3675155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3675478Z return mod(**inputs) 2025-09-07T06:59:42.3675839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3676234Z outputs = self.bert( 2025-09-07T06:59:42.3676607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3677015Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3677400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3677829Z layer_outputs = layer_module( 2025-09-07T06:59:42.3678189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3678553Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3678950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3679354Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3679745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3680127Z return func(*args, **kwargs) 2025-09-07T06:59:42.3680496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3680874Z self_outputs = self.self( 2025-09-07T06:59:42.3681241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3681616Z return func(*args, **kwargs) 2025-09-07T06:59:42.3681977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.3682360Z self.key(current_states) 2025-09-07T06:59:42.3682484Z 2025-09-07T06:59:42.3682588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3682959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3683291Z return mod(**inputs) 2025-09-07T06:59:42.3683665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3684043Z outputs = self.bert( 2025-09-07T06:59:42.3684410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3684818Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3685201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3685593Z layer_outputs = layer_module( 2025-09-07T06:59:42.3685953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3686346Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3686782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3687207Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3687600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3688001Z return func(*args, **kwargs) 2025-09-07T06:59:42.3688399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3688805Z self_outputs = self.self( 2025-09-07T06:59:42.3689195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3689598Z return func(*args, **kwargs) 2025-09-07T06:59:42.3690000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.3690417Z self.value(current_states) 2025-09-07T06:59:42.3690545Z 2025-09-07T06:59:42.3690634Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3690893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3691280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3691667Z return mod(**inputs) 2025-09-07T06:59:42.3692053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3692533Z outputs = self.bert( 2025-09-07T06:59:42.3692926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3693347Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3693754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3694139Z layer_outputs = layer_module( 2025-09-07T06:59:42.3694494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3694867Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3695264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3695656Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3696052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3696434Z return func(*args, **kwargs) 2025-09-07T06:59:42.3696813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3697203Z self_outputs = self.self( 2025-09-07T06:59:42.3697565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3697944Z return func(*args, **kwargs) 2025-09-07T06:59:42.3698320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.3698775Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.3698973Z 2025-09-07T06:59:42.3699093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3699496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3699852Z return mod(**inputs) 2025-09-07T06:59:42.3700246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3700658Z outputs = self.bert( 2025-09-07T06:59:42.3701041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3701475Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3701884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3702306Z layer_outputs = layer_module( 2025-09-07T06:59:42.3702680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3703069Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3703488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3703909Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3704324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3704724Z return func(*args, **kwargs) 2025-09-07T06:59:42.3705118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.3705662Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.3706158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.3706607Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3706761Z 2025-09-07T06:59:42.3706888Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3707260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3707648Z return mod(**inputs) 2025-09-07T06:59:42.3708025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3708415Z outputs = self.bert( 2025-09-07T06:59:42.3708777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3709180Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3709561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3709953Z layer_outputs = layer_module( 2025-09-07T06:59:42.3710301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3710672Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3711079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3711486Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3711898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3712301Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3712723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3713194Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3713641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.3714047Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3715096Z 2025-09-07T06:59:42.3715204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3715581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3715919Z return mod(**inputs) 2025-09-07T06:59:42.3716291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3716693Z outputs = self.bert( 2025-09-07T06:59:42.3717051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3717478Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3717866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3718258Z layer_outputs = layer_module( 2025-09-07T06:59:42.3718607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3718991Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3719394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3719936Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3720357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3720761Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3721183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3721663Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3722119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.3722548Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.3722929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.3723352Z return self.act(input) 2025-09-07T06:59:42.3723472Z 2025-09-07T06:59:42.3723584Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3723953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3724273Z return mod(**inputs) 2025-09-07T06:59:42.3724637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3725025Z outputs = self.bert( 2025-09-07T06:59:42.3725385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3725772Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3726145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3726527Z layer_outputs = layer_module( 2025-09-07T06:59:42.3726878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3727249Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3727629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3728026Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3728436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3728834Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3729247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.3729734Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.3730171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.3730569Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3730706Z 2025-09-07T06:59:42.3730819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3731178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3731494Z return mod(**inputs) 2025-09-07T06:59:42.3731886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3732269Z outputs = self.bert( 2025-09-07T06:59:42.3732622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3732999Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3733376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3733759Z layer_outputs = layer_module( 2025-09-07T06:59:42.3734110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3734495Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3734879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3735272Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3735664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3736045Z return func(*args, **kwargs) 2025-09-07T06:59:42.3736428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3736823Z self_outputs = self.self( 2025-09-07T06:59:42.3737189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3737601Z return func(*args, **kwargs) 2025-09-07T06:59:42.3737972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.3738482Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.3738750Z 2025-09-07T06:59:42.3738857Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3739215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3739556Z return mod(**inputs) 2025-09-07T06:59:42.3739910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3740282Z outputs = self.bert( 2025-09-07T06:59:42.3740639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3741032Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3741413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3741804Z layer_outputs = layer_module( 2025-09-07T06:59:42.3742147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3742518Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3742911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3743310Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3743692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3744100Z return func(*args, **kwargs) 2025-09-07T06:59:42.3744483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3744900Z self_outputs = self.self( 2025-09-07T06:59:42.3745269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3745715Z return func(*args, **kwargs) 2025-09-07T06:59:42.3746110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.3746549Z self.key(current_states) 2025-09-07T06:59:42.3746678Z 2025-09-07T06:59:42.3746806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3747191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3747529Z return mod(**inputs) 2025-09-07T06:59:42.3747900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3748293Z outputs = self.bert( 2025-09-07T06:59:42.3748662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3749043Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3749432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3749832Z layer_outputs = layer_module( 2025-09-07T06:59:42.3750180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3750544Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3750937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3751340Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3751773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3752169Z return func(*args, **kwargs) 2025-09-07T06:59:42.3752542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3752938Z self_outputs = self.self( 2025-09-07T06:59:42.3753308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3753694Z return func(*args, **kwargs) 2025-09-07T06:59:42.3754074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.3754511Z self.value(current_states) 2025-09-07T06:59:42.3754645Z 2025-09-07T06:59:42.3754730Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3754983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3755358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3755691Z return mod(**inputs) 2025-09-07T06:59:42.3756058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3756455Z outputs = self.bert( 2025-09-07T06:59:42.3756823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3757219Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3757594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3757986Z layer_outputs = layer_module( 2025-09-07T06:59:42.3758340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3758746Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3759134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3759535Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3759928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3760318Z return func(*args, **kwargs) 2025-09-07T06:59:42.3760695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3761093Z self_outputs = self.self( 2025-09-07T06:59:42.3761460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3761839Z return func(*args, **kwargs) 2025-09-07T06:59:42.3762216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.3762672Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.3762856Z 2025-09-07T06:59:42.3762964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3763334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3763671Z return mod(**inputs) 2025-09-07T06:59:42.3764037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3764418Z outputs = self.bert( 2025-09-07T06:59:42.3764767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3765143Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3765515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3765893Z layer_outputs = layer_module( 2025-09-07T06:59:42.3766268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3766631Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3767019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3767396Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3767762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3768142Z return func(*args, **kwargs) 2025-09-07T06:59:42.3768518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.3768974Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.3769422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.3769876Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3770027Z 2025-09-07T06:59:42.3770132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3770498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3770817Z return mod(**inputs) 2025-09-07T06:59:42.3771168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3771535Z outputs = self.bert( 2025-09-07T06:59:42.3771886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3772262Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3772629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3773016Z layer_outputs = layer_module( 2025-09-07T06:59:42.3773357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3773712Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3774085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3774474Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3774872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3775298Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3775700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3776148Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3776568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.3776950Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3777094Z 2025-09-07T06:59:42.3777194Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3777550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3777869Z return mod(**inputs) 2025-09-07T06:59:42.3778219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3778600Z outputs = self.bert( 2025-09-07T06:59:42.3778970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3779366Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3779758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3780141Z layer_outputs = layer_module( 2025-09-07T06:59:42.3780529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3780910Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3781313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3781762Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3782208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3782654Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3783115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3783624Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3784096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.3784582Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.3785030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.3785415Z return self.act(input) 2025-09-07T06:59:42.3785539Z 2025-09-07T06:59:42.3785745Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3786167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3786531Z return mod(**inputs) 2025-09-07T06:59:42.3786937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3787341Z outputs = self.bert( 2025-09-07T06:59:42.3787737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3788133Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3788525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3788919Z layer_outputs = layer_module( 2025-09-07T06:59:42.3789280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3789646Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3790063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3790461Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3790876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3791283Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3791700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.3792180Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.3792630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.3793033Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3793172Z 2025-09-07T06:59:42.3793285Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3793649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3793981Z return mod(**inputs) 2025-09-07T06:59:42.3794352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3794742Z outputs = self.bert( 2025-09-07T06:59:42.3795114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3795532Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3795917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3796309Z layer_outputs = layer_module( 2025-09-07T06:59:42.3796680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3797060Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3797468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3797885Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3798287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3798678Z return func(*args, **kwargs) 2025-09-07T06:59:42.3799083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3799489Z self_outputs = self.self( 2025-09-07T06:59:42.3799875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3800266Z return func(*args, **kwargs) 2025-09-07T06:59:42.3800685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.3801216Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.3801494Z 2025-09-07T06:59:42.3801601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3801975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3802322Z return mod(**inputs) 2025-09-07T06:59:42.3802677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3803059Z outputs = self.bert( 2025-09-07T06:59:42.3803415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3803805Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3804177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3804586Z layer_outputs = layer_module( 2025-09-07T06:59:42.3804937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3805302Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3805687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3806074Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3806476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3806854Z return func(*args, **kwargs) 2025-09-07T06:59:42.3807227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3807614Z self_outputs = self.self( 2025-09-07T06:59:42.3807973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3808348Z return func(*args, **kwargs) 2025-09-07T06:59:42.3808716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.3809099Z self.key(current_states) 2025-09-07T06:59:42.3809221Z 2025-09-07T06:59:42.3809331Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3809724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3810093Z return mod(**inputs) 2025-09-07T06:59:42.3810468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3810842Z outputs = self.bert( 2025-09-07T06:59:42.3811192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3811578Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3811955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3812338Z layer_outputs = layer_module( 2025-09-07T06:59:42.3812676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3813044Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3813430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3813825Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3814204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3814566Z return func(*args, **kwargs) 2025-09-07T06:59:42.3814983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3815365Z self_outputs = self.self( 2025-09-07T06:59:42.3815725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3816097Z return func(*args, **kwargs) 2025-09-07T06:59:42.3816457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.3816917Z self.value(current_states) 2025-09-07T06:59:42.3817037Z 2025-09-07T06:59:42.3817128Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3817371Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3817728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3818060Z return mod(**inputs) 2025-09-07T06:59:42.3818418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3818819Z outputs = self.bert( 2025-09-07T06:59:42.3819179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3819723Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3820128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3820530Z layer_outputs = layer_module( 2025-09-07T06:59:42.3820882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3821237Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3821629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3822033Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3822428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3822813Z return func(*args, **kwargs) 2025-09-07T06:59:42.3823182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3823569Z self_outputs = self.self( 2025-09-07T06:59:42.3823938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3824321Z return func(*args, **kwargs) 2025-09-07T06:59:42.3824756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.3825211Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.3825409Z 2025-09-07T06:59:42.3825515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3825949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3826292Z return mod(**inputs) 2025-09-07T06:59:42.3826662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3827053Z outputs = self.bert( 2025-09-07T06:59:42.3827426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3827832Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3828225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3828614Z layer_outputs = layer_module( 2025-09-07T06:59:42.3828971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3829340Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3829735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3830130Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3830523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3830903Z return func(*args, **kwargs) 2025-09-07T06:59:42.3831331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.3831796Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.3832247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.3832662Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3832815Z 2025-09-07T06:59:42.3832927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3833298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3833657Z return mod(**inputs) 2025-09-07T06:59:42.3834019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3834406Z outputs = self.bert( 2025-09-07T06:59:42.3834767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3835161Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3835545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3835933Z layer_outputs = layer_module( 2025-09-07T06:59:42.3836289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3836658Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3837059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3837467Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3837889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3838306Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3838736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3839240Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3839662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.3840061Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3840208Z 2025-09-07T06:59:42.3840315Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3840681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3840996Z return mod(**inputs) 2025-09-07T06:59:42.3841355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3841733Z outputs = self.bert( 2025-09-07T06:59:42.3842093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3842481Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3842852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3843240Z layer_outputs = layer_module( 2025-09-07T06:59:42.3843590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3843952Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3844332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3844723Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3845124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3845541Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3845951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3846399Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3846819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.3847241Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.3847620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.3847978Z return self.act(input) 2025-09-07T06:59:42.3848092Z 2025-09-07T06:59:42.3848194Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3848555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3848880Z return mod(**inputs) 2025-09-07T06:59:42.3849240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3849626Z outputs = self.bert( 2025-09-07T06:59:42.3849990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3850386Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3850774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3851219Z layer_outputs = layer_module( 2025-09-07T06:59:42.3851556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3851919Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3852299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3852694Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3853130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3853525Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3853935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.3854407Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.3854846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.3855229Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3855373Z 2025-09-07T06:59:42.3855478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3855847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3856170Z return mod(**inputs) 2025-09-07T06:59:42.3856529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3856901Z outputs = self.bert( 2025-09-07T06:59:42.3857263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3857667Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3858077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3858516Z layer_outputs = layer_module( 2025-09-07T06:59:42.3858896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3859306Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3859756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3860225Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3860672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3861100Z return func(*args, **kwargs) 2025-09-07T06:59:42.3861514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3861950Z self_outputs = self.self( 2025-09-07T06:59:42.3862354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3862778Z return func(*args, **kwargs) 2025-09-07T06:59:42.3863217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.3863813Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.3864109Z 2025-09-07T06:59:42.3864233Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3864642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3864998Z return mod(**inputs) 2025-09-07T06:59:42.3865401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3865930Z outputs = self.bert( 2025-09-07T06:59:42.3866337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3866779Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3867208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3867638Z layer_outputs = layer_module( 2025-09-07T06:59:42.3868042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3868415Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3868837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3869240Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3869635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3870022Z return func(*args, **kwargs) 2025-09-07T06:59:42.3870393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3870778Z self_outputs = self.self( 2025-09-07T06:59:42.3871154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3871542Z return func(*args, **kwargs) 2025-09-07T06:59:42.3871937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.3872342Z self.key(current_states) 2025-09-07T06:59:42.3872479Z 2025-09-07T06:59:42.3872592Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3872983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3873322Z return mod(**inputs) 2025-09-07T06:59:42.3873683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3874059Z outputs = self.bert( 2025-09-07T06:59:42.3874452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3874868Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3875277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3875705Z layer_outputs = layer_module( 2025-09-07T06:59:42.3876084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3876479Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3876898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3877322Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3877729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3878152Z return func(*args, **kwargs) 2025-09-07T06:59:42.3878556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3878962Z self_outputs = self.self( 2025-09-07T06:59:42.3879344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3879748Z return func(*args, **kwargs) 2025-09-07T06:59:42.3880156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.3880549Z self.value(current_states) 2025-09-07T06:59:42.3880672Z 2025-09-07T06:59:42.3880777Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3881016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3881384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3881722Z return mod(**inputs) 2025-09-07T06:59:42.3882092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3882476Z outputs = self.bert( 2025-09-07T06:59:42.3882831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3883216Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3883639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3884039Z layer_outputs = layer_module( 2025-09-07T06:59:42.3884397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3884797Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3885230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3885640Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3886041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3886425Z return func(*args, **kwargs) 2025-09-07T06:59:42.3886810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3887213Z self_outputs = self.self( 2025-09-07T06:59:42.3887587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3887966Z return func(*args, **kwargs) 2025-09-07T06:59:42.3888346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.3888814Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.3889018Z 2025-09-07T06:59:42.3889138Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3908268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3908870Z return mod(**inputs) 2025-09-07T06:59:42.3909318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3909898Z outputs = self.bert( 2025-09-07T06:59:42.3910323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3910759Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3911198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3911626Z layer_outputs = layer_module( 2025-09-07T06:59:42.3912018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3912467Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3912894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3913331Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3913761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3914179Z return func(*args, **kwargs) 2025-09-07T06:59:42.3914583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.3915057Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.3915540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.3915974Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3916135Z 2025-09-07T06:59:42.3916265Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3916662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3917025Z return mod(**inputs) 2025-09-07T06:59:42.3917428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3917840Z outputs = self.bert( 2025-09-07T06:59:42.3918290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3918720Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3919124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3919538Z layer_outputs = layer_module( 2025-09-07T06:59:42.3920083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3920487Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3920907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3921338Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3921792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3922231Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3922685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3923182Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3923655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.3924087Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3924238Z 2025-09-07T06:59:42.3924363Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3924761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3925107Z return mod(**inputs) 2025-09-07T06:59:42.3925569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3925995Z outputs = self.bert( 2025-09-07T06:59:42.3926369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3926769Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3927154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3927554Z layer_outputs = layer_module( 2025-09-07T06:59:42.3927940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3928314Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3928701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3929103Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3929520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3929929Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3930352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3930814Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3931255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.3931691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.3932083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.3932435Z return self.act(input) 2025-09-07T06:59:42.3932550Z 2025-09-07T06:59:42.3932659Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3933033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3933421Z return mod(**inputs) 2025-09-07T06:59:42.3933801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3934186Z outputs = self.bert( 2025-09-07T06:59:42.3934563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3934964Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3935352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3935746Z layer_outputs = layer_module( 2025-09-07T06:59:42.3936099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3936475Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3936871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3937268Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3937673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3938081Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3938498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.3938978Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.3939432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.3939826Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3939993Z 2025-09-07T06:59:42.3940102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3940476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3940818Z return mod(**inputs) 2025-09-07T06:59:42.3941189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3941573Z outputs = self.bert( 2025-09-07T06:59:42.3941943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3942385Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3942794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3943200Z layer_outputs = layer_module( 2025-09-07T06:59:42.3943586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3944002Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3944426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3944865Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3945297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3945801Z return func(*args, **kwargs) 2025-09-07T06:59:42.3946217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3946647Z self_outputs = self.self( 2025-09-07T06:59:42.3947057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3947471Z return func(*args, **kwargs) 2025-09-07T06:59:42.3947877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.3948522Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.3948814Z 2025-09-07T06:59:42.3948935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3949321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3949675Z return mod(**inputs) 2025-09-07T06:59:42.3950067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3950476Z outputs = self.bert( 2025-09-07T06:59:42.3950867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3951275Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3951687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3952096Z layer_outputs = layer_module( 2025-09-07T06:59:42.3952474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3952869Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3953280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3953709Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3954114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3954480Z return func(*args, **kwargs) 2025-09-07T06:59:42.3954830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3955219Z self_outputs = self.self( 2025-09-07T06:59:42.3955571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3955940Z return func(*args, **kwargs) 2025-09-07T06:59:42.3956296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.3956655Z self.key(current_states) 2025-09-07T06:59:42.3956772Z 2025-09-07T06:59:42.3956874Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3957226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3957560Z return mod(**inputs) 2025-09-07T06:59:42.3957905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3958279Z outputs = self.bert( 2025-09-07T06:59:42.3958633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3959021Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3959406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3959788Z layer_outputs = layer_module( 2025-09-07T06:59:42.3960151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3960528Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3960923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3961322Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3961697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3962068Z return func(*args, **kwargs) 2025-09-07T06:59:42.3962446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3962831Z self_outputs = self.self( 2025-09-07T06:59:42.3963213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3963585Z return func(*args, **kwargs) 2025-09-07T06:59:42.3963948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.3964323Z self.value(current_states) 2025-09-07T06:59:42.3964447Z 2025-09-07T06:59:42.3964535Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.3964769Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3965133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3965459Z return mod(**inputs) 2025-09-07T06:59:42.3965820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3966186Z outputs = self.bert( 2025-09-07T06:59:42.3966548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3966934Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3967313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3967695Z layer_outputs = layer_module( 2025-09-07T06:59:42.3968035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3968393Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3968770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3969154Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3969547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3969924Z return func(*args, **kwargs) 2025-09-07T06:59:42.3970291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.3970673Z self_outputs = self.self( 2025-09-07T06:59:42.3971021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3971404Z return func(*args, **kwargs) 2025-09-07T06:59:42.3971771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.3972208Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.3972391Z 2025-09-07T06:59:42.3972502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3972860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3973176Z return mod(**inputs) 2025-09-07T06:59:42.3973532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3973907Z outputs = self.bert( 2025-09-07T06:59:42.3974262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3974638Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3975014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3975395Z layer_outputs = layer_module( 2025-09-07T06:59:42.3975749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3976124Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3976514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.3976931Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.3977310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.3977670Z return func(*args, **kwargs) 2025-09-07T06:59:42.3978035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.3978476Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.3978922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.3979325Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3979478Z 2025-09-07T06:59:42.3979589Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3979967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3980307Z return mod(**inputs) 2025-09-07T06:59:42.3980681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3981066Z outputs = self.bert( 2025-09-07T06:59:42.3981440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3981841Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3982234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3982621Z layer_outputs = layer_module( 2025-09-07T06:59:42.3982981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3983370Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3983760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3984166Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3984573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3985000Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3985441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3986052Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3986532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.3986979Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.3987141Z 2025-09-07T06:59:42.3987264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3987672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3988044Z return mod(**inputs) 2025-09-07T06:59:42.3988410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3988805Z outputs = self.bert( 2025-09-07T06:59:42.3989180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3989585Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3989978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3990367Z layer_outputs = layer_module( 2025-09-07T06:59:42.3990725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3991105Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3991541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.3991944Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.3992356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.3992786Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.3993240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.3993736Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.3994188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.3994647Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.3995058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.3995430Z return self.act(input) 2025-09-07T06:59:42.3995555Z 2025-09-07T06:59:42.3995677Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.3996058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.3996411Z return mod(**inputs) 2025-09-07T06:59:42.3996802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.3997193Z outputs = self.bert( 2025-09-07T06:59:42.3997564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.3997957Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.3998365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.3998802Z layer_outputs = layer_module( 2025-09-07T06:59:42.3999180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.3999570Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.3999990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4000417Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4000854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4001303Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4001733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.4002222Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.4002688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.4003115Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4003264Z 2025-09-07T06:59:42.4003383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4003765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4004113Z return mod(**inputs) 2025-09-07T06:59:42.4004507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4004914Z outputs = self.bert( 2025-09-07T06:59:42.4005291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4005707Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4006114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4006525Z layer_outputs = layer_module( 2025-09-07T06:59:42.4006998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4007385Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4007806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4008233Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4008655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4009063Z return func(*args, **kwargs) 2025-09-07T06:59:42.4009462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4009874Z self_outputs = self.self( 2025-09-07T06:59:42.4010271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4010679Z return func(*args, **kwargs) 2025-09-07T06:59:42.4011072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.4011634Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.4011932Z 2025-09-07T06:59:42.4012046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4012440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4012790Z return mod(**inputs) 2025-09-07T06:59:42.4013170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4013577Z outputs = self.bert( 2025-09-07T06:59:42.4013981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4014402Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4014813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4015221Z layer_outputs = layer_module( 2025-09-07T06:59:42.4015600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4015996Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4016437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4016858Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4017283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4017376Z return func(*args, **kwargs) 2025-09-07T06:59:42.4017652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4017734Z self_outputs = self.self( 2025-09-07T06:59:42.4018006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4018086Z return func(*args, **kwargs) 2025-09-07T06:59:42.4018361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.4018443Z self.key(current_states) 2025-09-07T06:59:42.4018446Z 2025-09-07T06:59:42.4018570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4018792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4018867Z return mod(**inputs) 2025-09-07T06:59:42.4019151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4019225Z outputs = self.bert( 2025-09-07T06:59:42.4019543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4019805Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4020079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4020166Z layer_outputs = layer_module( 2025-09-07T06:59:42.4020410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4020503Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4020774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4020863Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4021141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4021220Z return func(*args, **kwargs) 2025-09-07T06:59:42.4021503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4021579Z self_outputs = self.self( 2025-09-07T06:59:42.4021842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4021927Z return func(*args, **kwargs) 2025-09-07T06:59:42.4022200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.4022287Z self.value(current_states) 2025-09-07T06:59:42.4022291Z 2025-09-07T06:59:42.4022381Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.4022502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4022783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4022857Z return mod(**inputs) 2025-09-07T06:59:42.4023139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4023209Z outputs = self.bert( 2025-09-07T06:59:42.4023488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4023568Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4023857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4023942Z layer_outputs = layer_module( 2025-09-07T06:59:42.4024182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4024276Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4024543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4024637Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4024908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4024981Z return func(*args, **kwargs) 2025-09-07T06:59:42.4025256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4025336Z self_outputs = self.self( 2025-09-07T06:59:42.4025662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4025750Z return func(*args, **kwargs) 2025-09-07T06:59:42.4026023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.4026186Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.4026190Z 2025-09-07T06:59:42.4026362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4026594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4026873Z return mod(**inputs) 2025-09-07T06:59:42.4027159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4027239Z outputs = self.bert( 2025-09-07T06:59:42.4027515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4027602Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4027875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4027960Z layer_outputs = layer_module( 2025-09-07T06:59:42.4028206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4028297Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4028585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4028674Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4028955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4029034Z return func(*args, **kwargs) 2025-09-07T06:59:42.4029308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.4029466Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.4029742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.4029869Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4029873Z 2025-09-07T06:59:42.4029991Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4030220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4030295Z return mod(**inputs) 2025-09-07T06:59:42.4030572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4030672Z outputs = self.bert( 2025-09-07T06:59:42.4030952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4031040Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4031313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4031394Z layer_outputs = layer_module( 2025-09-07T06:59:42.4031650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4031736Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4032015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4032110Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4032402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4032496Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4032809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4032953Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4033231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.4033330Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4033370Z 2025-09-07T06:59:42.4033486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4033710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4033791Z return mod(**inputs) 2025-09-07T06:59:42.4034071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4034154Z outputs = self.bert( 2025-09-07T06:59:42.4034436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4034517Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4034800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4034872Z layer_outputs = layer_module( 2025-09-07T06:59:42.4035099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4035177Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4035423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4035512Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4035772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4035859Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4036137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4036267Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4036526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.4036642Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.4036867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.4036936Z return self.act(input) 2025-09-07T06:59:42.4036940Z 2025-09-07T06:59:42.4037050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4037251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4037338Z return mod(**inputs) 2025-09-07T06:59:42.4037601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4037666Z outputs = self.bert( 2025-09-07T06:59:42.4037923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4038013Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4038264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4038344Z layer_outputs = layer_module( 2025-09-07T06:59:42.4038569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4038655Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4038907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4038993Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4039273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4039350Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4039659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.4039830Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.4040085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.4040183Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4040186Z 2025-09-07T06:59:42.4040288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4040495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4040562Z return mod(**inputs) 2025-09-07T06:59:42.4040815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4040880Z outputs = self.bert( 2025-09-07T06:59:42.4041127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4041209Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4041459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4041540Z layer_outputs = layer_module( 2025-09-07T06:59:42.4041765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4041845Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4042105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4042189Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4042445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4042547Z return func(*args, **kwargs) 2025-09-07T06:59:42.4042797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4042878Z self_outputs = self.self( 2025-09-07T06:59:42.4043124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4043200Z return func(*args, **kwargs) 2025-09-07T06:59:42.4043452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.4043694Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.4043697Z 2025-09-07T06:59:42.4043802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4044012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4044086Z return mod(**inputs) 2025-09-07T06:59:42.4044333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4044409Z outputs = self.bert( 2025-09-07T06:59:42.4044662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4044736Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4044993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4045066Z layer_outputs = layer_module( 2025-09-07T06:59:42.4045296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4045374Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4045638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4045719Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4045994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4046072Z return func(*args, **kwargs) 2025-09-07T06:59:42.4046314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4046388Z self_outputs = self.self( 2025-09-07T06:59:42.4046626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4046697Z return func(*args, **kwargs) 2025-09-07T06:59:42.4046950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.4047021Z self.key(current_states) 2025-09-07T06:59:42.4047024Z 2025-09-07T06:59:42.4047132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4047329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4047396Z return mod(**inputs) 2025-09-07T06:59:42.4047659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4047725Z outputs = self.bert( 2025-09-07T06:59:42.4047981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4048056Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4048308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4048387Z layer_outputs = layer_module( 2025-09-07T06:59:42.4048613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4048722Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4048972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4049063Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4049343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4049415Z return func(*args, **kwargs) 2025-09-07T06:59:42.4049691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4049788Z self_outputs = self.self( 2025-09-07T06:59:42.4050053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4050124Z return func(*args, **kwargs) 2025-09-07T06:59:42.4050390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.4050477Z self.value(current_states) 2025-09-07T06:59:42.4050481Z 2025-09-07T06:59:42.4050570Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.4050691Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4050904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4050974Z return mod(**inputs) 2025-09-07T06:59:42.4051248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4051319Z outputs = self.bert( 2025-09-07T06:59:42.4051592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4051670Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4051951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4052025Z layer_outputs = layer_module( 2025-09-07T06:59:42.4052278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4052369Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4052632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4052725Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4052987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4053062Z return func(*args, **kwargs) 2025-09-07T06:59:42.4053336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4053409Z self_outputs = self.self( 2025-09-07T06:59:42.4053676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4053750Z return func(*args, **kwargs) 2025-09-07T06:59:42.4054017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.4054171Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.4054175Z 2025-09-07T06:59:42.4054284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4054508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4054581Z return mod(**inputs) 2025-09-07T06:59:42.4054858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4054928Z outputs = self.bert( 2025-09-07T06:59:42.4055193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4055303Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4055573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4055655Z layer_outputs = layer_module( 2025-09-07T06:59:42.4055892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4055975Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4056248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4056354Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4056623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4056696Z return func(*args, **kwargs) 2025-09-07T06:59:42.4056968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.4057118Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.4057385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.4057483Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4057487Z 2025-09-07T06:59:42.4057599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4057817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4057888Z return mod(**inputs) 2025-09-07T06:59:42.4058154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4058232Z outputs = self.bert( 2025-09-07T06:59:42.4058495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4058583Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4058881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4058957Z layer_outputs = layer_module( 2025-09-07T06:59:42.4059197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4059280Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4059551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4059641Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4059925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4060007Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4060307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4060449Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4060713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.4060810Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4060814Z 2025-09-07T06:59:42.4060923Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4061136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4061216Z return mod(**inputs) 2025-09-07T06:59:42.4061486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4061561Z outputs = self.bert( 2025-09-07T06:59:42.4061830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4061926Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4062201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4062277Z layer_outputs = layer_module( 2025-09-07T06:59:42.4062520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4062603Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4062879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4062982Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4063264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4063354Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4063654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4063795Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4064066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.4064188Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.4064426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.4064503Z return self.act(input) 2025-09-07T06:59:42.4064507Z 2025-09-07T06:59:42.4064623Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4064838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4064914Z return mod(**inputs) 2025-09-07T06:59:42.4065182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4065251Z outputs = self.bert( 2025-09-07T06:59:42.4065555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4065745Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4066052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4066130Z layer_outputs = layer_module( 2025-09-07T06:59:42.4066373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4066466Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4066759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4066861Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4067162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4067247Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4067559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.4067706Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.4067981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.4068072Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4068076Z 2025-09-07T06:59:42.4068196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4068412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4068521Z return mod(**inputs) 2025-09-07T06:59:42.4068798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4068871Z outputs = self.bert( 2025-09-07T06:59:42.4069147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4069225Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4069486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4069590Z layer_outputs = layer_module( 2025-09-07T06:59:42.4069828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4069921Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4070187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4070285Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4070549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4070625Z return func(*args, **kwargs) 2025-09-07T06:59:42.4070897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4070972Z self_outputs = self.self( 2025-09-07T06:59:42.4071238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4071314Z return func(*args, **kwargs) 2025-09-07T06:59:42.4071578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.4071812Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.4071819Z 2025-09-07T06:59:42.4071928Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4072203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4072274Z return mod(**inputs) 2025-09-07T06:59:42.4072547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4072624Z outputs = self.bert( 2025-09-07T06:59:42.4072892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4072984Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4073253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4073336Z layer_outputs = layer_module( 2025-09-07T06:59:42.4073573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4073660Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4073935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4074024Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4074290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4074363Z return func(*args, **kwargs) 2025-09-07T06:59:42.4074647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4074728Z self_outputs = self.self( 2025-09-07T06:59:42.4074971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4075048Z return func(*args, **kwargs) 2025-09-07T06:59:42.4075296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.4075386Z self.key(current_states) 2025-09-07T06:59:42.4075397Z 2025-09-07T06:59:42.4075503Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4075708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4075780Z return mod(**inputs) 2025-09-07T06:59:42.4076038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4076129Z outputs = self.bert( 2025-09-07T06:59:42.4076383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4076456Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4076713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4076786Z layer_outputs = layer_module( 2025-09-07T06:59:42.4077020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4077098Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4077347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4077438Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4077682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4077760Z return func(*args, **kwargs) 2025-09-07T06:59:42.4078008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4078077Z self_outputs = self.self( 2025-09-07T06:59:42.4078326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4078397Z return func(*args, **kwargs) 2025-09-07T06:59:42.4078691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.4078767Z self.value(current_states) 2025-09-07T06:59:42.4078771Z 2025-09-07T06:59:42.4078862Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.4078967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4079167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4079243Z return mod(**inputs) 2025-09-07T06:59:42.4079497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4079570Z outputs = self.bert( 2025-09-07T06:59:42.4079822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4079897Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4080161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4080233Z layer_outputs = layer_module( 2025-09-07T06:59:42.4080465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4080544Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4080796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4080886Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4081132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4081208Z return func(*args, **kwargs) 2025-09-07T06:59:42.4081460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4081560Z self_outputs = self.self( 2025-09-07T06:59:42.4081809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4081879Z return func(*args, **kwargs) 2025-09-07T06:59:42.4082140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.4082277Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.4082295Z 2025-09-07T06:59:42.4082406Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4082608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4082676Z return mod(**inputs) 2025-09-07T06:59:42.4082936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4083003Z outputs = self.bert( 2025-09-07T06:59:42.4083269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4083344Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4083596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4083676Z layer_outputs = layer_module( 2025-09-07T06:59:42.4083899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4083986Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4084237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4084325Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4084573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4084642Z return func(*args, **kwargs) 2025-09-07T06:59:42.4084928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.4085063Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.4085319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.4085404Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4085409Z 2025-09-07T06:59:42.4085513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4085725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4085791Z return mod(**inputs) 2025-09-07T06:59:42.4086053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4086122Z outputs = self.bert( 2025-09-07T06:59:42.4086383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4086456Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4086713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4086791Z layer_outputs = layer_module( 2025-09-07T06:59:42.4087010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4087096Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4087337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4087420Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4087704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4087780Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4088062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4088182Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4088424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.4088558Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4088562Z 2025-09-07T06:59:42.4088669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4088883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4088952Z return mod(**inputs) 2025-09-07T06:59:42.4089225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4089295Z outputs = self.bert( 2025-09-07T06:59:42.4089554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4089638Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4089889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4089972Z layer_outputs = layer_module( 2025-09-07T06:59:42.4090204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4090287Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4090547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4090635Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4090924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4091030Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4091309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4091435Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4091678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.4091801Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.4092019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.4092098Z return self.act(input) 2025-09-07T06:59:42.4092101Z 2025-09-07T06:59:42.4092210Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4092419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4092498Z return mod(**inputs) 2025-09-07T06:59:42.4092757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4092832Z outputs = self.bert( 2025-09-07T06:59:42.4093090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4093166Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4093428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4093503Z layer_outputs = layer_module( 2025-09-07T06:59:42.4093740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4093835Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4094093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4094179Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4094447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4094537Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4094847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.4095031Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.4095311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.4095403Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4095408Z 2025-09-07T06:59:42.4095530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4095755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4095837Z return mod(**inputs) 2025-09-07T06:59:42.4096114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4096195Z outputs = self.bert( 2025-09-07T06:59:42.4096480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4096560Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4096836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4096912Z layer_outputs = layer_module( 2025-09-07T06:59:42.4097158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4097243Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4097544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4097645Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4097913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4098000Z return func(*args, **kwargs) 2025-09-07T06:59:42.4098275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4098353Z self_outputs = self.self( 2025-09-07T06:59:42.4098630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4098708Z return func(*args, **kwargs) 2025-09-07T06:59:42.4098988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.4099226Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.4099230Z 2025-09-07T06:59:42.4099352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4099588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4099660Z return mod(**inputs) 2025-09-07T06:59:42.4099956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4100029Z outputs = self.bert( 2025-09-07T06:59:42.4100322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4100403Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4100685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4100792Z layer_outputs = layer_module( 2025-09-07T06:59:42.4101038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4101131Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4101413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4101503Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4101803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4101879Z return func(*args, **kwargs) 2025-09-07T06:59:42.4102160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4102237Z self_outputs = self.self( 2025-09-07T06:59:42.4102516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4102592Z return func(*args, **kwargs) 2025-09-07T06:59:42.4102882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.4102967Z self.key(current_states) 2025-09-07T06:59:42.4102971Z 2025-09-07T06:59:42.4103083Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4103309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4103383Z return mod(**inputs) 2025-09-07T06:59:42.4103658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4103736Z outputs = self.bert( 2025-09-07T06:59:42.4104010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4104101Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4104417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4104498Z layer_outputs = layer_module( 2025-09-07T06:59:42.4104752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4104839Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4105123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4105215Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4105488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4105564Z return func(*args, **kwargs) 2025-09-07T06:59:42.4105925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4106020Z self_outputs = self.self( 2025-09-07T06:59:42.4106291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4106374Z return func(*args, **kwargs) 2025-09-07T06:59:42.4106656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.4106737Z self.value(current_states) 2025-09-07T06:59:42.4106745Z 2025-09-07T06:59:42.4106846Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.4106961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4107191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4107263Z return mod(**inputs) 2025-09-07T06:59:42.4107541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4107646Z outputs = self.bert( 2025-09-07T06:59:42.4107932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4108019Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4108303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4108388Z layer_outputs = layer_module( 2025-09-07T06:59:42.4108634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4108735Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4109015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4109103Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4109377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4109453Z return func(*args, **kwargs) 2025-09-07T06:59:42.4109727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4109810Z self_outputs = self.self( 2025-09-07T06:59:42.4110077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4110159Z return func(*args, **kwargs) 2025-09-07T06:59:42.4110432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.4110580Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.4110591Z 2025-09-07T06:59:42.4110704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4110924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4111003Z return mod(**inputs) 2025-09-07T06:59:42.4111318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4111399Z outputs = self.bert( 2025-09-07T06:59:42.4111683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4111762Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4112042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4112122Z layer_outputs = layer_module( 2025-09-07T06:59:42.4112374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4112460Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4112730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4112832Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4113098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4113182Z return func(*args, **kwargs) 2025-09-07T06:59:42.4113451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.4113594Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.4113875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.4113968Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4113972Z 2025-09-07T06:59:42.4114093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4114331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4114413Z return mod(**inputs) 2025-09-07T06:59:42.4114692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4114763Z outputs = self.bert( 2025-09-07T06:59:42.4115050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4115129Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4115427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4115506Z layer_outputs = layer_module( 2025-09-07T06:59:42.4115748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4115843Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4116120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4116221Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4116502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4116591Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4116891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4117022Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4117293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.4117382Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4117386Z 2025-09-07T06:59:42.4117501Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4117717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4117819Z return mod(**inputs) 2025-09-07T06:59:42.4118096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4118166Z outputs = self.bert( 2025-09-07T06:59:42.4118443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4118521Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4118786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4118867Z layer_outputs = layer_module( 2025-09-07T06:59:42.4119103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4119196Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4119459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4119710Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4120002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4120086Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4120395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4120527Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4120800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.4120923Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.4121204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.4121287Z return self.act(input) 2025-09-07T06:59:42.4121290Z 2025-09-07T06:59:42.4121401Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4121622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4121692Z return mod(**inputs) 2025-09-07T06:59:42.4121965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4122063Z outputs = self.bert( 2025-09-07T06:59:42.4122333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4122419Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4122685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4122773Z layer_outputs = layer_module( 2025-09-07T06:59:42.4123015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4123099Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4123374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4123464Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4123751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4123834Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4124134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.4124287Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.4124554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.4124704Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4124708Z 2025-09-07T06:59:42.4124819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4125042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4125112Z return mod(**inputs) 2025-09-07T06:59:42.4125379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4125459Z outputs = self.bert( 2025-09-07T06:59:42.4125725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4125809Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4126078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4126156Z layer_outputs = layer_module( 2025-09-07T06:59:42.4126408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4126492Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4126754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4126835Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4127074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4127152Z return func(*args, **kwargs) 2025-09-07T06:59:42.4127397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4127473Z self_outputs = self.self( 2025-09-07T06:59:42.4127739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4127823Z return func(*args, **kwargs) 2025-09-07T06:59:42.4128090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.4128317Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.4128321Z 2025-09-07T06:59:42.4128440Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4128674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4128751Z return mod(**inputs) 2025-09-07T06:59:42.4129029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4129100Z outputs = self.bert( 2025-09-07T06:59:42.4129384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4129464Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4129746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4129817Z layer_outputs = layer_module( 2025-09-07T06:59:42.4130053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4130131Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4130388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4130480Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4130730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4130819Z return func(*args, **kwargs) 2025-09-07T06:59:42.4131066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4131166Z self_outputs = self.self( 2025-09-07T06:59:42.4131411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4131479Z return func(*args, **kwargs) 2025-09-07T06:59:42.4131738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.4131810Z self.key(current_states) 2025-09-07T06:59:42.4131814Z 2025-09-07T06:59:42.4131918Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4132126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4132190Z return mod(**inputs) 2025-09-07T06:59:42.4132451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4132518Z outputs = self.bert( 2025-09-07T06:59:42.4132780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4132854Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4133104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4133181Z layer_outputs = layer_module( 2025-09-07T06:59:42.4133406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4133495Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4133753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4133835Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4134103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4134172Z return func(*args, **kwargs) 2025-09-07T06:59:42.4134428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4134499Z self_outputs = self.self( 2025-09-07T06:59:42.4134740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4134817Z return func(*args, **kwargs) 2025-09-07T06:59:42.4135082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.4135162Z self.value(current_states) 2025-09-07T06:59:42.4135166Z 2025-09-07T06:59:42.4135250Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.4135361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4135563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4135629Z return mod(**inputs) 2025-09-07T06:59:42.4135893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4135960Z outputs = self.bert( 2025-09-07T06:59:42.4136218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4136291Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4136552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4136635Z layer_outputs = layer_module( 2025-09-07T06:59:42.4136873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4136963Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4137225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4137341Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4137610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4137683Z return func(*args, **kwargs) 2025-09-07T06:59:42.4137953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4138028Z self_outputs = self.self( 2025-09-07T06:59:42.4138287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4138377Z return func(*args, **kwargs) 2025-09-07T06:59:42.4138626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.4138771Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.4138774Z 2025-09-07T06:59:42.4138880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4139090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4139157Z return mod(**inputs) 2025-09-07T06:59:42.4139411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4139486Z outputs = self.bert( 2025-09-07T06:59:42.4139740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4139819Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4140071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4140145Z layer_outputs = layer_module( 2025-09-07T06:59:42.4140403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4140490Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4140760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4140845Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4141109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4141208Z return func(*args, **kwargs) 2025-09-07T06:59:42.4141471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.4141619Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.4141880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.4141978Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4141982Z 2025-09-07T06:59:42.4142092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4142305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4142385Z return mod(**inputs) 2025-09-07T06:59:42.4142651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4142726Z outputs = self.bert( 2025-09-07T06:59:42.4142994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4143072Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4143344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4143421Z layer_outputs = layer_module( 2025-09-07T06:59:42.4143665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4144785Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4145080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4145174Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4145469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4145564Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4146123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4146275Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4146549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.4146645Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4146659Z 2025-09-07T06:59:42.4146775Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4146999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4147081Z return mod(**inputs) 2025-09-07T06:59:42.4147366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4147447Z outputs = self.bert( 2025-09-07T06:59:42.4147716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4147794Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4148069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4148186Z layer_outputs = layer_module( 2025-09-07T06:59:42.4148432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4148516Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4148779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4148875Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4149157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4149267Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4149565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4149695Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4149972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.4150096Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.4150334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.4150407Z return self.act(input) 2025-09-07T06:59:42.4150411Z 2025-09-07T06:59:42.4150528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4150748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4150819Z return mod(**inputs) 2025-09-07T06:59:42.4151098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4151168Z outputs = self.bert( 2025-09-07T06:59:42.4151444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4151524Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4151824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4151913Z layer_outputs = layer_module( 2025-09-07T06:59:42.4152151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4152243Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4152508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4152600Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4152887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4152969Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4153280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.4153428Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.4153701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.4153789Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4153793Z 2025-09-07T06:59:42.4153903Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4154128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4154201Z return mod(**inputs) 2025-09-07T06:59:42.4154475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4154546Z outputs = self.bert( 2025-09-07T06:59:42.4154813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4154917Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4155193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4155270Z layer_outputs = layer_module( 2025-09-07T06:59:42.4155497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4155584Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4155851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4155934Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4156184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4156255Z return func(*args, **kwargs) 2025-09-07T06:59:42.4156511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4156584Z self_outputs = self.self( 2025-09-07T06:59:42.4156828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4156904Z return func(*args, **kwargs) 2025-09-07T06:59:42.4157151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T06:59:42.4157367Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T06:59:42.4157371Z 2025-09-07T06:59:42.4157475Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4157684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4157752Z return mod(**inputs) 2025-09-07T06:59:42.4158019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4158098Z outputs = self.bert( 2025-09-07T06:59:42.4158400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4158487Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4158758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4158833Z layer_outputs = layer_module( 2025-09-07T06:59:42.4159084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4159169Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4159448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4159543Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4159812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4159891Z return func(*args, **kwargs) 2025-09-07T06:59:42.4160145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4160225Z self_outputs = self.self( 2025-09-07T06:59:42.4160471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4160550Z return func(*args, **kwargs) 2025-09-07T06:59:42.4160804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T06:59:42.4160875Z self.key(current_states) 2025-09-07T06:59:42.4160878Z 2025-09-07T06:59:42.4160992Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4161215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4161288Z return mod(**inputs) 2025-09-07T06:59:42.4161554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4161624Z outputs = self.bert( 2025-09-07T06:59:42.4161899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4161977Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4162266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4162341Z layer_outputs = layer_module( 2025-09-07T06:59:42.4162576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4162668Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4162937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4163035Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4163298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4163388Z return func(*args, **kwargs) 2025-09-07T06:59:42.4163649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4163721Z self_outputs = self.self( 2025-09-07T06:59:42.4163974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4164041Z return func(*args, **kwargs) 2025-09-07T06:59:42.4164298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T06:59:42.4164372Z self.value(current_states) 2025-09-07T06:59:42.4164377Z 2025-09-07T06:59:42.4164460Z cudagraph partition due to non gpu ops 2025-09-07T06:59:42.4164605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4164812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4164886Z return mod(**inputs) 2025-09-07T06:59:42.4165143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4165208Z outputs = self.bert( 2025-09-07T06:59:42.4165476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4165549Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4165809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4165882Z layer_outputs = layer_module( 2025-09-07T06:59:42.4166108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4166199Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4166454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4166546Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4166806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4166887Z return func(*args, **kwargs) 2025-09-07T06:59:42.4167156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T06:59:42.4167229Z self_outputs = self.self( 2025-09-07T06:59:42.4167501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4167594Z return func(*args, **kwargs) 2025-09-07T06:59:42.4167873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T06:59:42.4168019Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T06:59:42.4168023Z 2025-09-07T06:59:42.4168131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4168352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4168449Z return mod(**inputs) 2025-09-07T06:59:42.4168708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4168776Z outputs = self.bert( 2025-09-07T06:59:42.4169034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4169108Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4169358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4169438Z layer_outputs = layer_module( 2025-09-07T06:59:42.4169664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4169749Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4170003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T06:59:42.4170087Z self_attention_outputs = self.attention( 2025-09-07T06:59:42.4170339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T06:59:42.4170408Z return func(*args, **kwargs) 2025-09-07T06:59:42.4170671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T06:59:42.4170804Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T06:59:42.4171100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T06:59:42.4171194Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4171198Z 2025-09-07T06:59:42.4171302Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4171510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4171577Z return mod(**inputs) 2025-09-07T06:59:42.4171839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4171904Z outputs = self.bert( 2025-09-07T06:59:42.4172158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4172239Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4172491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4172569Z layer_outputs = layer_module( 2025-09-07T06:59:42.4172791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4172870Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4173126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4173213Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4173490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4173566Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4173848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4173995Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4174247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T06:59:42.4174338Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4174345Z 2025-09-07T06:59:42.4174451Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4174661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4174746Z return mod(**inputs) 2025-09-07T06:59:42.4175007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4175082Z outputs = self.bert( 2025-09-07T06:59:42.4175335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4175418Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4175674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4175746Z layer_outputs = layer_module( 2025-09-07T06:59:42.4175981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4176060Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4176328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4176413Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4176684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4176758Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4177039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T06:59:42.4177203Z intermediate_output = self.intermediate(attention_output) 2025-09-07T06:59:42.4177453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T06:59:42.4177578Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T06:59:42.4177806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T06:59:42.4177885Z return self.act(input) 2025-09-07T06:59:42.4177888Z 2025-09-07T06:59:42.4178006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4178221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4178299Z return mod(**inputs) 2025-09-07T06:59:42.4178566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-09-07T06:59:42.4178638Z outputs = self.bert( 2025-09-07T06:59:42.4178924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T06:59:42.4179000Z encoder_outputs = self.encoder( 2025-09-07T06:59:42.4179273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T06:59:42.4179349Z layer_outputs = layer_module( 2025-09-07T06:59:42.4179590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T06:59:42.4179674Z return super().__call__(*args, **kwargs) 2025-09-07T06:59:42.4179938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T06:59:42.4180036Z layer_output = apply_chunking_to_forward( 2025-09-07T06:59:42.4180341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T06:59:42.4180431Z return forward_fn(*input_tensors) 2025-09-07T06:59:42.4180736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T06:59:42.4180883Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T06:59:42.4181161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T06:59:42.4181267Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4181271Z 2025-09-07T06:59:42.4181389Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4181604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4181682Z return mod(**inputs) 2025-09-07T06:59:42.4181954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-09-07T06:59:42.4182060Z prediction_scores = self.cls(sequence_output) 2025-09-07T06:59:42.4182333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-09-07T06:59:42.4182455Z prediction_scores = self.predictions(sequence_output) 2025-09-07T06:59:42.4182726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward 2025-09-07T06:59:42.4182827Z hidden_states = self.transform(hidden_states) 2025-09-07T06:59:42.4183090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward 2025-09-07T06:59:42.4183186Z hidden_states = self.dense(hidden_states) 2025-09-07T06:59:42.4183190Z 2025-09-07T06:59:42.4183301Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4183524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4183592Z return mod(**inputs) 2025-09-07T06:59:42.4183898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-09-07T06:59:42.4183997Z prediction_scores = self.cls(sequence_output) 2025-09-07T06:59:42.4184265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-09-07T06:59:42.4184393Z prediction_scores = self.predictions(sequence_output) 2025-09-07T06:59:42.4184661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward 2025-09-07T06:59:42.4184771Z hidden_states = self.decoder(hidden_states) 2025-09-07T06:59:42.4184775Z 2025-09-07T06:59:42.4184887Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T06:59:42.4185108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T06:59:42.4185188Z return mod(**inputs) 2025-09-07T06:59:42.4185467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1328, in forward 2025-09-07T06:59:42.4185774Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T06:59:42.4185781Z 2025-09-07T06:59:53.8217367Z Compilation time (from dynamo_timed): 17.994753038 2025-09-07T06:59:53.8312605Z pass 2025-09-07T06:59:53.8313096Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T06:59:53.8314015Z TIMING: _recursive_pre_grad_passes:0.00761 _recursive_joint_graph_passes:0.38337 _recursive_post_grad_passes:0.0756 async_compile.wait:0.78465 code_gen:10.63397 inductor_compile:11.90096 backend_compile:15.15767 gc:0.00057 entire_frame_compile:17.99475 total_wall_time:17.99475 2025-09-07T06:59:53.8315353Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:12331 | FakeTensor.__torch_dispatch__:4342 | ProxyTorchDispatchMode.__torch_dispatch__:4495 2025-09-07T06:59:53.8315935Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-09-07T06:59:56.7685247Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T06:59:56.7686507Z import pynvml # type: ignore[import] 2025-09-07T06:59:59.5921144Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T06:59:59.5922087Z from pkg_resources import resource_filename 2025-09-07T07:00:00.2725011Z 2025-09-07T07:00:01.3001945Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:00:01.3002340Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:00:01.3013331Z cpu eval BertForQuestionAnswering 2025-09-07T07:00:01.7317468Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:01.9418513Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:02.1400244Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:09.9505932Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9506522Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9507478Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9507792Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9508505Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9508821Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9509063Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9509285Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9509884Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9510115Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9510345Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9510565Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9510833Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9511255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9511696Z return mod(**inputs) 2025-09-07T07:00:09.9512136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9512562Z outputs = self.bert( 2025-09-07T07:00:09.9512967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9513397Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9513835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9514345Z layer_outputs = layer_module( 2025-09-07T07:00:09.9514738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9515136Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9515568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9516004Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9516424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9516881Z return func(*args, **kwargs) 2025-09-07T07:00:09.9517355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9517769Z self_outputs = self.self( 2025-09-07T07:00:09.9518165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9518580Z return func(*args, **kwargs) 2025-09-07T07:00:09.9518978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9519554Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9520086Z 2025-09-07T07:00:09.9520214Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9520611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9520974Z return mod(**inputs) 2025-09-07T07:00:09.9521379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9521796Z outputs = self.bert( 2025-09-07T07:00:09.9522193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9522610Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9523211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9523631Z layer_outputs = layer_module( 2025-09-07T07:00:09.9524041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9524438Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9524856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9525288Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9525708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9526158Z return func(*args, **kwargs) 2025-09-07T07:00:09.9526573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9526969Z self_outputs = self.self( 2025-09-07T07:00:09.9527336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9527718Z return func(*args, **kwargs) 2025-09-07T07:00:09.9528101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9528494Z self.key(current_states) 2025-09-07T07:00:09.9528614Z 2025-09-07T07:00:09.9528731Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9529102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9529443Z return mod(**inputs) 2025-09-07T07:00:09.9529812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9530200Z outputs = self.bert( 2025-09-07T07:00:09.9530560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9530953Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9531339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9531729Z layer_outputs = layer_module( 2025-09-07T07:00:09.9532085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9532470Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9533613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9534017Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9534429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9534877Z return func(*args, **kwargs) 2025-09-07T07:00:09.9535258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9535680Z self_outputs = self.self( 2025-09-07T07:00:09.9536057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9536472Z return func(*args, **kwargs) 2025-09-07T07:00:09.9536867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9537286Z self.value(current_states) 2025-09-07T07:00:09.9537426Z 2025-09-07T07:00:09.9537517Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9537788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9538181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9538537Z return mod(**inputs) 2025-09-07T07:00:09.9538926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9539338Z outputs = self.bert( 2025-09-07T07:00:09.9539727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9540135Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9540547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9540960Z layer_outputs = layer_module( 2025-09-07T07:00:09.9541338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9541779Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9542191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9542621Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9543047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9543469Z return func(*args, **kwargs) 2025-09-07T07:00:09.9543870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9544294Z self_outputs = self.self( 2025-09-07T07:00:09.9544698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9545113Z return func(*args, **kwargs) 2025-09-07T07:00:09.9545535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9546223Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9546443Z 2025-09-07T07:00:09.9546563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9546969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9547335Z return mod(**inputs) 2025-09-07T07:00:09.9547740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9548165Z outputs = self.bert( 2025-09-07T07:00:09.9548568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9549076Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9549529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9549959Z layer_outputs = layer_module( 2025-09-07T07:00:09.9550352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9550755Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9551189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9551661Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9552083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9552500Z return func(*args, **kwargs) 2025-09-07T07:00:09.9552909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9553396Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9553878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9554319Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9554483Z 2025-09-07T07:00:09.9554602Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9555008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9555382Z return mod(**inputs) 2025-09-07T07:00:09.9555779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9556203Z outputs = self.bert( 2025-09-07T07:00:09.9556606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9557032Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9557438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9557900Z layer_outputs = layer_module( 2025-09-07T07:00:09.9558278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9558667Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9559076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9559493Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9559935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9560368Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9560814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9561322Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9561782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9562217Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9562381Z 2025-09-07T07:00:09.9562494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9562883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9563240Z return mod(**inputs) 2025-09-07T07:00:09.9563622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9564033Z outputs = self.bert( 2025-09-07T07:00:09.9564422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9564889Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9565293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9565708Z layer_outputs = layer_module( 2025-09-07T07:00:09.9566087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9566483Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9566898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9567345Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9567782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9568214Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9568620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9569085Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9569510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9569942Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9570324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9570675Z return self.act(input) 2025-09-07T07:00:09.9570791Z 2025-09-07T07:00:09.9570897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9571281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9571606Z return mod(**inputs) 2025-09-07T07:00:09.9571972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9572351Z outputs = self.bert( 2025-09-07T07:00:09.9572771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9573198Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9573610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9574027Z layer_outputs = layer_module( 2025-09-07T07:00:09.9574404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9574781Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9575178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9575569Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9575970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9576358Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9576763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9577229Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9577661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9578060Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9578198Z 2025-09-07T07:00:09.9578302Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9578663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9578987Z return mod(**inputs) 2025-09-07T07:00:09.9579383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9579781Z outputs = self.bert( 2025-09-07T07:00:09.9580151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9580555Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9580950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9581349Z layer_outputs = layer_module( 2025-09-07T07:00:09.9581737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9582141Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9582557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9582979Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9583398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9583810Z return func(*args, **kwargs) 2025-09-07T07:00:09.9584223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9584645Z self_outputs = self.self( 2025-09-07T07:00:09.9585043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9585453Z return func(*args, **kwargs) 2025-09-07T07:00:09.9585992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9586599Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9586909Z 2025-09-07T07:00:09.9587041Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9587449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9587848Z return mod(**inputs) 2025-09-07T07:00:09.9588243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9588659Z outputs = self.bert( 2025-09-07T07:00:09.9589047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9589481Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9589893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9590322Z layer_outputs = layer_module( 2025-09-07T07:00:09.9590699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9591094Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9591507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9591938Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9592354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9592761Z return func(*args, **kwargs) 2025-09-07T07:00:09.9593163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9593568Z self_outputs = self.self( 2025-09-07T07:00:09.9593960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9594397Z return func(*args, **kwargs) 2025-09-07T07:00:09.9594793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9595227Z self.key(current_states) 2025-09-07T07:00:09.9595353Z 2025-09-07T07:00:09.9595469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9595861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9596213Z return mod(**inputs) 2025-09-07T07:00:09.9596606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9597006Z outputs = self.bert( 2025-09-07T07:00:09.9597425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9597816Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9598199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9598591Z layer_outputs = layer_module( 2025-09-07T07:00:09.9598941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9599318Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9599711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9600111Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9600498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9600881Z return func(*args, **kwargs) 2025-09-07T07:00:09.9601254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9601643Z self_outputs = self.self( 2025-09-07T07:00:09.9602013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9602451Z return func(*args, **kwargs) 2025-09-07T07:00:09.9602913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9603342Z self.value(current_states) 2025-09-07T07:00:09.9603481Z 2025-09-07T07:00:09.9603584Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9603856Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9604265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9604616Z return mod(**inputs) 2025-09-07T07:00:09.9604992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9605389Z outputs = self.bert( 2025-09-07T07:00:09.9605747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9606148Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9606537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9606931Z layer_outputs = layer_module( 2025-09-07T07:00:09.9607276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9607649Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9608042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9608443Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9608841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9609223Z return func(*args, **kwargs) 2025-09-07T07:00:09.9609603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9610008Z self_outputs = self.self( 2025-09-07T07:00:09.9610383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9610766Z return func(*args, **kwargs) 2025-09-07T07:00:09.9611138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9611592Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9611788Z 2025-09-07T07:00:09.9611912Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9612285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9612612Z return mod(**inputs) 2025-09-07T07:00:09.9612980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9613373Z outputs = self.bert( 2025-09-07T07:00:09.9613742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9614136Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9614516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9614905Z layer_outputs = layer_module( 2025-09-07T07:00:09.9615262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9615647Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9616029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9616409Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9616789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9617160Z return func(*args, **kwargs) 2025-09-07T07:00:09.9617581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9618015Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9618451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9618844Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9618985Z 2025-09-07T07:00:09.9619100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9619460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9619936Z return mod(**inputs) 2025-09-07T07:00:09.9620301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9620689Z outputs = self.bert( 2025-09-07T07:00:09.9621061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9621459Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9621841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9622230Z layer_outputs = layer_module( 2025-09-07T07:00:09.9622602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9623009Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9623437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9623877Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9624326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9624801Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9625230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9625770Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9626276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9626728Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9626926Z 2025-09-07T07:00:09.9627047Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9627445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9627809Z return mod(**inputs) 2025-09-07T07:00:09.9628207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9628606Z outputs = self.bert( 2025-09-07T07:00:09.9628976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9629368Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9629755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9630150Z layer_outputs = layer_module( 2025-09-07T07:00:09.9630502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9630865Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9631250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9631645Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9632069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9632484Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9632953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9633426Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9633865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9634300Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9634696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9635043Z return self.act(input) 2025-09-07T07:00:09.9635168Z 2025-09-07T07:00:09.9635277Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9635648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9635985Z return mod(**inputs) 2025-09-07T07:00:09.9636358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9636739Z outputs = self.bert( 2025-09-07T07:00:09.9637106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9637503Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9637887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9638270Z layer_outputs = layer_module( 2025-09-07T07:00:09.9638628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9639001Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9639424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9639827Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9640234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9640644Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9641066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9641574Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9642033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9642434Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9642583Z 2025-09-07T07:00:09.9642691Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9643066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9643405Z return mod(**inputs) 2025-09-07T07:00:09.9643770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9644160Z outputs = self.bert( 2025-09-07T07:00:09.9644576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9644971Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9645356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9645734Z layer_outputs = layer_module( 2025-09-07T07:00:09.9646089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9646459Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9646847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9647283Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9647660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9648037Z return func(*args, **kwargs) 2025-09-07T07:00:09.9648404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9648787Z self_outputs = self.self( 2025-09-07T07:00:09.9649141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9649513Z return func(*args, **kwargs) 2025-09-07T07:00:09.9649876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9650391Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9650655Z 2025-09-07T07:00:09.9650768Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9651122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9651452Z return mod(**inputs) 2025-09-07T07:00:09.9651813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9652194Z outputs = self.bert( 2025-09-07T07:00:09.9652551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9652929Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9653312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9653760Z layer_outputs = layer_module( 2025-09-07T07:00:09.9654106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9654462Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9654847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9655239Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9655625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9656017Z return func(*args, **kwargs) 2025-09-07T07:00:09.9656381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9656764Z self_outputs = self.self( 2025-09-07T07:00:09.9657125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9657497Z return func(*args, **kwargs) 2025-09-07T07:00:09.9657856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9658234Z self.key(current_states) 2025-09-07T07:00:09.9658356Z 2025-09-07T07:00:09.9658459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9658823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9659148Z return mod(**inputs) 2025-09-07T07:00:09.9659501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9659877Z outputs = self.bert( 2025-09-07T07:00:09.9660236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9660624Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9661034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9661410Z layer_outputs = layer_module( 2025-09-07T07:00:09.9661786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9662184Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9662603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9663020Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9663435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9663841Z return func(*args, **kwargs) 2025-09-07T07:00:09.9664243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9664653Z self_outputs = self.self( 2025-09-07T07:00:09.9665043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9665455Z return func(*args, **kwargs) 2025-09-07T07:00:09.9665953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9666386Z self.value(current_states) 2025-09-07T07:00:09.9666522Z 2025-09-07T07:00:09.9666612Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9666886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9667296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9667653Z return mod(**inputs) 2025-09-07T07:00:09.9668045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9668480Z outputs = self.bert( 2025-09-07T07:00:09.9668876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9669297Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9669708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9670115Z layer_outputs = layer_module( 2025-09-07T07:00:09.9670493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9670915Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9671335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9671757Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9672162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9672571Z return func(*args, **kwargs) 2025-09-07T07:00:09.9672944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9673307Z self_outputs = self.self( 2025-09-07T07:00:09.9673656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9674006Z return func(*args, **kwargs) 2025-09-07T07:00:09.9674361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9674790Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9674969Z 2025-09-07T07:00:09.9675077Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9675420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9675736Z return mod(**inputs) 2025-09-07T07:00:09.9676140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9676512Z outputs = self.bert( 2025-09-07T07:00:09.9676857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9677223Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9677594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9677974Z layer_outputs = layer_module( 2025-09-07T07:00:09.9678317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9678662Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9679036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9679425Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9679812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9680192Z return func(*args, **kwargs) 2025-09-07T07:00:09.9680561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9681019Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9681471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9681872Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9682019Z 2025-09-07T07:00:09.9682128Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9682479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9682830Z return mod(**inputs) 2025-09-07T07:00:09.9683198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9683578Z outputs = self.bert( 2025-09-07T07:00:09.9683941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9684357Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9684739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9685149Z layer_outputs = layer_module( 2025-09-07T07:00:09.9685506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9685870Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9686276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9686672Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9687080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9687478Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9687897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9688365Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9688810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9689214Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9689355Z 2025-09-07T07:00:09.9689473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9689863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9690213Z return mod(**inputs) 2025-09-07T07:00:09.9690645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9691058Z outputs = self.bert( 2025-09-07T07:00:09.9691440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9691838Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9692223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9692622Z layer_outputs = layer_module( 2025-09-07T07:00:09.9692994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9693380Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9693771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9694178Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9694589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9694988Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9695416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9695884Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9696329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9696793Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9697203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9697605Z return self.act(input) 2025-09-07T07:00:09.9697734Z 2025-09-07T07:00:09.9697849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9698238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9698589Z return mod(**inputs) 2025-09-07T07:00:09.9698970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9699359Z outputs = self.bert( 2025-09-07T07:00:09.9699766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9700186Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9700587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9701011Z layer_outputs = layer_module( 2025-09-07T07:00:09.9701406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9701796Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9702218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9702655Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9703088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9703522Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9703969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9704497Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9704965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9705397Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9705551Z 2025-09-07T07:00:09.9705793Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9706218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9706593Z return mod(**inputs) 2025-09-07T07:00:09.9707002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9707421Z outputs = self.bert( 2025-09-07T07:00:09.9707803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9708253Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9708670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9709101Z layer_outputs = layer_module( 2025-09-07T07:00:09.9709498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9709915Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9710336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9710784Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9711229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9711654Z return func(*args, **kwargs) 2025-09-07T07:00:09.9712130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9712558Z self_outputs = self.self( 2025-09-07T07:00:09.9712966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9713416Z return func(*args, **kwargs) 2025-09-07T07:00:09.9713837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9714410Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9714703Z 2025-09-07T07:00:09.9714819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9715222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9715663Z return mod(**inputs) 2025-09-07T07:00:09.9716067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9716483Z outputs = self.bert( 2025-09-07T07:00:09.9716882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9717316Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9717724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9718138Z layer_outputs = layer_module( 2025-09-07T07:00:09.9718512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9718917Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9719337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9719906Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9720329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9720729Z return func(*args, **kwargs) 2025-09-07T07:00:09.9721138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9721554Z self_outputs = self.self( 2025-09-07T07:00:09.9722038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9722440Z return func(*args, **kwargs) 2025-09-07T07:00:09.9722843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9723254Z self.key(current_states) 2025-09-07T07:00:09.9723383Z 2025-09-07T07:00:09.9723506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9723904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9724273Z return mod(**inputs) 2025-09-07T07:00:09.9724667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9725087Z outputs = self.bert( 2025-09-07T07:00:09.9725480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9725902Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9726296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9726694Z layer_outputs = layer_module( 2025-09-07T07:00:09.9727045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9727410Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9727785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9728178Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9728563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9728967Z return func(*args, **kwargs) 2025-09-07T07:00:09.9729338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9729705Z self_outputs = self.self( 2025-09-07T07:00:09.9730060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9730434Z return func(*args, **kwargs) 2025-09-07T07:00:09.9730807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9731233Z self.value(current_states) 2025-09-07T07:00:09.9731361Z 2025-09-07T07:00:09.9731444Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9731691Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9732065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9732402Z return mod(**inputs) 2025-09-07T07:00:09.9732773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9733154Z outputs = self.bert( 2025-09-07T07:00:09.9733509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9733901Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9734284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9734687Z layer_outputs = layer_module( 2025-09-07T07:00:09.9735034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9735398Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9735779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9736165Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9736587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9736971Z return func(*args, **kwargs) 2025-09-07T07:00:09.9737352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9737753Z self_outputs = self.self( 2025-09-07T07:00:09.9738124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9738502Z return func(*args, **kwargs) 2025-09-07T07:00:09.9738881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9739327Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9739518Z 2025-09-07T07:00:09.9739621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9739998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9740338Z return mod(**inputs) 2025-09-07T07:00:09.9740716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9741116Z outputs = self.bert( 2025-09-07T07:00:09.9741484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9741889Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9742282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9742716Z layer_outputs = layer_module( 2025-09-07T07:00:09.9743093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9743501Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9743916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9744336Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9744733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9745104Z return func(*args, **kwargs) 2025-09-07T07:00:09.9745510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9746034Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9746508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9746941Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9747084Z 2025-09-07T07:00:09.9747190Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9747566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9747900Z return mod(**inputs) 2025-09-07T07:00:09.9748265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9748648Z outputs = self.bert( 2025-09-07T07:00:09.9749017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9749417Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9749801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9750189Z layer_outputs = layer_module( 2025-09-07T07:00:09.9750539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9750909Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9751338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9751748Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9752155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9752565Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9752986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9753455Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9753897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9754294Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9754445Z 2025-09-07T07:00:09.9754556Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9754931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9755267Z return mod(**inputs) 2025-09-07T07:00:09.9755643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9756025Z outputs = self.bert( 2025-09-07T07:00:09.9756394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9756789Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9757173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9757555Z layer_outputs = layer_module( 2025-09-07T07:00:09.9757932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9758318Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9758695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9759080Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9759467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9759877Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9760277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9760725Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9761143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9761557Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9761941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9762292Z return self.act(input) 2025-09-07T07:00:09.9762403Z 2025-09-07T07:00:09.9762513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9762908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9763328Z return mod(**inputs) 2025-09-07T07:00:09.9763694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9764084Z outputs = self.bert( 2025-09-07T07:00:09.9764455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9764847Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9765265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9765651Z layer_outputs = layer_module( 2025-09-07T07:00:09.9765996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9766369Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9766743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9767133Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9767531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9767919Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9768308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9768769Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9769198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9769585Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9769718Z 2025-09-07T07:00:09.9769827Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9770171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9770499Z return mod(**inputs) 2025-09-07T07:00:09.9770848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9771218Z outputs = self.bert( 2025-09-07T07:00:09.9771566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9771959Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9772333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9772710Z layer_outputs = layer_module( 2025-09-07T07:00:09.9773059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9773419Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9773799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9774204Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9774587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9774967Z return func(*args, **kwargs) 2025-09-07T07:00:09.9775340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9775719Z self_outputs = self.self( 2025-09-07T07:00:09.9776084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9776458Z return func(*args, **kwargs) 2025-09-07T07:00:09.9776822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9777339Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9777610Z 2025-09-07T07:00:09.9777713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9778071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9778398Z return mod(**inputs) 2025-09-07T07:00:09.9778753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9779130Z outputs = self.bert( 2025-09-07T07:00:09.9779523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9779916Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9780298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9780677Z layer_outputs = layer_module( 2025-09-07T07:00:09.9781034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9781406Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9781800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9782198Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9782582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9782964Z return func(*args, **kwargs) 2025-09-07T07:00:09.9783341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9783729Z self_outputs = self.self( 2025-09-07T07:00:09.9784089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9784471Z return func(*args, **kwargs) 2025-09-07T07:00:09.9784844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9785233Z self.key(current_states) 2025-09-07T07:00:09.9785351Z 2025-09-07T07:00:09.9785465Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9785904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9786288Z return mod(**inputs) 2025-09-07T07:00:09.9786684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9787099Z outputs = self.bert( 2025-09-07T07:00:09.9787462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9787861Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9788249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9788677Z layer_outputs = layer_module( 2025-09-07T07:00:09.9789026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9789382Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9789774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9790173Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9790562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9790933Z return func(*args, **kwargs) 2025-09-07T07:00:09.9791307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9791692Z self_outputs = self.self( 2025-09-07T07:00:09.9792062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9792442Z return func(*args, **kwargs) 2025-09-07T07:00:09.9792807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9793192Z self.value(current_states) 2025-09-07T07:00:09.9793320Z 2025-09-07T07:00:09.9793407Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9793652Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9794059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9794391Z return mod(**inputs) 2025-09-07T07:00:09.9794761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9795149Z outputs = self.bert( 2025-09-07T07:00:09.9795517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9795905Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9796290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9796678Z layer_outputs = layer_module( 2025-09-07T07:00:09.9797038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9797406Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9797794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9798197Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9798592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9798971Z return func(*args, **kwargs) 2025-09-07T07:00:09.9799342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9799731Z self_outputs = self.self( 2025-09-07T07:00:09.9800099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9800500Z return func(*args, **kwargs) 2025-09-07T07:00:09.9800873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9801315Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9801510Z 2025-09-07T07:00:09.9801618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9801985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9802316Z return mod(**inputs) 2025-09-07T07:00:09.9802711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9803092Z outputs = self.bert( 2025-09-07T07:00:09.9803458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9803854Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9804302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9804696Z layer_outputs = layer_module( 2025-09-07T07:00:09.9805053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9805424Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9805818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9806249Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9806680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9807086Z return func(*args, **kwargs) 2025-09-07T07:00:09.9807496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9807987Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9808514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9808958Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9809121Z 2025-09-07T07:00:09.9809237Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9809639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9810010Z return mod(**inputs) 2025-09-07T07:00:09.9810393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9810802Z outputs = self.bert( 2025-09-07T07:00:09.9811187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9811617Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9812039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9812468Z layer_outputs = layer_module( 2025-09-07T07:00:09.9812858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9813262Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9813722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9814153Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9814591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9815021Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9815475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9816010Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9816486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9816932Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9817090Z 2025-09-07T07:00:09.9817206Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9817614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9817994Z return mod(**inputs) 2025-09-07T07:00:09.9818384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9818807Z outputs = self.bert( 2025-09-07T07:00:09.9819207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9819795Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9823104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9824070Z layer_outputs = layer_module( 2025-09-07T07:00:09.9825031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9825829Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9826493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9827161Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9827818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9828777Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9829562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9830364Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9831292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9831988Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9832619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9833200Z return self.act(input) 2025-09-07T07:00:09.9833376Z 2025-09-07T07:00:09.9833537Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9834182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9834790Z return mod(**inputs) 2025-09-07T07:00:09.9835378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9836013Z outputs = self.bert( 2025-09-07T07:00:09.9836635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9837281Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9837907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9838541Z layer_outputs = layer_module( 2025-09-07T07:00:09.9839074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9839649Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9840254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9840892Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9841548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9842182Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9842790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9843500Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9844166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9844810Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9845007Z 2025-09-07T07:00:09.9845146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9845670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9846147Z return mod(**inputs) 2025-09-07T07:00:09.9846659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9847227Z outputs = self.bert( 2025-09-07T07:00:09.9847735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9848314Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9848871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9849436Z layer_outputs = layer_module( 2025-09-07T07:00:09.9849936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9853372Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9853991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9854596Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9855185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9855755Z return func(*args, **kwargs) 2025-09-07T07:00:09.9856842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9857467Z self_outputs = self.self( 2025-09-07T07:00:09.9858019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9858592Z return func(*args, **kwargs) 2025-09-07T07:00:09.9859168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9859975Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9860400Z 2025-09-07T07:00:09.9860554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9861135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9861636Z return mod(**inputs) 2025-09-07T07:00:09.9862205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9862830Z outputs = self.bert( 2025-09-07T07:00:09.9863429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9864062Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9864659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9865252Z layer_outputs = layer_module( 2025-09-07T07:00:09.9865776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9866243Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9866716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9867135Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9867535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9867923Z return func(*args, **kwargs) 2025-09-07T07:00:09.9868304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9868705Z self_outputs = self.self( 2025-09-07T07:00:09.9869082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9869465Z return func(*args, **kwargs) 2025-09-07T07:00:09.9869849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9870267Z self.key(current_states) 2025-09-07T07:00:09.9870394Z 2025-09-07T07:00:09.9870511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9870916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9871269Z return mod(**inputs) 2025-09-07T07:00:09.9871666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9872065Z outputs = self.bert( 2025-09-07T07:00:09.9872441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9872868Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9873279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9873704Z layer_outputs = layer_module( 2025-09-07T07:00:09.9874075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9874469Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9874937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9875377Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9875798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9876216Z return func(*args, **kwargs) 2025-09-07T07:00:09.9876629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9877056Z self_outputs = self.self( 2025-09-07T07:00:09.9877461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9877874Z return func(*args, **kwargs) 2025-09-07T07:00:09.9878284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9878801Z self.value(current_states) 2025-09-07T07:00:09.9878986Z 2025-09-07T07:00:09.9879087Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9879348Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9879756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9880128Z return mod(**inputs) 2025-09-07T07:00:09.9880527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9880962Z outputs = self.bert( 2025-09-07T07:00:09.9881354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9881788Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9882238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9882680Z layer_outputs = layer_module( 2025-09-07T07:00:09.9883043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9883433Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9883841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9884264Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9884652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9885023Z return func(*args, **kwargs) 2025-09-07T07:00:09.9885398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9885783Z self_outputs = self.self( 2025-09-07T07:00:09.9886155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9886539Z return func(*args, **kwargs) 2025-09-07T07:00:09.9886914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9887378Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9887586Z 2025-09-07T07:00:09.9887693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9888068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9888402Z return mod(**inputs) 2025-09-07T07:00:09.9888781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9889180Z outputs = self.bert( 2025-09-07T07:00:09.9889700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9890322Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9890895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9891450Z layer_outputs = layer_module( 2025-09-07T07:00:09.9892035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9892491Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9892888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9893283Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9893678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9894067Z return func(*args, **kwargs) 2025-09-07T07:00:09.9894447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9894893Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9895342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9895745Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9895889Z 2025-09-07T07:00:09.9896009Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9896382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9896722Z return mod(**inputs) 2025-09-07T07:00:09.9897116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9897540Z outputs = self.bert( 2025-09-07T07:00:09.9897910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9898310Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9898691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9899083Z layer_outputs = layer_module( 2025-09-07T07:00:09.9899444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9899837Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9900237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9900666Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9901267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9901700Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9902131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9902624Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9903096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9903523Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9903679Z 2025-09-07T07:00:09.9903803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9904196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9904549Z return mod(**inputs) 2025-09-07T07:00:09.9904941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9905351Z outputs = self.bert( 2025-09-07T07:00:09.9905930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9906363Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9906774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9907185Z layer_outputs = layer_module( 2025-09-07T07:00:09.9907567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9907942Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9908329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9908739Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9909177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9909606Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9910049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9910543Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9911006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9911458Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9911851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9912196Z return self.act(input) 2025-09-07T07:00:09.9912319Z 2025-09-07T07:00:09.9912424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9912796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9913152Z return mod(**inputs) 2025-09-07T07:00:09.9913522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9913928Z outputs = self.bert( 2025-09-07T07:00:09.9914319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9914740Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9915158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9915601Z layer_outputs = layer_module( 2025-09-07T07:00:09.9915996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9916403Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9916834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9917270Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9917688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9918131Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9918589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9919117Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9919856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9920448Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9920618Z 2025-09-07T07:00:09.9920735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9921149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9921518Z return mod(**inputs) 2025-09-07T07:00:09.9922061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9922487Z outputs = self.bert( 2025-09-07T07:00:09.9922890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9923326Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9923757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9924175Z layer_outputs = layer_module( 2025-09-07T07:00:09.9924568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9924977Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9925405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9925797Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9926173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9926547Z return func(*args, **kwargs) 2025-09-07T07:00:09.9926917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9927297Z self_outputs = self.self( 2025-09-07T07:00:09.9927656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9928032Z return func(*args, **kwargs) 2025-09-07T07:00:09.9928400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9928946Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9929207Z 2025-09-07T07:00:09.9929323Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9929682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9930013Z return mod(**inputs) 2025-09-07T07:00:09.9930375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9930781Z outputs = self.bert( 2025-09-07T07:00:09.9931135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9931509Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9931883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9932263Z layer_outputs = layer_module( 2025-09-07T07:00:09.9932606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9932962Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9933345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9933738Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9934122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9934195Z return func(*args, **kwargs) 2025-09-07T07:00:09.9934446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9934517Z self_outputs = self.self( 2025-09-07T07:00:09.9934755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9934832Z return func(*args, **kwargs) 2025-09-07T07:00:09.9935107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9935187Z self.key(current_states) 2025-09-07T07:00:09.9935190Z 2025-09-07T07:00:09.9935296Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9935502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9935567Z return mod(**inputs) 2025-09-07T07:00:09.9935820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9935895Z outputs = self.bert( 2025-09-07T07:00:09.9936141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9936222Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9936466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9936541Z layer_outputs = layer_module( 2025-09-07T07:00:09.9936769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9936848Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9937096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9937180Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9937418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9937495Z return func(*args, **kwargs) 2025-09-07T07:00:09.9937742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9937835Z self_outputs = self.self( 2025-09-07T07:00:09.9938077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9938146Z return func(*args, **kwargs) 2025-09-07T07:00:09.9938600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9938671Z self.value(current_states) 2025-09-07T07:00:09.9938675Z 2025-09-07T07:00:09.9938778Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9938900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9939105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9939169Z return mod(**inputs) 2025-09-07T07:00:09.9939463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9939540Z outputs = self.bert( 2025-09-07T07:00:09.9939791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9939877Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9940125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9940196Z layer_outputs = layer_module( 2025-09-07T07:00:09.9940426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9940506Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9940762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9940842Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9941085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9941155Z return func(*args, **kwargs) 2025-09-07T07:00:09.9941436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9941515Z self_outputs = self.self( 2025-09-07T07:00:09.9941760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9941847Z return func(*args, **kwargs) 2025-09-07T07:00:09.9942101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9942237Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9942241Z 2025-09-07T07:00:09.9942352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9942554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9942627Z return mod(**inputs) 2025-09-07T07:00:09.9942881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9942948Z outputs = self.bert( 2025-09-07T07:00:09.9943211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9943286Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9943545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9943619Z layer_outputs = layer_module( 2025-09-07T07:00:09.9943852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9943932Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9944192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9944306Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9944576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9944657Z return func(*args, **kwargs) 2025-09-07T07:00:09.9944933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9945064Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9945326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9945430Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9945434Z 2025-09-07T07:00:09.9945545Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9945881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9945959Z return mod(**inputs) 2025-09-07T07:00:09.9946244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9946315Z outputs = self.bert( 2025-09-07T07:00:09.9946593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9946671Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9946955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9947034Z layer_outputs = layer_module( 2025-09-07T07:00:09.9947272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9947377Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9947617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9947707Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9947998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9948078Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9948356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9948474Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9948721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9948804Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9948807Z 2025-09-07T07:00:09.9948912Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9949107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9949172Z return mod(**inputs) 2025-09-07T07:00:09.9949424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9949490Z outputs = self.bert( 2025-09-07T07:00:09.9949745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9949818Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9950061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9950141Z layer_outputs = layer_module( 2025-09-07T07:00:09.9950359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9950444Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9950690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9950827Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9951097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9951172Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9951462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9951578Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9951855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9951967Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9952175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9952255Z return self.act(input) 2025-09-07T07:00:09.9952258Z 2025-09-07T07:00:09.9952358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9952567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9952635Z return mod(**inputs) 2025-09-07T07:00:09.9952896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9952972Z outputs = self.bert( 2025-09-07T07:00:09.9953237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9953325Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9953593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9953677Z layer_outputs = layer_module( 2025-09-07T07:00:09.9953919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9954039Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9954321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9954405Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9954675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9954754Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9955032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9955173Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9955420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9955512Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9955515Z 2025-09-07T07:00:09.9955632Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9955833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9955898Z return mod(**inputs) 2025-09-07T07:00:09.9956139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9956210Z outputs = self.bert( 2025-09-07T07:00:09.9956455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9956533Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9956782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9956876Z layer_outputs = layer_module( 2025-09-07T07:00:09.9957121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9957209Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9957478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9957567Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9957830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9957934Z return func(*args, **kwargs) 2025-09-07T07:00:09.9958200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9958281Z self_outputs = self.self( 2025-09-07T07:00:09.9958539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9958617Z return func(*args, **kwargs) 2025-09-07T07:00:09.9958897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9959129Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9959133Z 2025-09-07T07:00:09.9959253Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9959472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9959553Z return mod(**inputs) 2025-09-07T07:00:09.9959830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9959901Z outputs = self.bert( 2025-09-07T07:00:09.9960188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9960271Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9960584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9960666Z layer_outputs = layer_module( 2025-09-07T07:00:09.9960909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9961003Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9961275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9961374Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9961638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9961722Z return func(*args, **kwargs) 2025-09-07T07:00:09.9961993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9962070Z self_outputs = self.self( 2025-09-07T07:00:09.9962349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9962424Z return func(*args, **kwargs) 2025-09-07T07:00:09.9962704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9962780Z self.key(current_states) 2025-09-07T07:00:09.9962784Z 2025-09-07T07:00:09.9962897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9963128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9963199Z return mod(**inputs) 2025-09-07T07:00:09.9963483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9963574Z outputs = self.bert( 2025-09-07T07:00:09.9963849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9963940Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9964213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9964299Z layer_outputs = layer_module( 2025-09-07T07:00:09.9964545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9964654Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9964896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9964975Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9965219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9965289Z return func(*args, **kwargs) 2025-09-07T07:00:09.9965542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9965611Z self_outputs = self.self( 2025-09-07T07:00:09.9965849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9965924Z return func(*args, **kwargs) 2025-09-07T07:00:09.9966165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9966249Z self.value(current_states) 2025-09-07T07:00:09.9966252Z 2025-09-07T07:00:09.9966333Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9966435Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9966641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9966709Z return mod(**inputs) 2025-09-07T07:00:09.9966993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9967061Z outputs = self.bert( 2025-09-07T07:00:09.9967309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9967390Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9967633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9967714Z layer_outputs = layer_module( 2025-09-07T07:00:09.9967932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9968018Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9968267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9968350Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9968599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9968667Z return func(*args, **kwargs) 2025-09-07T07:00:09.9968931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9969002Z self_outputs = self.self( 2025-09-07T07:00:09.9969240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9969321Z return func(*args, **kwargs) 2025-09-07T07:00:09.9969563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9969704Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9969726Z 2025-09-07T07:00:09.9969829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9970038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9970105Z return mod(**inputs) 2025-09-07T07:00:09.9970363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9970435Z outputs = self.bert( 2025-09-07T07:00:09.9970696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9970794Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9971040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9971112Z layer_outputs = layer_module( 2025-09-07T07:00:09.9971337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9971416Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9971669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9971750Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9971990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9972066Z return func(*args, **kwargs) 2025-09-07T07:00:09.9972309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9972451Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9972697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9972786Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9972790Z 2025-09-07T07:00:09.9972893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9973133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9973209Z return mod(**inputs) 2025-09-07T07:00:09.9973463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9973535Z outputs = self.bert( 2025-09-07T07:00:09.9973787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9973864Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9974124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9974197Z layer_outputs = layer_module( 2025-09-07T07:00:09.9974428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9974510Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9974772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9974864Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9975122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9975207Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9975484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9975610Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9975855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:09.9975964Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9975968Z 2025-09-07T07:00:09.9976075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9976273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9976346Z return mod(**inputs) 2025-09-07T07:00:09.9976593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9976658Z outputs = self.bert( 2025-09-07T07:00:09.9976906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9976997Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9977247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9977318Z layer_outputs = layer_module( 2025-09-07T07:00:09.9977552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9977629Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9977873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9977962Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9978219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9978303Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9978577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:09.9978695Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:09.9978949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:09.9979064Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:09.9979312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:09.9979386Z return self.act(input) 2025-09-07T07:00:09.9979389Z 2025-09-07T07:00:09.9979498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9979703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9979770Z return mod(**inputs) 2025-09-07T07:00:09.9980034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9980100Z outputs = self.bert( 2025-09-07T07:00:09.9980362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9980436Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9980691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9980775Z layer_outputs = layer_module( 2025-09-07T07:00:09.9981004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9981094Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9981344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:09.9981431Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:09.9981705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:09.9981783Z return forward_fn(*input_tensors) 2025-09-07T07:00:09.9982073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:09.9982227Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:09.9982487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:09.9982570Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9982574Z 2025-09-07T07:00:09.9982679Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9982886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9982971Z return mod(**inputs) 2025-09-07T07:00:09.9983232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9983299Z outputs = self.bert( 2025-09-07T07:00:09.9983553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9983636Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9983898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9983981Z layer_outputs = layer_module( 2025-09-07T07:00:09.9984221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9984306Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9984586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9984669Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9984924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9984995Z return func(*args, **kwargs) 2025-09-07T07:00:09.9985253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9985326Z self_outputs = self.self( 2025-09-07T07:00:09.9985710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9985816Z return func(*args, **kwargs) 2025-09-07T07:00:09.9986103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:09.9986335Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:09.9986344Z 2025-09-07T07:00:09.9986455Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9986670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9986748Z return mod(**inputs) 2025-09-07T07:00:09.9987021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9987103Z outputs = self.bert( 2025-09-07T07:00:09.9987387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9987471Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9987723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9987797Z layer_outputs = layer_module( 2025-09-07T07:00:09.9988030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9988113Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9988373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9988455Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9988719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9988798Z return func(*args, **kwargs) 2025-09-07T07:00:09.9989053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9989135Z self_outputs = self.self( 2025-09-07T07:00:09.9989381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9989450Z return func(*args, **kwargs) 2025-09-07T07:00:09.9989725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:09.9989796Z self.key(current_states) 2025-09-07T07:00:09.9989800Z 2025-09-07T07:00:09.9989911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9990116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9990190Z return mod(**inputs) 2025-09-07T07:00:09.9990447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9990515Z outputs = self.bert( 2025-09-07T07:00:09.9990775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9990849Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9991101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9991173Z layer_outputs = layer_module( 2025-09-07T07:00:09.9991396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9991484Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9991731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9991821Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9992096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9992169Z return func(*args, **kwargs) 2025-09-07T07:00:09.9992429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9992500Z self_outputs = self.self( 2025-09-07T07:00:09.9992751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9992823Z return func(*args, **kwargs) 2025-09-07T07:00:09.9993074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:09.9993149Z self.value(current_states) 2025-09-07T07:00:09.9993154Z 2025-09-07T07:00:09.9993239Z cudagraph partition due to non gpu ops 2025-09-07T07:00:09.9993353Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9993557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9993633Z return mod(**inputs) 2025-09-07T07:00:09.9993886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9993953Z outputs = self.bert( 2025-09-07T07:00:09.9994217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9994292Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9994555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9994627Z layer_outputs = layer_module( 2025-09-07T07:00:09.9994849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9994959Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9995213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9995305Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9995550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9995620Z return func(*args, **kwargs) 2025-09-07T07:00:09.9995895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:09.9995966Z self_outputs = self.self( 2025-09-07T07:00:09.9996227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9996295Z return func(*args, **kwargs) 2025-09-07T07:00:09.9996546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:09.9996683Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:09.9996687Z 2025-09-07T07:00:09.9996789Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:09.9996992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:09.9997057Z return mod(**inputs) 2025-09-07T07:00:09.9997312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:09.9997380Z outputs = self.bert( 2025-09-07T07:00:09.9997624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:09.9997704Z encoder_outputs = self.encoder( 2025-09-07T07:00:09.9997950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:09.9998032Z layer_outputs = layer_module( 2025-09-07T07:00:09.9998326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:09.9998415Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:09.9998657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:09.9998739Z self_attention_outputs = self.attention( 2025-09-07T07:00:09.9998993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:09.9999065Z return func(*args, **kwargs) 2025-09-07T07:00:09.9999325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:09.9999460Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:09.9999715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:09.9999816Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:09.9999819Z 2025-09-07T07:00:09.9999929Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0000142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0000212Z return mod(**inputs) 2025-09-07T07:00:10.0000469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0000549Z outputs = self.bert( 2025-09-07T07:00:10.0000808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0000892Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0001167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0001244Z layer_outputs = layer_module( 2025-09-07T07:00:10.0001461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0001540Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0001793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0001876Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0002164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0002243Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0002544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0002685Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0002957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:10.0003052Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0003056Z 2025-09-07T07:00:10.0003165Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0003389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0003458Z return mod(**inputs) 2025-09-07T07:00:10.0003732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0003810Z outputs = self.bert( 2025-09-07T07:00:10.0004076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0004160Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0004424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0004534Z layer_outputs = layer_module( 2025-09-07T07:00:10.0004784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0004864Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0005118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0005204Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0005472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0005556Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0005841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0005969Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0006223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:10.0006345Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:10.0006563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:10.0006645Z return self.act(input) 2025-09-07T07:00:10.0006648Z 2025-09-07T07:00:10.0006760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0006956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0007027Z return mod(**inputs) 2025-09-07T07:00:10.0007273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0007363Z outputs = self.bert( 2025-09-07T07:00:10.0007618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0007695Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0007953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0008027Z layer_outputs = layer_module( 2025-09-07T07:00:10.0008249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0008364Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0008606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0008697Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0008958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0009045Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0009331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:10.0009470Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:10.0009731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:10.0009817Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0009822Z 2025-09-07T07:00:10.0009933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0010138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0010205Z return mod(**inputs) 2025-09-07T07:00:10.0010467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0010536Z outputs = self.bert( 2025-09-07T07:00:10.0010842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0010916Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0011170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0011242Z layer_outputs = layer_module( 2025-09-07T07:00:10.0011459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0011549Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0011796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0011885Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0012130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0012205Z return func(*args, **kwargs) 2025-09-07T07:00:10.0012466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0012539Z self_outputs = self.self( 2025-09-07T07:00:10.0012800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0012875Z return func(*args, **kwargs) 2025-09-07T07:00:10.0013143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:10.0013382Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:10.0013385Z 2025-09-07T07:00:10.0013504Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0013752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0013824Z return mod(**inputs) 2025-09-07T07:00:10.0014113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0014185Z outputs = self.bert( 2025-09-07T07:00:10.0014460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0014548Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0014827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0014930Z layer_outputs = layer_module( 2025-09-07T07:00:10.0015176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0015272Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0015531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0015618Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0015871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0015943Z return func(*args, **kwargs) 2025-09-07T07:00:10.0016203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0016275Z self_outputs = self.self( 2025-09-07T07:00:10.0016525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0016606Z return func(*args, **kwargs) 2025-09-07T07:00:10.0016869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:10.0016953Z self.key(current_states) 2025-09-07T07:00:10.0016957Z 2025-09-07T07:00:10.0017068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0018031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0018126Z return mod(**inputs) 2025-09-07T07:00:10.0018399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0018475Z outputs = self.bert( 2025-09-07T07:00:10.0018744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0018827Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0019099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0019176Z layer_outputs = layer_module( 2025-09-07T07:00:10.0019423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0019509Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0020177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0020300Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0020571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0020659Z return func(*args, **kwargs) 2025-09-07T07:00:10.0020933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0021018Z self_outputs = self.self( 2025-09-07T07:00:10.0021283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0021363Z return func(*args, **kwargs) 2025-09-07T07:00:10.0021719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:10.0021802Z self.value(current_states) 2025-09-07T07:00:10.0021806Z 2025-09-07T07:00:10.0021908Z cudagraph partition due to non gpu ops 2025-09-07T07:00:10.0022020Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0022243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0022322Z return mod(**inputs) 2025-09-07T07:00:10.0022598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0022711Z outputs = self.bert( 2025-09-07T07:00:10.0022986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0023068Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0023348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0023427Z layer_outputs = layer_module( 2025-09-07T07:00:10.0023684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0023770Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0024051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0024142Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0024411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0024495Z return func(*args, **kwargs) 2025-09-07T07:00:10.0024764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0024848Z self_outputs = self.self( 2025-09-07T07:00:10.0025113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0025252Z return func(*args, **kwargs) 2025-09-07T07:00:10.0025533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:10.0025735Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:10.0025742Z 2025-09-07T07:00:10.0025873Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0026106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0026179Z return mod(**inputs) 2025-09-07T07:00:10.0026468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0026542Z outputs = self.bert( 2025-09-07T07:00:10.0026826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0026901Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0027160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0027233Z layer_outputs = layer_module( 2025-09-07T07:00:10.0027457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0027545Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0027805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0027895Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0028138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0028231Z return func(*args, **kwargs) 2025-09-07T07:00:10.0028493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:10.0028630Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:10.0028887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:10.0028972Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0028976Z 2025-09-07T07:00:10.0029089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0029308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0029376Z return mod(**inputs) 2025-09-07T07:00:10.0029637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0029704Z outputs = self.bert( 2025-09-07T07:00:10.0029964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0030044Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0030295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0030375Z layer_outputs = layer_module( 2025-09-07T07:00:10.0030597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0030701Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0030963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0031043Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0031301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0031377Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0031691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0031811Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0032059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:10.0032139Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0032142Z 2025-09-07T07:00:10.0032241Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0032439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0032502Z return mod(**inputs) 2025-09-07T07:00:10.0032749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0032814Z outputs = self.bert( 2025-09-07T07:00:10.0033052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0033132Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0033367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0033444Z layer_outputs = layer_module( 2025-09-07T07:00:10.0033661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0033747Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0033989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0034070Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0034334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0034426Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0034715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0034836Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0035080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:10.0035206Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:10.0035441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:10.0035519Z return self.act(input) 2025-09-07T07:00:10.0035522Z 2025-09-07T07:00:10.0035625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0035829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0035906Z return mod(**inputs) 2025-09-07T07:00:10.0036159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0036233Z outputs = self.bert( 2025-09-07T07:00:10.0036485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0036563Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0036813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0036887Z layer_outputs = layer_module( 2025-09-07T07:00:10.0037118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0037206Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0037447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0037530Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0037813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0037896Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0038161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:10.0038298Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:10.0038537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:10.0038623Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0038626Z 2025-09-07T07:00:10.0038724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0038920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0038995Z return mod(**inputs) 2025-09-07T07:00:10.0039251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0039324Z outputs = self.bert( 2025-09-07T07:00:10.0039574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0039647Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0039900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0039973Z layer_outputs = layer_module( 2025-09-07T07:00:10.0040202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0040282Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0040556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0040648Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0040901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0040976Z return func(*args, **kwargs) 2025-09-07T07:00:10.0041218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0041295Z self_outputs = self.self( 2025-09-07T07:00:10.0041548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0041618Z return func(*args, **kwargs) 2025-09-07T07:00:10.0041867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:10.0042074Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:10.0042077Z 2025-09-07T07:00:10.0042189Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0042388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0042452Z return mod(**inputs) 2025-09-07T07:00:10.0042706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0042772Z outputs = self.bert( 2025-09-07T07:00:10.0043023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0043094Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0043345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0043417Z layer_outputs = layer_module( 2025-09-07T07:00:10.0043638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0043754Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0043995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0044083Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0044320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0044391Z return func(*args, **kwargs) 2025-09-07T07:00:10.0044639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0044708Z self_outputs = self.self( 2025-09-07T07:00:10.0044951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0045022Z return func(*args, **kwargs) 2025-09-07T07:00:10.0045266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:10.0045344Z self.key(current_states) 2025-09-07T07:00:10.0045347Z 2025-09-07T07:00:10.0045448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0045653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0045719Z return mod(**inputs) 2025-09-07T07:00:10.0045971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0046036Z outputs = self.bert( 2025-09-07T07:00:10.0046282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0046360Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0046620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0046697Z layer_outputs = layer_module( 2025-09-07T07:00:10.0046917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0046997Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0047245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0047327Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0047596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0047664Z return func(*args, **kwargs) 2025-09-07T07:00:10.0047905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0047984Z self_outputs = self.self( 2025-09-07T07:00:10.0048220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0048299Z return func(*args, **kwargs) 2025-09-07T07:00:10.0048539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:10.0048612Z self.value(current_states) 2025-09-07T07:00:10.0048622Z 2025-09-07T07:00:10.0048706Z cudagraph partition due to non gpu ops 2025-09-07T07:00:10.0048811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0049021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0049088Z return mod(**inputs) 2025-09-07T07:00:10.0049360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0049430Z outputs = self.bert( 2025-09-07T07:00:10.0049699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0049846Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0050112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0050197Z layer_outputs = layer_module( 2025-09-07T07:00:10.0050441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0050523Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0050781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0050863Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0051122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0051190Z return func(*args, **kwargs) 2025-09-07T07:00:10.0051435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0051510Z self_outputs = self.self( 2025-09-07T07:00:10.0051746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0051819Z return func(*args, **kwargs) 2025-09-07T07:00:10.0052063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:10.0052205Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:10.0052208Z 2025-09-07T07:00:10.0052311Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0052510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0052601Z return mod(**inputs) 2025-09-07T07:00:10.0052873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0052950Z outputs = self.bert( 2025-09-07T07:00:10.0053217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0053291Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0053553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0053644Z layer_outputs = layer_module( 2025-09-07T07:00:10.0053880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0053959Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0054212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0054303Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0054552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0054639Z return func(*args, **kwargs) 2025-09-07T07:00:10.0054885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:10.0055019Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:10.0055268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:10.0055354Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0055357Z 2025-09-07T07:00:10.0055471Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0055674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0055751Z return mod(**inputs) 2025-09-07T07:00:10.0056007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0056103Z outputs = self.bert( 2025-09-07T07:00:10.0056367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0056443Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0056699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0056773Z layer_outputs = layer_module( 2025-09-07T07:00:10.0057005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0057085Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0057335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0057430Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0057702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0057788Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0058069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0058191Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0058453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:10.0058537Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0058540Z 2025-09-07T07:00:10.0058653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0058891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0058989Z return mod(**inputs) 2025-09-07T07:00:10.0059260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0059330Z outputs = self.bert( 2025-09-07T07:00:10.0059601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0059678Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0059952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0060046Z layer_outputs = layer_module( 2025-09-07T07:00:10.0060284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0060377Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0060653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0060750Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0061019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0061096Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0061386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0061511Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0061775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:10.0061889Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:10.0062118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:10.0062192Z return self.act(input) 2025-09-07T07:00:10.0062196Z 2025-09-07T07:00:10.0062306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0062579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0062653Z return mod(**inputs) 2025-09-07T07:00:10.0062938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0063007Z outputs = self.bert( 2025-09-07T07:00:10.0063287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0063375Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0063652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0063734Z layer_outputs = layer_module( 2025-09-07T07:00:10.0063970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0064058Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0064332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0064422Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0064710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0064790Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0065094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:10.0065237Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:10.0065523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:10.0065726Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0065732Z 2025-09-07T07:00:10.0065848Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0066077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0066149Z return mod(**inputs) 2025-09-07T07:00:10.0066417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0066495Z outputs = self.bert( 2025-09-07T07:00:10.0066799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0066886Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0067232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0067322Z layer_outputs = layer_module( 2025-09-07T07:00:10.0067563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0067651Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0067918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0068006Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0068284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0068364Z return func(*args, **kwargs) 2025-09-07T07:00:10.0068625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0068709Z self_outputs = self.self( 2025-09-07T07:00:10.0068967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0069055Z return func(*args, **kwargs) 2025-09-07T07:00:10.0069391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-09-07T07:00:10.0069715Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:00:10.0069731Z 2025-09-07T07:00:10.0069846Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0070063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0070144Z return mod(**inputs) 2025-09-07T07:00:10.0070423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0070500Z outputs = self.bert( 2025-09-07T07:00:10.0070778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0070859Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0071137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0071218Z layer_outputs = layer_module( 2025-09-07T07:00:10.0071465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0071551Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0071831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0071933Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0072195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0072279Z return func(*args, **kwargs) 2025-09-07T07:00:10.0072566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0072679Z self_outputs = self.self( 2025-09-07T07:00:10.0072951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0073026Z return func(*args, **kwargs) 2025-09-07T07:00:10.0073311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-09-07T07:00:10.0073386Z self.key(current_states) 2025-09-07T07:00:10.0073391Z 2025-09-07T07:00:10.0073508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0073739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0073810Z return mod(**inputs) 2025-09-07T07:00:10.0074086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0074158Z outputs = self.bert( 2025-09-07T07:00:10.0074436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0074517Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0074781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0074866Z layer_outputs = layer_module( 2025-09-07T07:00:10.0075104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0075198Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0075460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0075548Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0075819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0075895Z return func(*args, **kwargs) 2025-09-07T07:00:10.0076204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0076280Z self_outputs = self.self( 2025-09-07T07:00:10.0076543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0076616Z return func(*args, **kwargs) 2025-09-07T07:00:10.0076876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-09-07T07:00:10.0076962Z self.value(current_states) 2025-09-07T07:00:10.0076966Z 2025-09-07T07:00:10.0077053Z cudagraph partition due to non gpu ops 2025-09-07T07:00:10.0077169Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0077382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0077464Z return mod(**inputs) 2025-09-07T07:00:10.0077725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0077794Z outputs = self.bert( 2025-09-07T07:00:10.0078055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0078131Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0078379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0078460Z layer_outputs = layer_module( 2025-09-07T07:00:10.0078689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0078781Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0079046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0079158Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0079418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0079491Z return func(*args, **kwargs) 2025-09-07T07:00:10.0079760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-09-07T07:00:10.0079833Z self_outputs = self.self( 2025-09-07T07:00:10.0080103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0080192Z return func(*args, **kwargs) 2025-09-07T07:00:10.0080470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-09-07T07:00:10.0080617Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:10.0080624Z 2025-09-07T07:00:10.0080726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0080941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0081007Z return mod(**inputs) 2025-09-07T07:00:10.0081275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0081349Z outputs = self.bert( 2025-09-07T07:00:10.0081605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0081688Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0081943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0082022Z layer_outputs = layer_module( 2025-09-07T07:00:10.0082248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0082330Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0082620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-09-07T07:00:10.0082703Z self_attention_outputs = self.attention( 2025-09-07T07:00:10.0082960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:00:10.0083028Z return func(*args, **kwargs) 2025-09-07T07:00:10.0083275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-09-07T07:00:10.0083415Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:00:10.0083665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-09-07T07:00:10.0083756Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0083761Z 2025-09-07T07:00:10.0083864Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0084078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0084146Z return mod(**inputs) 2025-09-07T07:00:10.0084413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0084487Z outputs = self.bert( 2025-09-07T07:00:10.0084740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0084822Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0085076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0085147Z layer_outputs = layer_module( 2025-09-07T07:00:10.0085380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0085478Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0085734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0085818Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0086085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0086169Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0086450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0086598Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0086852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-09-07T07:00:10.0086944Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0086947Z 2025-09-07T07:00:10.0087050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0087253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0087330Z return mod(**inputs) 2025-09-07T07:00:10.0087599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0087676Z outputs = self.bert( 2025-09-07T07:00:10.0087947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0088030Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0088284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0088356Z layer_outputs = layer_module( 2025-09-07T07:00:10.0088588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0088675Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0088976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0089076Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0089361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0089447Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0089727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-09-07T07:00:10.0089858Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:00:10.0090106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-09-07T07:00:10.0090223Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:00:10.0090449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:10.0090522Z return self.act(input) 2025-09-07T07:00:10.0090525Z 2025-09-07T07:00:10.0090639Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0090839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0090905Z return mod(**inputs) 2025-09-07T07:00:10.0091172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-09-07T07:00:10.0091239Z outputs = self.bert( 2025-09-07T07:00:10.0091512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-09-07T07:00:10.0091586Z encoder_outputs = self.encoder( 2025-09-07T07:00:10.0091865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-09-07T07:00:10.0091937Z layer_outputs = layer_module( 2025-09-07T07:00:10.0092166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:10.0092253Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:10.0092501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-09-07T07:00:10.0092590Z layer_output = apply_chunking_to_forward( 2025-09-07T07:00:10.0092871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:00:10.0092948Z return forward_fn(*input_tensors) 2025-09-07T07:00:10.0093239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-09-07T07:00:10.0093377Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:00:10.0093639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-09-07T07:00:10.0093723Z hidden_states = self.dense(hidden_states) 2025-09-07T07:00:10.0093726Z 2025-09-07T07:00:10.0093839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0094043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0094111Z return mod(**inputs) 2025-09-07T07:00:10.0094376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1781, in forward 2025-09-07T07:00:10.0094461Z logits = self.qa_outputs(sequence_output) 2025-09-07T07:00:10.0094465Z 2025-09-07T07:00:10.0094575Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0094774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0094842Z return mod(**inputs) 2025-09-07T07:00:10.0095145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1799, in forward 2025-09-07T07:00:10.0095257Z start_loss = loss_fct(start_logits, start_positions) 2025-09-07T07:00:10.0095261Z 2025-09-07T07:00:10.0095371Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:10.0095575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:10.0095644Z return mod(**inputs) 2025-09-07T07:00:10.0095911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1800, in forward 2025-09-07T07:00:10.0096007Z end_loss = loss_fct(end_logits, end_positions) 2025-09-07T07:00:10.0096012Z 2025-09-07T07:00:20.4004555Z Compilation time (from dynamo_timed): 16.981795059 2025-09-07T07:00:20.4005194Z pass 2025-09-07T07:00:20.4005588Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:20.4006562Z TIMING: _recursive_pre_grad_passes:0.00796 _recursive_joint_graph_passes:0.38442 _recursive_post_grad_passes:0.08381 async_compile.wait:0.00225 code_gen:9.75276 inductor_compile:11.0304 backend_compile:14.26468 gc:0.00042 entire_frame_compile:16.9818 total_wall_time:16.9818 2025-09-07T07:00:20.4007700Z STATS: call_* op count: 296 | FakeTensorMode.__torch_dispatch__:12365 | FakeTensor.__torch_dispatch__:4381 | ProxyTorchDispatchMode.__torch_dispatch__:4531 2025-09-07T07:00:20.4019689Z Dynamo produced 1 graphs covering 296 ops with 0 graph breaks (0 unique) 2025-09-07T07:00:23.1105121Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:00:23.1106626Z import pynvml # type: ignore[import] 2025-09-07T07:00:25.9123542Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:00:25.9124540Z from pkg_resources import resource_filename 2025-09-07T07:00:26.6768224Z 2025-09-07T07:00:46.0417806Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:00:46.0419947Z loading model: 0it [00:19, ?it/s] 2025-09-07T07:00:46.0454555Z cpu eval BlenderbotForCausalLM 2025-09-07T07:00:46.2460379Z Compilation time (from dynamo_timed): 0 2025-09-07T07:00:46.2460705Z pass_due_to_skip 2025-09-07T07:00:46.2463029Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:46.2463446Z TIMING: total_wall_time:0 2025-09-07T07:00:46.2463655Z STATS: call_* op count: 0 2025-09-07T07:00:46.2463948Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-09-07T07:00:48.2939360Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:00:48.2940649Z import pynvml # type: ignore[import] 2025-09-07T07:00:51.0605760Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:00:51.0606973Z from pkg_resources import resource_filename 2025-09-07T07:00:51.7331989Z 2025-09-07T07:00:52.4601563Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:00:52.4607582Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:00:52.4613639Z cpu eval BlenderbotSmallForCausalLM 2025-09-07T07:00:52.6295885Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:52.6828589Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:52.7343130Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:00:58.5981599Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5984282Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5984614Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5984837Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5985070Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5985296Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5985555Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5985845Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.5986114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.5986563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.5986959Z return mod(**inputs) 2025-09-07T07:00:58.5987482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.5988013Z outputs = self.model.decoder( 2025-09-07T07:00:58.5988612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.5989121Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.5989502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.5990156Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.5990617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.5991093Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.5991577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.5992120Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.5992397Z 2025-09-07T07:00:58.5992516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.5992889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.5993234Z return mod(**inputs) 2025-09-07T07:00:58.5993709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.5994194Z outputs = self.model.decoder( 2025-09-07T07:00:58.5994663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.5995143Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.5995525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.5995901Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.5996354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.5996827Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.5997332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.5997785Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.5997924Z 2025-09-07T07:00:58.5998133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.5998585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.5998916Z return mod(**inputs) 2025-09-07T07:00:58.5999346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.5999805Z outputs = self.model.decoder( 2025-09-07T07:00:58.6000251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6000699Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6001052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6001438Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6001906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6002389Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6002865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6003389Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6003552Z 2025-09-07T07:00:58.6003640Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6003878Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6004105Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6004323Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6004583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6004974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6005416Z return mod(**inputs) 2025-09-07T07:00:58.6005852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6006306Z outputs = self.model.decoder( 2025-09-07T07:00:58.6006773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6007231Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6007590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6008008Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6008486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6008988Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6009487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6009975Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6010421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6010934Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6011144Z 2025-09-07T07:00:58.6011258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6011653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6012024Z return mod(**inputs) 2025-09-07T07:00:58.6012471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6012957Z outputs = self.model.decoder( 2025-09-07T07:00:58.6013477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6013958Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6014338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6014732Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6015212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6015717Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6016219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6016727Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6017203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6017711Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6017891Z 2025-09-07T07:00:58.6018006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6018398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6018745Z return mod(**inputs) 2025-09-07T07:00:58.6019206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6019900Z outputs = self.model.decoder( 2025-09-07T07:00:58.6020354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6020808Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6021208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6021585Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6022037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6022514Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6022981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6023466Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6023628Z 2025-09-07T07:00:58.6023742Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6024140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6024495Z return mod(**inputs) 2025-09-07T07:00:58.6024951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6025421Z outputs = self.model.decoder( 2025-09-07T07:00:58.6026083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6026582Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6026967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6027359Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6027802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6028337Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6028522Z 2025-09-07T07:00:58.6028629Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6028987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6029381Z return mod(**inputs) 2025-09-07T07:00:58.6029802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6030247Z outputs = self.model.decoder( 2025-09-07T07:00:58.6030685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6031131Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6031476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6031844Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6032290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6032778Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6033175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6033514Z return self.act(input) 2025-09-07T07:00:58.6033630Z 2025-09-07T07:00:58.6033732Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6034105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6034430Z return mod(**inputs) 2025-09-07T07:00:58.6034843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6035287Z outputs = self.model.decoder( 2025-09-07T07:00:58.6035720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6036177Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6036524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6036875Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6037322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6037766Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6037923Z 2025-09-07T07:00:58.6038035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6038395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6038715Z return mod(**inputs) 2025-09-07T07:00:58.6039146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6039593Z outputs = self.model.decoder( 2025-09-07T07:00:58.6040041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6040476Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6040816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6041172Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6041610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6042071Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6042512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.6043020Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.6043233Z 2025-09-07T07:00:58.6043368Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6043735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6044069Z return mod(**inputs) 2025-09-07T07:00:58.6044493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6044964Z outputs = self.model.decoder( 2025-09-07T07:00:58.6045413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6045840Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6046188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6046560Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6047022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6047488Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6047956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.6048399Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.6048541Z 2025-09-07T07:00:58.6048645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6049008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6049345Z return mod(**inputs) 2025-09-07T07:00:58.6049759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6050214Z outputs = self.model.decoder( 2025-09-07T07:00:58.6050648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6051087Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6051435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6051803Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6052223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6052700Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6053188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6053644Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6053786Z 2025-09-07T07:00:58.6053876Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6054093Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6054314Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6054525Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6054766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6055129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6055466Z return mod(**inputs) 2025-09-07T07:00:58.6055900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6056350Z outputs = self.model.decoder( 2025-09-07T07:00:58.6056784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6057223Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6057605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6057974Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6058434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6058910Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6059392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6059873Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6060342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6060845Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6061038Z 2025-09-07T07:00:58.6061149Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6061529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6061870Z return mod(**inputs) 2025-09-07T07:00:58.6062309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6062768Z outputs = self.model.decoder( 2025-09-07T07:00:58.6063214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6063675Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6064044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6064427Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6064911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6065421Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6066017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6066558Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6067045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6067565Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6067759Z 2025-09-07T07:00:58.6067869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6068242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6068583Z return mod(**inputs) 2025-09-07T07:00:58.6069012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6069458Z outputs = self.model.decoder( 2025-09-07T07:00:58.6069933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6070422Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6070815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6071231Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6071717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6072246Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6072737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6073252Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6073402Z 2025-09-07T07:00:58.6073525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6073915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6074281Z return mod(**inputs) 2025-09-07T07:00:58.6074756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6075265Z outputs = self.model.decoder( 2025-09-07T07:00:58.6075750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6076244Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6076641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6077061Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6077561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6078114Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6078320Z 2025-09-07T07:00:58.6078442Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6078853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6079241Z return mod(**inputs) 2025-09-07T07:00:58.6079714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6080208Z outputs = self.model.decoder( 2025-09-07T07:00:58.6080714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6081204Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6081601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6081962Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6082412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6082936Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6083334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6083686Z return self.act(input) 2025-09-07T07:00:58.6083800Z 2025-09-07T07:00:58.6083906Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6084274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6084605Z return mod(**inputs) 2025-09-07T07:00:58.6085035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6085482Z outputs = self.model.decoder( 2025-09-07T07:00:58.6085918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6086373Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6086727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6087097Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6087555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6088004Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6088150Z 2025-09-07T07:00:58.6088289Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6088658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6088988Z return mod(**inputs) 2025-09-07T07:00:58.6089406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6089863Z outputs = self.model.decoder( 2025-09-07T07:00:58.6090304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6090762Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6091114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6091482Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6091941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6092410Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6092858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.6093373Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.6093575Z 2025-09-07T07:00:58.6093680Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6094042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6094366Z return mod(**inputs) 2025-09-07T07:00:58.6094783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6095244Z outputs = self.model.decoder( 2025-09-07T07:00:58.6095672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6096106Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6096453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6096818Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6097284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6097762Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6098276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.6098767Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.6098914Z 2025-09-07T07:00:58.6099046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6099406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6099743Z return mod(**inputs) 2025-09-07T07:00:58.6100158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6100601Z outputs = self.model.decoder( 2025-09-07T07:00:58.6101039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6101480Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6101840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6102217Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6102711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6103196Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6103695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6104198Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6104358Z 2025-09-07T07:00:58.6104448Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6104683Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6104905Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6105131Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6105384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6105861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6106257Z return mod(**inputs) 2025-09-07T07:00:58.6106724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6107213Z outputs = self.model.decoder( 2025-09-07T07:00:58.6107684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6108181Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6108565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6108962Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6109443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6109990Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6110502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6111010Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6111501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6112037Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6112238Z 2025-09-07T07:00:58.6112379Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6112770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6113135Z return mod(**inputs) 2025-09-07T07:00:58.6113604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6114089Z outputs = self.model.decoder( 2025-09-07T07:00:58.6114563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6115027Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6115410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6115781Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6116232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6116717Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6117172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6117633Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6118110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6118570Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6118728Z 2025-09-07T07:00:58.6118839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6119193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6119514Z return mod(**inputs) 2025-09-07T07:00:58.6120072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6120533Z outputs = self.model.decoder( 2025-09-07T07:00:58.6120967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6121416Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6121766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6122133Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6122585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6123046Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6123521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6123987Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6124124Z 2025-09-07T07:00:58.6124239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6124608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6125008Z return mod(**inputs) 2025-09-07T07:00:58.6125447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6125908Z outputs = self.model.decoder( 2025-09-07T07:00:58.6126344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6126776Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6127131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6127521Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6127964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6128449Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6128624Z 2025-09-07T07:00:58.6128733Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6129103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6129444Z return mod(**inputs) 2025-09-07T07:00:58.6129856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6130293Z outputs = self.model.decoder( 2025-09-07T07:00:58.6130718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6131160Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6131508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6131868Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6132315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6132824Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6133202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6133540Z return self.act(input) 2025-09-07T07:00:58.6133649Z 2025-09-07T07:00:58.6133759Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6134111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6134435Z return mod(**inputs) 2025-09-07T07:00:58.6134846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6135287Z outputs = self.model.decoder( 2025-09-07T07:00:58.6135729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6136174Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6136521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6136878Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6137318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6137763Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6137901Z 2025-09-07T07:00:58.6138008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6138366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6138691Z return mod(**inputs) 2025-09-07T07:00:58.6139107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6139573Z outputs = self.model.decoder( 2025-09-07T07:00:58.6140013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6140450Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6140799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6141162Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6141625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6142103Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6142576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.6143119Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.6143341Z 2025-09-07T07:00:58.6143465Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6143847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6144205Z return mod(**inputs) 2025-09-07T07:00:58.6144658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6145142Z outputs = self.model.decoder( 2025-09-07T07:00:58.6145668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6146223Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6146619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6147029Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6147567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6148035Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6148489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.6148933Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.6149079Z 2025-09-07T07:00:58.6149184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6149542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6149860Z return mod(**inputs) 2025-09-07T07:00:58.6150278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6150719Z outputs = self.model.decoder( 2025-09-07T07:00:58.6151154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6151595Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6151935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6152297Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6152757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6153244Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6153719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6154212Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6154364Z 2025-09-07T07:00:58.6154447Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6154667Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6154883Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6155086Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6155328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6155695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6156048Z return mod(**inputs) 2025-09-07T07:00:58.6156476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6156933Z outputs = self.model.decoder( 2025-09-07T07:00:58.6157385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6157835Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6158195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6158560Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6159020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6159487Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6159951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6160416Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6160856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6161342Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6161528Z 2025-09-07T07:00:58.6161631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6162014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6162332Z return mod(**inputs) 2025-09-07T07:00:58.6162733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6163161Z outputs = self.model.decoder( 2025-09-07T07:00:58.6163584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6164019Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6164368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6164730Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6165190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6165665Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6166137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6166611Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6167061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6167540Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6167712Z 2025-09-07T07:00:58.6167819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6168187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6168547Z return mod(**inputs) 2025-09-07T07:00:58.6168975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6169430Z outputs = self.model.decoder( 2025-09-07T07:00:58.6169882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6170339Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6170700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6171063Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6171498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6171960Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6172424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6172864Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6173010Z 2025-09-07T07:00:58.6173116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6173483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6173821Z return mod(**inputs) 2025-09-07T07:00:58.6174251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6174712Z outputs = self.model.decoder( 2025-09-07T07:00:58.6175214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6175648Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6175999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6176394Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6176837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6177329Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6177513Z 2025-09-07T07:00:58.6177621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6177992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6178330Z return mod(**inputs) 2025-09-07T07:00:58.6178748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6179186Z outputs = self.model.decoder( 2025-09-07T07:00:58.6179627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6180070Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6180414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6180780Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6181232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6181736Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6182135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6182479Z return self.act(input) 2025-09-07T07:00:58.6182599Z 2025-09-07T07:00:58.6182730Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6183094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6183436Z return mod(**inputs) 2025-09-07T07:00:58.6183885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6184363Z outputs = self.model.decoder( 2025-09-07T07:00:58.6184839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6185336Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6185793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6186190Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6186672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6187136Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6187275Z 2025-09-07T07:00:58.6187390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6187747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6188070Z return mod(**inputs) 2025-09-07T07:00:58.6188500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6188952Z outputs = self.model.decoder( 2025-09-07T07:00:58.6189399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6189844Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6190193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6190553Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6191051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6191515Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6191967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.6192477Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.6192690Z 2025-09-07T07:00:58.6192792Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6193145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6193467Z return mod(**inputs) 2025-09-07T07:00:58.6193875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6194316Z outputs = self.model.decoder( 2025-09-07T07:00:58.6194752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6195185Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6195528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6195885Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6196331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6196791Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6197259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.6197715Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.6197849Z 2025-09-07T07:00:58.6197954Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6198314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6198638Z return mod(**inputs) 2025-09-07T07:00:58.6199052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6199500Z outputs = self.model.decoder( 2025-09-07T07:00:58.6199936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6200373Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6200732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6201083Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6201509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6201960Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6202406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6202846Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6202986Z 2025-09-07T07:00:58.6203073Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6203276Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6203482Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6203686Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6204029Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6204389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6204715Z return mod(**inputs) 2025-09-07T07:00:58.6205176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6205609Z outputs = self.model.decoder( 2025-09-07T07:00:58.6206033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6206450Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6206792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6207138Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6207572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6208024Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6208464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6208919Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6209346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6209811Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6209992Z 2025-09-07T07:00:58.6210099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6210444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6210762Z return mod(**inputs) 2025-09-07T07:00:58.6211165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6211617Z outputs = self.model.decoder( 2025-09-07T07:00:58.6212033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6212473Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6212828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6213185Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6213680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6214155Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6214661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6215135Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6215592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6216058Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6216223Z 2025-09-07T07:00:58.6216330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6216697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6217029Z return mod(**inputs) 2025-09-07T07:00:58.6217459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6217912Z outputs = self.model.decoder( 2025-09-07T07:00:58.6218348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6218799Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6219192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6219700Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6220171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6220647Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6221118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6221585Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6221727Z 2025-09-07T07:00:58.6221842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6222203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6222552Z return mod(**inputs) 2025-09-07T07:00:58.6223007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6223501Z outputs = self.model.decoder( 2025-09-07T07:00:58.6223985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6224525Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6224907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6225302Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6225835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6226371Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6226631Z 2025-09-07T07:00:58.6226749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6227153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6227497Z return mod(**inputs) 2025-09-07T07:00:58.6227919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6228369Z outputs = self.model.decoder( 2025-09-07T07:00:58.6228805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6229284Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6229638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6230007Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6230455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6230950Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6231347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6231698Z return self.act(input) 2025-09-07T07:00:58.6231810Z 2025-09-07T07:00:58.6231921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6232282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6232614Z return mod(**inputs) 2025-09-07T07:00:58.6233040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6233484Z outputs = self.model.decoder( 2025-09-07T07:00:58.6233928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6234415Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6234774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6235143Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6235592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6236040Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6236189Z 2025-09-07T07:00:58.6236294Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6236657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6236990Z return mod(**inputs) 2025-09-07T07:00:58.6237420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6237852Z outputs = self.model.decoder( 2025-09-07T07:00:58.6238288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6238720Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6239072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6239432Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6239862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6240325Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6240782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.6241318Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.6241521Z 2025-09-07T07:00:58.6241630Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6241978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6242298Z return mod(**inputs) 2025-09-07T07:00:58.6242711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6243168Z outputs = self.model.decoder( 2025-09-07T07:00:58.6243582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6244005Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6244350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6244708Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6245147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6245602Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6246047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.6246477Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.6246608Z 2025-09-07T07:00:58.6246718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6247072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6247392Z return mod(**inputs) 2025-09-07T07:00:58.6247801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6248239Z outputs = self.model.decoder( 2025-09-07T07:00:58.6248715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6249154Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6249492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6249851Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6250291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6250755Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6251216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6251666Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6251811Z 2025-09-07T07:00:58.6251893Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6252106Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6252317Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6252511Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6252740Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6253084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6253397Z return mod(**inputs) 2025-09-07T07:00:58.6253795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6254230Z outputs = self.model.decoder( 2025-09-07T07:00:58.6254662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6255123Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6255474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6255829Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6256275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6256742Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6257240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6257717Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6258157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6258654Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6258848Z 2025-09-07T07:00:58.6258959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6259328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6259666Z return mod(**inputs) 2025-09-07T07:00:58.6260087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6260554Z outputs = self.model.decoder( 2025-09-07T07:00:58.6261030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6261507Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6261886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6262279Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6262884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6263413Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6263938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6264457Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6264960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6265475Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6265723Z 2025-09-07T07:00:58.6265843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6266247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6266605Z return mod(**inputs) 2025-09-07T07:00:58.6267055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6267496Z outputs = self.model.decoder( 2025-09-07T07:00:58.6267936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6268382Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6268733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6269111Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6269569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6270099Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6270602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6271091Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6271249Z 2025-09-07T07:00:58.6271361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6271747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6272107Z return mod(**inputs) 2025-09-07T07:00:58.6272570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6273048Z outputs = self.model.decoder( 2025-09-07T07:00:58.6273518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6273965Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6274328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6274683Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6275123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6275607Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6275777Z 2025-09-07T07:00:58.6275890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6276245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6276562Z return mod(**inputs) 2025-09-07T07:00:58.6276977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6277415Z outputs = self.model.decoder( 2025-09-07T07:00:58.6277917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6278400Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6278771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6279174Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6279666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6280164Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6280576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6280962Z return self.act(input) 2025-09-07T07:00:58.6281091Z 2025-09-07T07:00:58.6281202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6281594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6281944Z return mod(**inputs) 2025-09-07T07:00:58.6282388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6282867Z outputs = self.model.decoder( 2025-09-07T07:00:58.6283335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6283811Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6284185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6284579Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6285057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6285575Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6285726Z 2025-09-07T07:00:58.6285845Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6286228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6286569Z return mod(**inputs) 2025-09-07T07:00:58.6287019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6287514Z outputs = self.model.decoder( 2025-09-07T07:00:58.6287980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6288443Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6288828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6289198Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6289655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6290132Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6290595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.6291125Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.6291342Z 2025-09-07T07:00:58.6291448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6291814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6292144Z return mod(**inputs) 2025-09-07T07:00:58.6292565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6293049Z outputs = self.model.decoder( 2025-09-07T07:00:58.6293502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6293960Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6294326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6294700Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6295163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6295650Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6296131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.6296598Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.6296742Z 2025-09-07T07:00:58.6296859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6297236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6297581Z return mod(**inputs) 2025-09-07T07:00:58.6298016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6298469Z outputs = self.model.decoder( 2025-09-07T07:00:58.6298922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6299382Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6299754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6300140Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6300584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6301056Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6301522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6301982Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6302157Z 2025-09-07T07:00:58.6302246Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6302465Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6302690Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6302910Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6303158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6303540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6303889Z return mod(**inputs) 2025-09-07T07:00:58.6304340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6304815Z outputs = self.model.decoder( 2025-09-07T07:00:58.6305283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6305838Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6306237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6306641Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6307142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6307673Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6308226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6308731Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6309218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6309745Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6309946Z 2025-09-07T07:00:58.6310064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6310462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6310812Z return mod(**inputs) 2025-09-07T07:00:58.6311275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6311752Z outputs = self.model.decoder( 2025-09-07T07:00:58.6312218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6312692Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6313085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6313490Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6313988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6314507Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6315021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6315573Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6316059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6316562Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6316735Z 2025-09-07T07:00:58.6316847Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6317234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6317618Z return mod(**inputs) 2025-09-07T07:00:58.6318065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6318534Z outputs = self.model.decoder( 2025-09-07T07:00:58.6319005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6319481Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6320025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6320473Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6320959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6321490Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6322003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6322485Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6322633Z 2025-09-07T07:00:58.6322753Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6323131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6323483Z return mod(**inputs) 2025-09-07T07:00:58.6324012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6324454Z outputs = self.model.decoder( 2025-09-07T07:00:58.6324920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6325374Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6325724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6326086Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6326531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6327013Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6327194Z 2025-09-07T07:00:58.6327299Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6327662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6327988Z return mod(**inputs) 2025-09-07T07:00:58.6328406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6328857Z outputs = self.model.decoder( 2025-09-07T07:00:58.6329305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6329755Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6330121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6330529Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6330975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6331467Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6331862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6332228Z return self.act(input) 2025-09-07T07:00:58.6332337Z 2025-09-07T07:00:58.6332449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6332825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6333149Z return mod(**inputs) 2025-09-07T07:00:58.6333559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6334002Z outputs = self.model.decoder( 2025-09-07T07:00:58.6334452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6334896Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6335255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6335635Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6336076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6336516Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6336660Z 2025-09-07T07:00:58.6336764Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6337124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6337449Z return mod(**inputs) 2025-09-07T07:00:58.6337865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6338320Z outputs = self.model.decoder( 2025-09-07T07:00:58.6338756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6339193Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6339544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6339900Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6340346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6340812Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6341274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:00:58.6341796Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:00:58.6341998Z 2025-09-07T07:00:58.6342101Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6342458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6342781Z return mod(**inputs) 2025-09-07T07:00:58.6343208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6343680Z outputs = self.model.decoder( 2025-09-07T07:00:58.6344140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6344617Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6345008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6345396Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6345947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6346467Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6346984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:00:58.6347514Z key_states = self.k_proj(current_states) 2025-09-07T07:00:58.6347663Z 2025-09-07T07:00:58.6347791Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6348151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6348472Z return mod(**inputs) 2025-09-07T07:00:58.6348891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6349336Z outputs = self.model.decoder( 2025-09-07T07:00:58.6349767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6350200Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6350544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6350904Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6351343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6351803Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6352257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:00:58.6352707Z value_states = self.v_proj(current_states) 2025-09-07T07:00:58.6352852Z 2025-09-07T07:00:58.6354010Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6354247Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6354456Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6354656Z cudagraph partition due to non gpu ops 2025-09-07T07:00:58.6354892Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6355254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6355592Z return mod(**inputs) 2025-09-07T07:00:58.6355998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6356432Z outputs = self.model.decoder( 2025-09-07T07:00:58.6356861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6357283Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6357625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6357969Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6358400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6358853Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6359296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6359742Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6360165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:00:58.6360666Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:00:58.6360852Z 2025-09-07T07:00:58.6360959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6361313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6361626Z return mod(**inputs) 2025-09-07T07:00:58.6362034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6362482Z outputs = self.model.decoder( 2025-09-07T07:00:58.6362902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6363326Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6363660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6364014Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6364444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6364913Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6365362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:00:58.6365806Z attn_output, attn_weights = attention_interface( 2025-09-07T07:00:58.6366239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:00:58.6366682Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:00:58.6366837Z 2025-09-07T07:00:58.6366944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6367295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6367606Z return mod(**inputs) 2025-09-07T07:00:58.6368043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6368471Z outputs = self.model.decoder( 2025-09-07T07:00:58.6368892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6369307Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6369648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6370003Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6370436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:00:58.6370888Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:00:58.6371330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:00:58.6371762Z attn_output = self.out_proj(attn_output) 2025-09-07T07:00:58.6371900Z 2025-09-07T07:00:58.6372001Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6372353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6372681Z return mod(**inputs) 2025-09-07T07:00:58.6373072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6373493Z outputs = self.model.decoder( 2025-09-07T07:00:58.6373920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6374378Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6374731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6375086Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6375524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6375995Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6376180Z 2025-09-07T07:00:58.6376288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6376641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6376951Z return mod(**inputs) 2025-09-07T07:00:58.6377354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6377782Z outputs = self.model.decoder( 2025-09-07T07:00:58.6378208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6378634Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6378982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6379342Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6379783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:00:58.6380261Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:00:58.6380644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:00:58.6380996Z return self.act(input) 2025-09-07T07:00:58.6381114Z 2025-09-07T07:00:58.6381216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6381643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6381973Z return mod(**inputs) 2025-09-07T07:00:58.6382399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-09-07T07:00:58.6382874Z outputs = self.model.decoder( 2025-09-07T07:00:58.6383349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:00:58.6383838Z layer_outputs = decoder_layer( 2025-09-07T07:00:58.6384219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:00:58.6384632Z return super().__call__(*args, **kwargs) 2025-09-07T07:00:58.6385127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:00:58.6385698Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:00:58.6385860Z 2025-09-07T07:00:58.6385990Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6386384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6386762Z return mod(**inputs) 2025-09-07T07:00:58.6387212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1528, in forward 2025-09-07T07:00:58.6387691Z logits = self.lm_head(outputs[0]) 2025-09-07T07:00:58.6387827Z 2025-09-07T07:00:58.6387942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:00:58.6388318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:00:58.6388698Z return mod(**inputs) 2025-09-07T07:00:58.6389154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1534, in forward 2025-09-07T07:00:58.6389709Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:00:58.6389924Z 2025-09-07T07:01:08.8566602Z Compilation time (from dynamo_timed): 14.942170786 2025-09-07T07:01:08.8587155Z pass 2025-09-07T07:01:08.8587580Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:08.8588736Z TIMING: _recursive_pre_grad_passes:0.00621 _recursive_joint_graph_passes:0.56212 _recursive_post_grad_passes:0.05924 async_compile.wait:0.80618 code_gen:9.93265 inductor_compile:10.84685 backend_compile:13.31039 gc:0.00153 entire_frame_compile:14.94217 total_wall_time:14.94217 2025-09-07T07:01:08.8589822Z STATS: call_* op count: 252 | FakeTensorMode.__torch_dispatch__:9090 | FakeTensor.__torch_dispatch__:3104 | ProxyTorchDispatchMode.__torch_dispatch__:3279 2025-09-07T07:01:08.8596385Z Dynamo produced 1 graphs covering 252 ops with 0 graph breaks (0 unique) 2025-09-07T07:01:11.4891242Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:01:11.4892152Z import pynvml # type: ignore[import] 2025-09-07T07:01:14.3421239Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:01:14.3422273Z from pkg_resources import resource_filename 2025-09-07T07:01:15.0125266Z 2025-09-07T07:01:15.9946787Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:01:15.9947104Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:01:15.9966220Z cpu eval BlenderbotSmallForConditionalGeneration 2025-09-07T07:01:16.2567311Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:16.3566206Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:16.4541504Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:28.4728382Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4728730Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4729003Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4729220Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4729432Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4729646Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4729858Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4730073Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4730327Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4730743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4731098Z return mod(**inputs) 2025-09-07T07:01:28.4731601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4732124Z outputs = self.model( 2025-09-07T07:01:28.4732680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4733396Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4733917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4735316Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4735716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4736128Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4736623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4737124Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4737630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.4738282Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.4738516Z 2025-09-07T07:01:28.4738644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4739059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4739420Z return mod(**inputs) 2025-09-07T07:01:28.4739913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4740418Z outputs = self.model( 2025-09-07T07:01:28.4740897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4741400Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4741900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4742406Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4742817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4743236Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4743742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4744350Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4744871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.4745380Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.4745533Z 2025-09-07T07:01:28.4745938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4746362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4746725Z return mod(**inputs) 2025-09-07T07:01:28.4747199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4747689Z outputs = self.model( 2025-09-07T07:01:28.4748163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4748664Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4749113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4749586Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4749977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4750357Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4750803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4751312Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4751808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.4752325Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.4752482Z 2025-09-07T07:01:28.4752580Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4752804Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4753031Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4753253Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4753505Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4753893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4754273Z return mod(**inputs) 2025-09-07T07:01:28.4754740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4755219Z outputs = self.model( 2025-09-07T07:01:28.4755676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4756149Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4756637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4757111Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4757488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4757877Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4758353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4758849Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4759341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4759847Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4760365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.4760887Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.4761096Z 2025-09-07T07:01:28.4761208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4761596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4761949Z return mod(**inputs) 2025-09-07T07:01:28.4762403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4762883Z outputs = self.model( 2025-09-07T07:01:28.4763345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4763826Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4764314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4764795Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4765196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4765590Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4766066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4766563Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4767064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4767607Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4768104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.4768623Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.4768807Z 2025-09-07T07:01:28.4768931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4769326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4769708Z return mod(**inputs) 2025-09-07T07:01:28.4770174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4770658Z outputs = self.model( 2025-09-07T07:01:28.4771113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4771612Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4772096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4772582Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4772968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4773361Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4773851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4774360Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4774865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.4775362Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.4775514Z 2025-09-07T07:01:28.4775630Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4776062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4776430Z return mod(**inputs) 2025-09-07T07:01:28.4776894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4777389Z outputs = self.model( 2025-09-07T07:01:28.4777856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4778348Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4778832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4779315Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4779686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4780078Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4780558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4781093Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4781288Z 2025-09-07T07:01:28.4781413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4781816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4782198Z return mod(**inputs) 2025-09-07T07:01:28.4782650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4783142Z outputs = self.model( 2025-09-07T07:01:28.4783600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4784071Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4784547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4785021Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4785404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4785924Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4786419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4786970Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4787413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.4787789Z return self.act(input) 2025-09-07T07:01:28.4787912Z 2025-09-07T07:01:28.4788033Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4788442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4788793Z return mod(**inputs) 2025-09-07T07:01:28.4789246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4789720Z outputs = self.model( 2025-09-07T07:01:28.4790163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4790643Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4791116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4791587Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4792027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4792422Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4792909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.4793402Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.4793557Z 2025-09-07T07:01:28.4793680Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4794078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4794431Z return mod(**inputs) 2025-09-07T07:01:28.4794894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4795375Z outputs = self.model( 2025-09-07T07:01:28.4795848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4796403Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4796876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4797353Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4797745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4798142Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4798620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4799141Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4799644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.4800207Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.4800432Z 2025-09-07T07:01:28.4800552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4800935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4801312Z return mod(**inputs) 2025-09-07T07:01:28.4801761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4802232Z outputs = self.model( 2025-09-07T07:01:28.4802685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4803158Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4803640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4804100Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4804461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4804830Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4805279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4805773Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4806254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.4806710Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.4806847Z 2025-09-07T07:01:28.4806959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4807365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4807728Z return mod(**inputs) 2025-09-07T07:01:28.4808190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4808662Z outputs = self.model( 2025-09-07T07:01:28.4809109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4809589Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4810062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4810538Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4810928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4811321Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4811811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4812306Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4812811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.4813314Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.4813466Z 2025-09-07T07:01:28.4813555Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4813794Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4814023Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4814274Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4814525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4814928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4815286Z return mod(**inputs) 2025-09-07T07:01:28.4815750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4816237Z outputs = self.model( 2025-09-07T07:01:28.4816697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4817214Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4817704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4818192Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4818578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4818982Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4819478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4820194Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4820710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4821229Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4821734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.4822270Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.4822477Z 2025-09-07T07:01:28.4823100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4823505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4823946Z return mod(**inputs) 2025-09-07T07:01:28.4824417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4824897Z outputs = self.model( 2025-09-07T07:01:28.4825364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4825913Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4826397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4826884Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4827270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4827675Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4828157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4828641Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4829133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4829639Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4830124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.4830624Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.4830799Z 2025-09-07T07:01:28.4830910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4831343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4831694Z return mod(**inputs) 2025-09-07T07:01:28.4832147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4832609Z outputs = self.model( 2025-09-07T07:01:28.4833060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4833566Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4834036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4834510Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4834889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4835328Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4835839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4836344Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4836850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.4837325Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.4837476Z 2025-09-07T07:01:28.4837581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4837946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4838280Z return mod(**inputs) 2025-09-07T07:01:28.4838704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4839157Z outputs = self.model( 2025-09-07T07:01:28.4839624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4840082Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4840532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4841003Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4841387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4841782Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4842261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4842784Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4842964Z 2025-09-07T07:01:28.4843071Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4843446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4843780Z return mod(**inputs) 2025-09-07T07:01:28.4844211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4844657Z outputs = self.model( 2025-09-07T07:01:28.4845078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4845528Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4845975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4846445Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4846797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4847174Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4847624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4848114Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4848514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.4848888Z return self.act(input) 2025-09-07T07:01:28.4849009Z 2025-09-07T07:01:28.4849116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4849486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4849818Z return mod(**inputs) 2025-09-07T07:01:28.4850245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4850682Z outputs = self.model( 2025-09-07T07:01:28.4851109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4851557Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4852002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4852445Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4852794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4853161Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4853608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.4854067Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.4854205Z 2025-09-07T07:01:28.4854382Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4854754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4855086Z return mod(**inputs) 2025-09-07T07:01:28.4855524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4855986Z outputs = self.model( 2025-09-07T07:01:28.4856414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4856875Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4857327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4857786Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4858154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4858524Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4858986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4859465Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4859942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.4860477Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.4860691Z 2025-09-07T07:01:28.4860797Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4861189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4861521Z return mod(**inputs) 2025-09-07T07:01:28.4861947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4862386Z outputs = self.model( 2025-09-07T07:01:28.4862830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4863319Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4863789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4864262Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4864629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4865022Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4865510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4866083Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4866593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.4867080Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.4867241Z 2025-09-07T07:01:28.4867353Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4867751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4868111Z return mod(**inputs) 2025-09-07T07:01:28.4868571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4869049Z outputs = self.model( 2025-09-07T07:01:28.4869555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4870040Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4870510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4870978Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4871360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4871750Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4872229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4872718Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4873205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.4873693Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.4873852Z 2025-09-07T07:01:28.4873941Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4874174Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4874402Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4874619Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4874875Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4875261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4875603Z return mod(**inputs) 2025-09-07T07:01:28.4876012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4876471Z outputs = self.model( 2025-09-07T07:01:28.4876892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4877335Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4877768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4878199Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4878547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4878930Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4879373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4879826Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4880271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4880734Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4881190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.4881682Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.4881863Z 2025-09-07T07:01:28.4881972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4882323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4882648Z return mod(**inputs) 2025-09-07T07:01:28.4883064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4883497Z outputs = self.model( 2025-09-07T07:01:28.4883906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4884383Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4884817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4885248Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4885601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4885961Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4886397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4886849Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4887356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4887834Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4888283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.4888751Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.4888921Z 2025-09-07T07:01:28.4889028Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4889401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4889721Z return mod(**inputs) 2025-09-07T07:01:28.4890131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4890563Z outputs = self.model( 2025-09-07T07:01:28.4890993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4891427Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4891847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4892279Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4892625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4893006Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4893448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4893898Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4894353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.4894802Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.4894938Z 2025-09-07T07:01:28.4895054Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4895414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4895731Z return mod(**inputs) 2025-09-07T07:01:28.4896152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4896595Z outputs = self.model( 2025-09-07T07:01:28.4897035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4897480Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4897913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4898361Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4898752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4899124Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4899572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4900076Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4900265Z 2025-09-07T07:01:28.4900377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4900751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4901091Z return mod(**inputs) 2025-09-07T07:01:28.4901520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4901989Z outputs = self.model( 2025-09-07T07:01:28.4902450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4902939Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4903416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4903887Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4904277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4904685Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4905179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4905976Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4906411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.4906796Z return self.act(input) 2025-09-07T07:01:28.4906919Z 2025-09-07T07:01:28.4907049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4907418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4907744Z return mod(**inputs) 2025-09-07T07:01:28.4908173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4908644Z outputs = self.model( 2025-09-07T07:01:28.4909073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4909525Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4909965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4910413Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4910767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4911137Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4911583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.4912033Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.4912181Z 2025-09-07T07:01:28.4912287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4912654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4912986Z return mod(**inputs) 2025-09-07T07:01:28.4913409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4913887Z outputs = self.model( 2025-09-07T07:01:28.4914317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4914768Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4915213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4915655Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4916025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4916388Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4916829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4917291Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4917743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.4918255Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.4918467Z 2025-09-07T07:01:28.4918571Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4918928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4919253Z return mod(**inputs) 2025-09-07T07:01:28.4919837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4920289Z outputs = self.model( 2025-09-07T07:01:28.4920710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4921210Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4921642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4922078Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4922429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4922795Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4923270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4923717Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4924173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.4924621Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.4924755Z 2025-09-07T07:01:28.4924872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4925232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4925550Z return mod(**inputs) 2025-09-07T07:01:28.4925966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4926400Z outputs = self.model( 2025-09-07T07:01:28.4926813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4927246Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4927675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4928114Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4928509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4928878Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4929322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4929787Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4930249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.4930712Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.4930856Z 2025-09-07T07:01:28.4930945Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4931154Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4931367Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4931577Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4931820Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4932177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4932510Z return mod(**inputs) 2025-09-07T07:01:28.4932940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4933391Z outputs = self.model( 2025-09-07T07:01:28.4933823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4934267Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4934712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4935174Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4935526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4935888Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4936336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4936810Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4937263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4937748Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4938208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.4938680Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.4938872Z 2025-09-07T07:01:28.4938977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4939341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4939675Z return mod(**inputs) 2025-09-07T07:01:28.4940094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4940543Z outputs = self.model( 2025-09-07T07:01:28.4940972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4941425Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4941861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4942306Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4942664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4943070Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4943538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4944025Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4944501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4945008Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4945505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.4946040Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.4946221Z 2025-09-07T07:01:28.4946341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4946726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4947074Z return mod(**inputs) 2025-09-07T07:01:28.4947521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4947967Z outputs = self.model( 2025-09-07T07:01:28.4948387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4948845Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4949296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4949746Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4950132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4950499Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4950955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4951417Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4951891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.4952367Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.4952508Z 2025-09-07T07:01:28.4952616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4952987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4953321Z return mod(**inputs) 2025-09-07T07:01:28.4953755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4954207Z outputs = self.model( 2025-09-07T07:01:28.4954632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4955084Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4955530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4955983Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4956336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4956704Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4957157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4957653Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4957824Z 2025-09-07T07:01:28.4957973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4958317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4958632Z return mod(**inputs) 2025-09-07T07:01:28.4959043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4959473Z outputs = self.model( 2025-09-07T07:01:28.4959888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4960310Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4960735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4961170Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4961526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4961894Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4962331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.4962822Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.4963211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.4963565Z return self.act(input) 2025-09-07T07:01:28.4963676Z 2025-09-07T07:01:28.4963782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4964136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4964494Z return mod(**inputs) 2025-09-07T07:01:28.4964944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4965420Z outputs = self.model( 2025-09-07T07:01:28.4965865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4966336Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4966782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4967235Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4967581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4967939Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4968376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.4968827Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.4968963Z 2025-09-07T07:01:28.4969073Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4969432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4969746Z return mod(**inputs) 2025-09-07T07:01:28.4970165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4970601Z outputs = self.model( 2025-09-07T07:01:28.4971017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4971448Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4971882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4972349Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4972700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4973066Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4973504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4973960Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4974409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.4974924Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.4975137Z 2025-09-07T07:01:28.4975248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4975611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4975942Z return mod(**inputs) 2025-09-07T07:01:28.4976369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4976812Z outputs = self.model( 2025-09-07T07:01:28.4977245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4977680Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4978112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4978549Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4978928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4979282Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4979725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4980180Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4980640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.4981116Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.4981254Z 2025-09-07T07:01:28.4981361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4981731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4982070Z return mod(**inputs) 2025-09-07T07:01:28.4982500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4982951Z outputs = self.model( 2025-09-07T07:01:28.4983398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4983873Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4984339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4984824Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4985208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4985692Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4986218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4986740Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4987341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.4987793Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.4987934Z 2025-09-07T07:01:28.4988018Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4988236Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4988448Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4988660Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.4988890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4989255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4989581Z return mod(**inputs) 2025-09-07T07:01:28.4990000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4990440Z outputs = self.model( 2025-09-07T07:01:28.4990874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4991327Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4991775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4992247Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.4992619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.4993017Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.4993504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.4994030Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.4994531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.4995028Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.4995508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.4995999Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.4996213Z 2025-09-07T07:01:28.4996328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.4996700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.4997032Z return mod(**inputs) 2025-09-07T07:01:28.4997468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.4997924Z outputs = self.model( 2025-09-07T07:01:28.4998361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.4998810Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.4999263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.4999718Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5000086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5000460Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5000908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5001381Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5001888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5002365Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5002814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5003269Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5003440Z 2025-09-07T07:01:28.5003548Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5003913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5004241Z return mod(**inputs) 2025-09-07T07:01:28.5004667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5005107Z outputs = self.model( 2025-09-07T07:01:28.5005533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5005981Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5006417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5006862Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5007210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5007585Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5008026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5008483Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5008952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5009404Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5009548Z 2025-09-07T07:01:28.5009652Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5010013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5010337Z return mod(**inputs) 2025-09-07T07:01:28.5010746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5011205Z outputs = self.model( 2025-09-07T07:01:28.5011629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5012075Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5012518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5012955Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5013310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5013690Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5014138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5014630Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5014807Z 2025-09-07T07:01:28.5014915Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5015283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5015616Z return mod(**inputs) 2025-09-07T07:01:28.5016043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5016528Z outputs = self.model( 2025-09-07T07:01:28.5016948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5017386Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5017822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5018270Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5018618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5018981Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5019433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5020068Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5020483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5020846Z return self.act(input) 2025-09-07T07:01:28.5020973Z 2025-09-07T07:01:28.5021085Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5021477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5021843Z return mod(**inputs) 2025-09-07T07:01:28.5022287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5022760Z outputs = self.model( 2025-09-07T07:01:28.5023213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5023742Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5024213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5024688Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5025080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5025474Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5026013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.5026534Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5026685Z 2025-09-07T07:01:28.5026798Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5027191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5027545Z return mod(**inputs) 2025-09-07T07:01:28.5028002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5028477Z outputs = self.model( 2025-09-07T07:01:28.5028924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5029404Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5029877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5030353Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5030726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5031117Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5031593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5032244Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5032807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5033371Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5033604Z 2025-09-07T07:01:28.5033718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5034118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5034473Z return mod(**inputs) 2025-09-07T07:01:28.5034928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5035393Z outputs = self.model( 2025-09-07T07:01:28.5035848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5036326Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5036800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5037221Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5037551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5037902Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5038335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5038778Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5039238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5039674Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5039815Z 2025-09-07T07:01:28.5039916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5040267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5040589Z return mod(**inputs) 2025-09-07T07:01:28.5040994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5041450Z outputs = self.model( 2025-09-07T07:01:28.5041868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5042312Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5042733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5043153Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5043495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5043845Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5044267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5044706Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5045138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5045584Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5045732Z 2025-09-07T07:01:28.5045824Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5046035Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5046235Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5046477Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5046711Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5047062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5047377Z return mod(**inputs) 2025-09-07T07:01:28.5047780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5048206Z outputs = self.model( 2025-09-07T07:01:28.5048613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5049040Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5049453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5049877Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5050226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5050585Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5051025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5051470Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5051925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5052400Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5052834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5053320Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5053499Z 2025-09-07T07:01:28.5053605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5053961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5054286Z return mod(**inputs) 2025-09-07T07:01:28.5054701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5055160Z outputs = self.model( 2025-09-07T07:01:28.5055575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5056016Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5056453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5056894Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5057252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5057617Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5058064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5058526Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5058986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5059445Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5059898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5060363Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5060524Z 2025-09-07T07:01:28.5060676Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5061054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5061380Z return mod(**inputs) 2025-09-07T07:01:28.5061815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5062264Z outputs = self.model( 2025-09-07T07:01:28.5062693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5063147Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5063585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5064038Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5064404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5064780Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5065230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5065761Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5066234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5066696Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5066838Z 2025-09-07T07:01:28.5066954Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5067315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5067681Z return mod(**inputs) 2025-09-07T07:01:28.5068119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5068567Z outputs = self.model( 2025-09-07T07:01:28.5068997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5069440Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5069883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5070364Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5070725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5071093Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5071541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5072042Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5072229Z 2025-09-07T07:01:28.5072336Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5072703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5073034Z return mod(**inputs) 2025-09-07T07:01:28.5073460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5073916Z outputs = self.model( 2025-09-07T07:01:28.5074345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5074799Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5075242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5075737Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5076106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5076473Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5076926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5077416Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5077820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5078169Z return self.act(input) 2025-09-07T07:01:28.5078281Z 2025-09-07T07:01:28.5078395Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5078759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5079081Z return mod(**inputs) 2025-09-07T07:01:28.5079509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5079949Z outputs = self.model( 2025-09-07T07:01:28.5080373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5080811Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5081258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5081693Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5082044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5082418Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5082840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.5083275Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5083416Z 2025-09-07T07:01:28.5083517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5083868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5084208Z return mod(**inputs) 2025-09-07T07:01:28.5084615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5085050Z outputs = self.model( 2025-09-07T07:01:28.5085471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5085926Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5086370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5086815Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5087177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5087548Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5087997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5088451Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5091771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5092337Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5092542Z 2025-09-07T07:01:28.5092658Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5093044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5093370Z return mod(**inputs) 2025-09-07T07:01:28.5093784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5094219Z outputs = self.model( 2025-09-07T07:01:28.5094635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5095062Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5095527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5095965Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5096308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5096666Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5097108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5097567Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5098022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5098465Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5098610Z 2025-09-07T07:01:28.5098717Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5099081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5099431Z return mod(**inputs) 2025-09-07T07:01:28.5099854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5100287Z outputs = self.model( 2025-09-07T07:01:28.5100714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5101164Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5101611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5102076Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5102429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5102802Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5103256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5103724Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5104184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5104646Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5104806Z 2025-09-07T07:01:28.5104896Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5105127Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5105357Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5105659Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5105942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5106350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5106804Z return mod(**inputs) 2025-09-07T07:01:28.5107260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5107758Z outputs = self.model( 2025-09-07T07:01:28.5108216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5108655Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5109110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5109556Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5109913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5110284Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5110738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5111203Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5111661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5112136Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5112579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5113056Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5113236Z 2025-09-07T07:01:28.5113347Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5113695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5114019Z return mod(**inputs) 2025-09-07T07:01:28.5114461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5114897Z outputs = self.model( 2025-09-07T07:01:28.5115310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5115762Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5116207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5116676Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5117035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5117404Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5117854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5118320Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5118782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5119255Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5119830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5120317Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5120498Z 2025-09-07T07:01:28.5120606Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5120984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5121326Z return mod(**inputs) 2025-09-07T07:01:28.5121838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5122325Z outputs = self.model( 2025-09-07T07:01:28.5122792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5123256Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5123700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5124218Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5124585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5124972Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5125430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5125898Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5126373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5126856Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5127002Z 2025-09-07T07:01:28.5127117Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5127486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5127815Z return mod(**inputs) 2025-09-07T07:01:28.5128247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5128696Z outputs = self.model( 2025-09-07T07:01:28.5129122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5129627Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5130065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5130509Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5130874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5131235Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5131665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5132188Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5132368Z 2025-09-07T07:01:28.5132470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5132827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5133156Z return mod(**inputs) 2025-09-07T07:01:28.5133570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5134007Z outputs = self.model( 2025-09-07T07:01:28.5134426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5134874Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5135319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5135766Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5136127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5136525Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5136988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5137489Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5137877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5138211Z return self.act(input) 2025-09-07T07:01:28.5138329Z 2025-09-07T07:01:28.5138434Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5138790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5139111Z return mod(**inputs) 2025-09-07T07:01:28.5139528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5139958Z outputs = self.model( 2025-09-07T07:01:28.5140384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5140836Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5141271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5141722Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5142092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5142484Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5142976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.5143461Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5143618Z 2025-09-07T07:01:28.5143732Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5144146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5144497Z return mod(**inputs) 2025-09-07T07:01:28.5144940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5145422Z outputs = self.model( 2025-09-07T07:01:28.5145939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5146460Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5146948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5147421Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5147796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5148173Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5148650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5149147Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5149641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5150213Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5150447Z 2025-09-07T07:01:28.5150562Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5150952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5151299Z return mod(**inputs) 2025-09-07T07:01:28.5151763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5152237Z outputs = self.model( 2025-09-07T07:01:28.5152709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5153192Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5153667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5154136Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5154515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5154920Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5155409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5155899Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5156406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5156892Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5157037Z 2025-09-07T07:01:28.5157157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5157546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5157894Z return mod(**inputs) 2025-09-07T07:01:28.5158353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5158804Z outputs = self.model( 2025-09-07T07:01:28.5159235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5159706Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5160143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5160589Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5160944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5161315Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5161794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5162312Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5162811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5162913Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5162918Z 2025-09-07T07:01:28.5163008Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5163094Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5163187Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5163279Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5163392Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5163598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5163666Z return mod(**inputs) 2025-09-07T07:01:28.5163988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5164059Z outputs = self.model( 2025-09-07T07:01:28.5164390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5164490Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5164817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5164900Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5165128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5165216Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5165527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5165623Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5165942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5166045Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5166350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5166487Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5166491Z 2025-09-07T07:01:28.5166601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5166813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5166886Z return mod(**inputs) 2025-09-07T07:01:28.5167203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5167273Z outputs = self.model( 2025-09-07T07:01:28.5167593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5167668Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5168010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5168086Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5168322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5168403Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5168713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5168838Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5169149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5169255Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5169562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5169679Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5169692Z 2025-09-07T07:01:28.5169796Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5169998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5170073Z return mod(**inputs) 2025-09-07T07:01:28.5170388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5170469Z outputs = self.model( 2025-09-07T07:01:28.5170785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5170858Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5171196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5171272Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5171533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5171616Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5171924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-09-07T07:01:28.5172024Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:01:28.5172334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5172426Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5172429Z 2025-09-07T07:01:28.5172534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5172742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5172808Z return mod(**inputs) 2025-09-07T07:01:28.5173124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5173202Z outputs = self.model( 2025-09-07T07:01:28.5173513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5173592Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5173901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5173981Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5174207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5174312Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5174630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5174751Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5174755Z 2025-09-07T07:01:28.5174867Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5175067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5175151Z return mod(**inputs) 2025-09-07T07:01:28.5175482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5175552Z outputs = self.model( 2025-09-07T07:01:28.5175877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5175953Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5176279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5176352Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5176580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5176668Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5176987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-09-07T07:01:28.5177113Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5177326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5177412Z return self.act(input) 2025-09-07T07:01:28.5177417Z 2025-09-07T07:01:28.5177528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5177738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5177812Z return mod(**inputs) 2025-09-07T07:01:28.5178117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5178184Z outputs = self.model( 2025-09-07T07:01:28.5178496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-09-07T07:01:28.5178569Z encoder_outputs = self.encoder( 2025-09-07T07:01:28.5178879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-09-07T07:01:28.5178950Z layer_outputs = encoder_layer( 2025-09-07T07:01:28.5179176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5179254Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5179558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-09-07T07:01:28.5179648Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5179652Z 2025-09-07T07:01:28.5179752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5179955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5180021Z return mod(**inputs) 2025-09-07T07:01:28.5180327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5180403Z outputs = self.model( 2025-09-07T07:01:28.5180718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5180827Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5181143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5181221Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5181445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5181546Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5181862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5181966Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5182297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5182462Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5182468Z 2025-09-07T07:01:28.5182586Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5182798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5182867Z return mod(**inputs) 2025-09-07T07:01:28.5183204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5183277Z outputs = self.model( 2025-09-07T07:01:28.5183623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5183699Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5184056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5184145Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5184423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5184516Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5184852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5184970Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5185317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5185404Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5185408Z 2025-09-07T07:01:28.5185529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5185828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5185911Z return mod(**inputs) 2025-09-07T07:01:28.5186245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5186318Z outputs = self.model( 2025-09-07T07:01:28.5186654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5186733Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5187088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5187165Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5187417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5187522Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5187827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5187936Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5188235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5188329Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5188352Z 2025-09-07T07:01:28.5188435Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5188518Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5188606Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5188682Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5188794Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5188998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5189066Z return mod(**inputs) 2025-09-07T07:01:28.5189390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5189457Z outputs = self.model( 2025-09-07T07:01:28.5189777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5189851Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5190176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5190248Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5190494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5190583Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5190909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5191018Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5191325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5191425Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5191730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5191866Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5191869Z 2025-09-07T07:01:28.5191977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5192190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5192262Z return mod(**inputs) 2025-09-07T07:01:28.5192570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5192635Z outputs = self.model( 2025-09-07T07:01:28.5192950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5193022Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5193342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5193416Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5193640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5193728Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5194066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5194174Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5194482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5194589Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5194883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5195021Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5195024Z 2025-09-07T07:01:28.5195135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5195342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5195420Z return mod(**inputs) 2025-09-07T07:01:28.5195736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5195804Z outputs = self.model( 2025-09-07T07:01:28.5196126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5196200Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5196526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5196601Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5196839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5196920Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5197269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5197397Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5197720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5197812Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5197815Z 2025-09-07T07:01:28.5197919Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5198129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5198204Z return mod(**inputs) 2025-09-07T07:01:28.5198529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5198606Z outputs = self.model( 2025-09-07T07:01:28.5198930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5199012Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5199332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5199405Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5199643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5199726Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5200055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5200167Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5200487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5200669Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5200673Z 2025-09-07T07:01:28.5200777Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5200989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5201055Z return mod(**inputs) 2025-09-07T07:01:28.5201380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5201466Z outputs = self.model( 2025-09-07T07:01:28.5201780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5201862Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5202177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5202257Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5202485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5202565Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5202882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5202994Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5203311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5203392Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5203395Z 2025-09-07T07:01:28.5203521Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5203727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5203793Z return mod(**inputs) 2025-09-07T07:01:28.5204128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5204200Z outputs = self.model( 2025-09-07T07:01:28.5204516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5204590Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5204901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5204981Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5205206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5205293Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5205602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5205718Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5206083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5206165Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5206170Z 2025-09-07T07:01:28.5206252Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5206327Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5206409Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5206481Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5206579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5206802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5206864Z return mod(**inputs) 2025-09-07T07:01:28.5207170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5207233Z outputs = self.model( 2025-09-07T07:01:28.5207530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5207624Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5207930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5208003Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5208217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5208301Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5208598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5208701Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5209004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5209096Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5209383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5209510Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5209514Z 2025-09-07T07:01:28.5209618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5209831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5209896Z return mod(**inputs) 2025-09-07T07:01:28.5210217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5210284Z outputs = self.model( 2025-09-07T07:01:28.5210590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5210659Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5210959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5211034Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5211248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5211333Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5211631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5211742Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5212040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5212137Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5212432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5212542Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5212546Z 2025-09-07T07:01:28.5212653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5212854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5212945Z return mod(**inputs) 2025-09-07T07:01:28.5213267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5213333Z outputs = self.model( 2025-09-07T07:01:28.5213638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5213707Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5214011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5214101Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5214316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5214402Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5214700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5214812Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5215107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5215186Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5215196Z 2025-09-07T07:01:28.5215294Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5215486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5215558Z return mod(**inputs) 2025-09-07T07:01:28.5215875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5215950Z outputs = self.model( 2025-09-07T07:01:28.5216263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5216332Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5216635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5216704Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5216924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5217001Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5217298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5217425Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5217429Z 2025-09-07T07:01:28.5217528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5217727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5217790Z return mod(**inputs) 2025-09-07T07:01:28.5218094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5218158Z outputs = self.model( 2025-09-07T07:01:28.5218456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5218534Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5218829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5218905Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5219147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5219223Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5219528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5219855Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5220081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5220934Z return self.act(input) 2025-09-07T07:01:28.5220938Z 2025-09-07T07:01:28.5221049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5221252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5221318Z return mod(**inputs) 2025-09-07T07:01:28.5221645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5221715Z outputs = self.model( 2025-09-07T07:01:28.5222041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5222114Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5222427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5222507Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5222737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5222827Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5223172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5223266Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5223270Z 2025-09-07T07:01:28.5223373Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5223608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5223684Z return mod(**inputs) 2025-09-07T07:01:28.5223995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5224069Z outputs = self.model( 2025-09-07T07:01:28.5224382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5224453Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5224785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5224864Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5225110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5225194Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5225523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5225667Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5225993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5226168Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5226172Z 2025-09-07T07:01:28.5226280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5226499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5226604Z return mod(**inputs) 2025-09-07T07:01:28.5226931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5227004Z outputs = self.model( 2025-09-07T07:01:28.5227322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5227402Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5227729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5227829Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5228054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5228136Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5228458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5228560Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5228882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5228965Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5228968Z 2025-09-07T07:01:28.5229072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5229282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5229349Z return mod(**inputs) 2025-09-07T07:01:28.5229688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5229761Z outputs = self.model( 2025-09-07T07:01:28.5230095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5230170Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5230483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5230564Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5230790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5230878Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5231191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5231290Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5231612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5231704Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5231708Z 2025-09-07T07:01:28.5231800Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5231880Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5231965Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5232043Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5232147Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5232359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5232436Z return mod(**inputs) 2025-09-07T07:01:28.5232754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5232824Z outputs = self.model( 2025-09-07T07:01:28.5233160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5233243Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5233555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5233636Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5233860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5233999Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5234320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5234419Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5234738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5234839Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5235143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5235278Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5235282Z 2025-09-07T07:01:28.5235383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5235593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5235661Z return mod(**inputs) 2025-09-07T07:01:28.5235979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5236048Z outputs = self.model( 2025-09-07T07:01:28.5236379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5236463Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5236801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5236894Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5237120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5237208Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5237551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5237648Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5237962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5238058Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5238353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5238461Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5238464Z 2025-09-07T07:01:28.5238572Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5238769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5238836Z return mod(**inputs) 2025-09-07T07:01:28.5239149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5239215Z outputs = self.model( 2025-09-07T07:01:28.5239533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5239626Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5239934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5240014Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5240237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5240322Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5240644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5240738Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5241122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5241206Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5241209Z 2025-09-07T07:01:28.5241319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5241515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5241641Z return mod(**inputs) 2025-09-07T07:01:28.5241951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5242015Z outputs = self.model( 2025-09-07T07:01:28.5242330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5242401Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5242759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5242835Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5243089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5243175Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5243499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5243616Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5243931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5244093Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5244097Z 2025-09-07T07:01:28.5244199Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5244401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5244477Z return mod(**inputs) 2025-09-07T07:01:28.5244792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5244868Z outputs = self.model( 2025-09-07T07:01:28.5245184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5245264Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5245586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5245676Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5245906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5246003Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5246315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5263355Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5263871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5263984Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5263993Z 2025-09-07T07:01:28.5264242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5264474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5264563Z return mod(**inputs) 2025-09-07T07:01:28.5264936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5265036Z outputs = self.model( 2025-09-07T07:01:28.5265387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5265479Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5265913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5266000Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5266261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5266355Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5266707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5266876Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5267226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5267378Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5267384Z 2025-09-07T07:01:28.5267470Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5267557Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5267635Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5267710Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5267828Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5268037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5268115Z return mod(**inputs) 2025-09-07T07:01:28.5268426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5268499Z outputs = self.model( 2025-09-07T07:01:28.5268814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5268888Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5269199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5269271Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5269500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5269583Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5269887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5270005Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5270339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5270453Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5270751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5270894Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5270899Z 2025-09-07T07:01:28.5271017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5271241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5271308Z return mod(**inputs) 2025-09-07T07:01:28.5271631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5271703Z outputs = self.model( 2025-09-07T07:01:28.5272027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5272104Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5272429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5272500Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5272721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5272811Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5273113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5273227Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5273551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5273654Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5273987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5274100Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5274104Z 2025-09-07T07:01:28.5274216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5274423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5274497Z return mod(**inputs) 2025-09-07T07:01:28.5274819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5274884Z outputs = self.model( 2025-09-07T07:01:28.5275200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5275275Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5275586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5275656Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5275877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5275965Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5276263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5276377Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5276681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5276789Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5276793Z 2025-09-07T07:01:28.5276896Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5277094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5277169Z return mod(**inputs) 2025-09-07T07:01:28.5277471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5277561Z outputs = self.model( 2025-09-07T07:01:28.5277872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5277945Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5278259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5278332Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5278560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5278638Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5278949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5279070Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5279076Z 2025-09-07T07:01:28.5279178Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5279384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5279448Z return mod(**inputs) 2025-09-07T07:01:28.5279780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5279849Z outputs = self.model( 2025-09-07T07:01:28.5280173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5280254Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5280568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5280647Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5280875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5280962Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5281274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5281396Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5281623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5281695Z return self.act(input) 2025-09-07T07:01:28.5281699Z 2025-09-07T07:01:28.5281810Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5282013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5282080Z return mod(**inputs) 2025-09-07T07:01:28.5282400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5282469Z outputs = self.model( 2025-09-07T07:01:28.5282787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5282861Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5283196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5283270Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5283494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5283581Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5283890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5283998Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5284002Z 2025-09-07T07:01:28.5284104Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5284308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5284382Z return mod(**inputs) 2025-09-07T07:01:28.5284693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5284770Z outputs = self.model( 2025-09-07T07:01:28.5285082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5285160Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5285470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5285543Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5285775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5285853Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5286186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5286293Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5286621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5286787Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5286790Z 2025-09-07T07:01:28.5286894Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5287103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5287172Z return mod(**inputs) 2025-09-07T07:01:28.5287493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5287562Z outputs = self.model( 2025-09-07T07:01:28.5287875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5287956Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5288270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5288350Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5288574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5288656Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5288973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5289074Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5289392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5289494Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5289497Z 2025-09-07T07:01:28.5289610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5289813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5289879Z return mod(**inputs) 2025-09-07T07:01:28.5290198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5290288Z outputs = self.model( 2025-09-07T07:01:28.5290605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5290678Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5290994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5291077Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5291302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5291388Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5291697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5291805Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5292118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5292207Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5292211Z 2025-09-07T07:01:28.5292300Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5292397Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5292487Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5292566Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5292692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5292901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5292976Z return mod(**inputs) 2025-09-07T07:01:28.5293285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5293355Z outputs = self.model( 2025-09-07T07:01:28.5293660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5293736Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5294041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5294121Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5294342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5294425Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5294727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5294824Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5295139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5295237Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5295544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5295695Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5295699Z 2025-09-07T07:01:28.5295807Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5296004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5296069Z return mod(**inputs) 2025-09-07T07:01:28.5296378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5296444Z outputs = self.model( 2025-09-07T07:01:28.5296778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5296850Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5297156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5297238Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5297459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5297545Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5297843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5297944Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5298252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5298348Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5298645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5298778Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5298783Z 2025-09-07T07:01:28.5298892Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5299103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5299170Z return mod(**inputs) 2025-09-07T07:01:28.5299485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5299552Z outputs = self.model( 2025-09-07T07:01:28.5299872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5299944Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5300257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5300330Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5300556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5300644Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5300971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5301073Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5301383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5301466Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5301469Z 2025-09-07T07:01:28.5301577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5301776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5301869Z return mod(**inputs) 2025-09-07T07:01:28.5302180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5302254Z outputs = self.model( 2025-09-07T07:01:28.5302567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5302638Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5302959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5303045Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5303274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5303351Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5303656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5303773Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5304080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5304241Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5304245Z 2025-09-07T07:01:28.5304350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5304561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5304631Z return mod(**inputs) 2025-09-07T07:01:28.5304964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5305062Z outputs = self.model( 2025-09-07T07:01:28.5305403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5305504Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5306100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5306188Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5306456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5306545Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5306893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5307013Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5307366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5307458Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5307465Z 2025-09-07T07:01:28.5307579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5307808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5307881Z return mod(**inputs) 2025-09-07T07:01:28.5308218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5308287Z outputs = self.model( 2025-09-07T07:01:28.5308591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5308670Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5308978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5309078Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5309300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5309382Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5309685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5309790Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5310114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5310200Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5310204Z 2025-09-07T07:01:28.5310291Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5310370Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5310446Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5310527Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5310629Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5310829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5310893Z return mod(**inputs) 2025-09-07T07:01:28.5311194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5311269Z outputs = self.model( 2025-09-07T07:01:28.5311569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5311649Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5311967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5312047Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5312288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5312368Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5312678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5312784Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5313095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5313191Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5313483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5313624Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5313628Z 2025-09-07T07:01:28.5313729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5313934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5313998Z return mod(**inputs) 2025-09-07T07:01:28.5314311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5314381Z outputs = self.model( 2025-09-07T07:01:28.5314685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5314765Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5315072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5315170Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5315391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5315470Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5315779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5315886Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5316214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5316309Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5316604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5316711Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5316714Z 2025-09-07T07:01:28.5316814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5317020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5317084Z return mod(**inputs) 2025-09-07T07:01:28.5317392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5317460Z outputs = self.model( 2025-09-07T07:01:28.5317772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5317842Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5318161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5318243Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5318480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5318565Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5318863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5318967Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5319274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5319357Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5319360Z 2025-09-07T07:01:28.5319467Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5319879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5319961Z return mod(**inputs) 2025-09-07T07:01:28.5320268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5320337Z outputs = self.model( 2025-09-07T07:01:28.5320649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5320733Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5321045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5321127Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5321360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5321441Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5321813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5321935Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5321939Z 2025-09-07T07:01:28.5322050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5322248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5322322Z return mod(**inputs) 2025-09-07T07:01:28.5322636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5322747Z outputs = self.model( 2025-09-07T07:01:28.5323075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5323150Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5323479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5323555Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5323788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5323877Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5324192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5324320Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5324543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5324619Z return self.act(input) 2025-09-07T07:01:28.5324622Z 2025-09-07T07:01:28.5324756Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5324959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5325033Z return mod(**inputs) 2025-09-07T07:01:28.5325372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5325449Z outputs = self.model( 2025-09-07T07:01:28.5325760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5325833Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5326152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5326224Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5326457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5326538Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5326859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5326942Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5326946Z 2025-09-07T07:01:28.5327048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5327259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5327328Z return mod(**inputs) 2025-09-07T07:01:28.5327646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5327713Z outputs = self.model( 2025-09-07T07:01:28.5328024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5328124Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5328436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5328514Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5328737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5328823Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5329132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5329292Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5329614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5329767Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5329771Z 2025-09-07T07:01:28.5329881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5330083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5330149Z return mod(**inputs) 2025-09-07T07:01:28.5330469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5330537Z outputs = self.model( 2025-09-07T07:01:28.5330859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5330931Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5331267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5331344Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5331582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5331670Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5331981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5332089Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5332403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5332487Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5332499Z 2025-09-07T07:01:28.5332604Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5332808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5332882Z return mod(**inputs) 2025-09-07T07:01:28.5333199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5333276Z outputs = self.model( 2025-09-07T07:01:28.5333586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5333659Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5333988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5334062Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5334287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5334368Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5334687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5334797Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5335109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5335206Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5335210Z 2025-09-07T07:01:28.5335291Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5335396Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5335475Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5335552Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5335664Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5335865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5335939Z return mod(**inputs) 2025-09-07T07:01:28.5336256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5336324Z outputs = self.model( 2025-09-07T07:01:28.5336645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5336716Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5337045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5337116Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5337334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5337417Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5337742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5337863Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5338166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5338269Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5338556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5338690Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5338694Z 2025-09-07T07:01:28.5338803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5339003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5339077Z return mod(**inputs) 2025-09-07T07:01:28.5339386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5339455Z outputs = self.model( 2025-09-07T07:01:28.5339773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5339845Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5340169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5340242Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5340471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5340548Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5340860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5341016Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5341329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5341434Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5341729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5341857Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5341869Z 2025-09-07T07:01:28.5341971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5342175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5342248Z return mod(**inputs) 2025-09-07T07:01:28.5342560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5342636Z outputs = self.model( 2025-09-07T07:01:28.5342948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5343020Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5343338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5343411Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5343650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5343729Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5344070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5344179Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5344541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5344638Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5344642Z 2025-09-07T07:01:28.5344754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5344973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5345045Z return mod(**inputs) 2025-09-07T07:01:28.5345374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5345454Z outputs = self.model( 2025-09-07T07:01:28.5345868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5345965Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5346306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5346391Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5346638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5346724Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5347072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5347182Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5347496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5347676Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5347680Z 2025-09-07T07:01:28.5347781Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5347986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5348052Z return mod(**inputs) 2025-09-07T07:01:28.5348374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5348459Z outputs = self.model( 2025-09-07T07:01:28.5348821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5348892Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5349194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5349273Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5349493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5349578Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5349880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5349986Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5350298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5350378Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5350381Z 2025-09-07T07:01:28.5350490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5350702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5350779Z return mod(**inputs) 2025-09-07T07:01:28.5351107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5351174Z outputs = self.model( 2025-09-07T07:01:28.5351478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5351547Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5351857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5351928Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5352150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5352238Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5352550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5352664Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5352972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5353064Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5353068Z 2025-09-07T07:01:28.5353149Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5353230Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5353317Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5353393Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5353512Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5353706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5353787Z return mod(**inputs) 2025-09-07T07:01:28.5354102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5354168Z outputs = self.model( 2025-09-07T07:01:28.5354476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5354546Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5354847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5354940Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5355152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5355235Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5355529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5355641Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5355936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5356032Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5356319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5356449Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5356452Z 2025-09-07T07:01:28.5356557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5356767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5356834Z return mod(**inputs) 2025-09-07T07:01:28.5357151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5357220Z outputs = self.model( 2025-09-07T07:01:28.5357537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5357605Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5357911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5357981Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5358195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5358277Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5358579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5358695Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5358996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5359089Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5359381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5359485Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5359489Z 2025-09-07T07:01:28.5359592Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5359784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5359854Z return mod(**inputs) 2025-09-07T07:01:28.5360193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5360260Z outputs = self.model( 2025-09-07T07:01:28.5360561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5360628Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5360926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5361010Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5361224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5361306Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5361603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5361714Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5362008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5362092Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5362095Z 2025-09-07T07:01:28.5362194Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5362387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5362459Z return mod(**inputs) 2025-09-07T07:01:28.5362756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5362829Z outputs = self.model( 2025-09-07T07:01:28.5363151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5363226Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5363543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5363613Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5363834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5363909Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5364212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5364327Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5364330Z 2025-09-07T07:01:28.5364431Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5364630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5364693Z return mod(**inputs) 2025-09-07T07:01:28.5364996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5365061Z outputs = self.model( 2025-09-07T07:01:28.5365357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5365436Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5365739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5365817Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5366033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5366130Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5366424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5366537Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5366750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5366818Z return self.act(input) 2025-09-07T07:01:28.5366822Z 2025-09-07T07:01:28.5366946Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5367138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5367201Z return mod(**inputs) 2025-09-07T07:01:28.5367509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5367576Z outputs = self.model( 2025-09-07T07:01:28.5367879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5367948Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5368250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5368318Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5368534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5368620Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5368916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5369023Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5369028Z 2025-09-07T07:01:28.5369125Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5369340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5369414Z return mod(**inputs) 2025-09-07T07:01:28.5369712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5369785Z outputs = self.model( 2025-09-07T07:01:28.5370079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5370154Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5370450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5370518Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5370741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5370816Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5371119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5371215Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5371507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5371662Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5371666Z 2025-09-07T07:01:28.5371763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5371964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5372029Z return mod(**inputs) 2025-09-07T07:01:28.5372350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5372416Z outputs = self.model( 2025-09-07T07:01:28.5372710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5372786Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5373081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5373175Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5373387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5373462Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5373773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5373871Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5374183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5374261Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5374265Z 2025-09-07T07:01:28.5374371Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5374565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5374630Z return mod(**inputs) 2025-09-07T07:01:28.5374939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5375004Z outputs = self.model( 2025-09-07T07:01:28.5375357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5375428Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5375751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5375820Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5376032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5376113Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5376410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5376514Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5376818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5376905Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5376908Z 2025-09-07T07:01:28.5376995Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5377074Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5377164Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5377248Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5377349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5377553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5377621Z return mod(**inputs) 2025-09-07T07:01:28.5377937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5378012Z outputs = self.model( 2025-09-07T07:01:28.5378315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5378411Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5378720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5378796Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5379019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5379096Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5379425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5379524Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5379837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5379935Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5380227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5380366Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5380370Z 2025-09-07T07:01:28.5380472Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5380678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5380747Z return mod(**inputs) 2025-09-07T07:01:28.5381067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5381137Z outputs = self.model( 2025-09-07T07:01:28.5381471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5381556Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5381882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5381963Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5382190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5382269Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5382597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5382697Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5383015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5383114Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5383417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5383528Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5383532Z 2025-09-07T07:01:28.5383634Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5383844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5383912Z return mod(**inputs) 2025-09-07T07:01:28.5384229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5384296Z outputs = self.model( 2025-09-07T07:01:28.5384611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5384708Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5385033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5385117Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5385354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5385445Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5385851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5385985Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5386328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5386416Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5386421Z 2025-09-07T07:01:28.5386539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5386754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5386828Z return mod(**inputs) 2025-09-07T07:01:28.5387180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5387250Z outputs = self.model( 2025-09-07T07:01:28.5387565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5387643Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5387960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5388051Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5388274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5388377Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5388677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5388793Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5389097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5389257Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5389260Z 2025-09-07T07:01:28.5389361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5389559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5389632Z return mod(**inputs) 2025-09-07T07:01:28.5389942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5390015Z outputs = self.model( 2025-09-07T07:01:28.5390324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5390393Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5390709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5390782Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5391006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5391084Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5391393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5391520Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5391820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5391907Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5391910Z 2025-09-07T07:01:28.5392011Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5392212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5392299Z return mod(**inputs) 2025-09-07T07:01:28.5392614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5392688Z outputs = self.model( 2025-09-07T07:01:28.5393002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5393078Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5393376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5393450Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5393665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5393743Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5394044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5394145Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5394483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5394572Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5394575Z 2025-09-07T07:01:28.5394669Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5394758Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5394835Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5394918Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5395018Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5395222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5395295Z return mod(**inputs) 2025-09-07T07:01:28.5395594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5395668Z outputs = self.model( 2025-09-07T07:01:28.5395966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5396044Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5396344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5396413Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5396635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5396709Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5397021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5397125Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5397423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5397541Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5397822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5397958Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5397961Z 2025-09-07T07:01:28.5398059Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5398252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5398332Z return mod(**inputs) 2025-09-07T07:01:28.5398629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5398702Z outputs = self.model( 2025-09-07T07:01:28.5399002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5399081Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5399380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5399448Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5399673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5399748Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5400050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5400154Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5400483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5400581Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5400883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5401000Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5401004Z 2025-09-07T07:01:28.5401102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5401304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5401371Z return mod(**inputs) 2025-09-07T07:01:28.5401675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5401750Z outputs = self.model( 2025-09-07T07:01:28.5402074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5402153Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5402455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5402530Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5402746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5402821Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5403132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5403238Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5403542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5403640Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5403644Z 2025-09-07T07:01:28.5403753Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5403948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5404016Z return mod(**inputs) 2025-09-07T07:01:28.5404322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5404392Z outputs = self.model( 2025-09-07T07:01:28.5404713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5404786Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5405097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5405179Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5405406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5405500Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5405829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5405958Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5405971Z 2025-09-07T07:01:28.5406079Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5406299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5406393Z return mod(**inputs) 2025-09-07T07:01:28.5406749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5406831Z outputs = self.model( 2025-09-07T07:01:28.5407188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5407267Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5407625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5407703Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5407958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5408046Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5408387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5408528Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5408766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5408850Z return self.act(input) 2025-09-07T07:01:28.5408856Z 2025-09-07T07:01:28.5408968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5409194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5409266Z return mod(**inputs) 2025-09-07T07:01:28.5409609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5409694Z outputs = self.model( 2025-09-07T07:01:28.5410034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5410120Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5410469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5410572Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5410827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5410913Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5411262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5411352Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5411373Z 2025-09-07T07:01:28.5411494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5411716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5411787Z return mod(**inputs) 2025-09-07T07:01:28.5412138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5412214Z outputs = self.model( 2025-09-07T07:01:28.5412567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5412657Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5412989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5413072Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5413312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5413401Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5413749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5413869Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5414227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5414394Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5414399Z 2025-09-07T07:01:28.5414520Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5414738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5414813Z return mod(**inputs) 2025-09-07T07:01:28.5415151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5415225Z outputs = self.model( 2025-09-07T07:01:28.5415573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5415652Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5416002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5416079Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5416330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5416416Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5416751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5416870Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5417226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5417340Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5417344Z 2025-09-07T07:01:28.5417456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5417683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5417754Z return mod(**inputs) 2025-09-07T07:01:28.5418095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5418175Z outputs = self.model( 2025-09-07T07:01:28.5418513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5418629Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5418974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5419054Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5419307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5419393Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5419926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5420041Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5420381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5420489Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5420493Z 2025-09-07T07:01:28.5420581Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5420677Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5420805Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5420903Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5421018Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5421264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5421345Z return mod(**inputs) 2025-09-07T07:01:28.5421695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5421777Z outputs = self.model( 2025-09-07T07:01:28.5422115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5422196Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5422550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5422631Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5422889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5422979Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5423321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5423441Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5423786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5423903Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5424226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5424383Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5424414Z 2025-09-07T07:01:28.5424530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5424751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5424831Z return mod(**inputs) 2025-09-07T07:01:28.5425171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5425252Z outputs = self.model( 2025-09-07T07:01:28.5425644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5425776Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5426121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5426200Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5426455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5426539Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5426884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5426990Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5427329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5427442Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5427764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5427895Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5427921Z 2025-09-07T07:01:28.5428039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5428267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5428358Z return mod(**inputs) 2025-09-07T07:01:28.5428701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5428785Z outputs = self.model( 2025-09-07T07:01:28.5429125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5429213Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5429556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5429634Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5429889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5429977Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5430337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5430445Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5430792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5430883Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5430886Z 2025-09-07T07:01:28.5430998Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5431225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5431298Z return mod(**inputs) 2025-09-07T07:01:28.5431646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5431745Z outputs = self.model( 2025-09-07T07:01:28.5432077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5432157Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5432461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5432558Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5432788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5432869Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5433171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5433276Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5433584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5433726Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5433730Z 2025-09-07T07:01:28.5433837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5434038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5434104Z return mod(**inputs) 2025-09-07T07:01:28.5434422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5434490Z outputs = self.model( 2025-09-07T07:01:28.5434827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5434900Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5435226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5435297Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5435517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5435601Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5435906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5436019Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5436345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5436422Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5436431Z 2025-09-07T07:01:28.5436529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5436721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5436792Z return mod(**inputs) 2025-09-07T07:01:28.5437090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5437163Z outputs = self.model( 2025-09-07T07:01:28.5437462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5437532Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5437845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5437937Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5438161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5438237Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5438538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5438649Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5438950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5439059Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5439062Z 2025-09-07T07:01:28.5439139Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5439236Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5439312Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5439386Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5439493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5439684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5439754Z return mod(**inputs) 2025-09-07T07:01:28.5440049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5440113Z outputs = self.model( 2025-09-07T07:01:28.5440416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5440484Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5440804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5440877Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5441093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5441225Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5441528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5441641Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5441945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5442050Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5442339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5442471Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5442475Z 2025-09-07T07:01:28.5442583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5442782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5442862Z return mod(**inputs) 2025-09-07T07:01:28.5443158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5443226Z outputs = self.model( 2025-09-07T07:01:28.5443527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5443606Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5443912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5443984Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5444230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5444307Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5444600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5444711Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5445012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5445130Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5445413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5445526Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5445531Z 2025-09-07T07:01:28.5445630Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5445823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5445895Z return mod(**inputs) 2025-09-07T07:01:28.5446195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5446268Z outputs = self.model( 2025-09-07T07:01:28.5446569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5446641Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5446948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5447036Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5447266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5447345Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5447671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5447777Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5448079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5448171Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5448175Z 2025-09-07T07:01:28.5448276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5448486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5448551Z return mod(**inputs) 2025-09-07T07:01:28.5448846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5448921Z outputs = self.model( 2025-09-07T07:01:28.5449217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5449293Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5449592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5449670Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5449887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5449963Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5450276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5450410Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5450414Z 2025-09-07T07:01:28.5450522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5450717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5450782Z return mod(**inputs) 2025-09-07T07:01:28.5451087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5451170Z outputs = self.model( 2025-09-07T07:01:28.5451488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5451558Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5451876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5451948Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5452171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5452255Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5452561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5452685Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5452905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5452975Z return self.act(input) 2025-09-07T07:01:28.5452978Z 2025-09-07T07:01:28.5453086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5453301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5453376Z return mod(**inputs) 2025-09-07T07:01:28.5453691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5453766Z outputs = self.model( 2025-09-07T07:01:28.5454076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5454148Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5454458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5454529Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5454753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5454831Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5455139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5455231Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5455235Z 2025-09-07T07:01:28.5455337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5455540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5455605Z return mod(**inputs) 2025-09-07T07:01:28.5455917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5455986Z outputs = self.model( 2025-09-07T07:01:28.5456287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5456368Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5456689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5456768Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5456987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5457064Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5457375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5457493Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5457801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5457951Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5457956Z 2025-09-07T07:01:28.5458065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5458262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5458326Z return mod(**inputs) 2025-09-07T07:01:28.5458635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5458702Z outputs = self.model( 2025-09-07T07:01:28.5459015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5459087Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5459390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5459487Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5459707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5459790Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5460106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5460213Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5460522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5460604Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5460608Z 2025-09-07T07:01:28.5460720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5460924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5461002Z return mod(**inputs) 2025-09-07T07:01:28.5461334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5461407Z outputs = self.model( 2025-09-07T07:01:28.5461775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5461853Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5462213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5462293Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5462543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5462629Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5462968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5463101Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5463442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5463543Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5463547Z 2025-09-07T07:01:28.5463644Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5463730Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5463821Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5463920Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5464040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5464251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5464320Z return mod(**inputs) 2025-09-07T07:01:28.5464658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5464732Z outputs = self.model( 2025-09-07T07:01:28.5465070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5465148Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5465489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5465569Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5465889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5465987Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5466354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5466480Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5466851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5466970Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5467294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5467436Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5467442Z 2025-09-07T07:01:28.5467561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5467772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5467849Z return mod(**inputs) 2025-09-07T07:01:28.5468183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5468258Z outputs = self.model( 2025-09-07T07:01:28.5468600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5468675Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5469009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5469085Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5469323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5469414Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5469755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5469893Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5470244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5470355Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5470668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5470785Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5470805Z 2025-09-07T07:01:28.5470922Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5471137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5471216Z return mod(**inputs) 2025-09-07T07:01:28.5471549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5471624Z outputs = self.model( 2025-09-07T07:01:28.5471966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5472044Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5472380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5472458Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5472702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5472787Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5473116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5473247Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5473595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5473692Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5473696Z 2025-09-07T07:01:28.5473805Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5474022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5474092Z return mod(**inputs) 2025-09-07T07:01:28.5474428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5474511Z outputs = self.model( 2025-09-07T07:01:28.5474849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5474938Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5475278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5475350Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5475580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5475656Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5475972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5476081Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5476388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5476549Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5476569Z 2025-09-07T07:01:28.5476672Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5476889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5476956Z return mod(**inputs) 2025-09-07T07:01:28.5477277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5477346Z outputs = self.model( 2025-09-07T07:01:28.5477659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5477756Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5478067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5478145Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5478366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5478452Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5478754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5478859Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5479168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5479248Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5479251Z 2025-09-07T07:01:28.5479358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5479564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5479644Z return mod(**inputs) 2025-09-07T07:01:28.5479952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5480034Z outputs = self.model( 2025-09-07T07:01:28.5480346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5480417Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5480735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5480809Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5481029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5481118Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5481423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5481540Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5481845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5481933Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5481936Z 2025-09-07T07:01:28.5482026Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5482107Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5482196Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5482273Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5482377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5482583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5482659Z return mod(**inputs) 2025-09-07T07:01:28.5482984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5483050Z outputs = self.model( 2025-09-07T07:01:28.5483355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5483425Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5483724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5483818Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5484035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5484118Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5484420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5484528Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5484852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5484947Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5485238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5485371Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5485375Z 2025-09-07T07:01:28.5485481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5485677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5485741Z return mod(**inputs) 2025-09-07T07:01:28.5486079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5486151Z outputs = self.model( 2025-09-07T07:01:28.5486487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5486562Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5486878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5486962Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5487200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5487291Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5487621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5487744Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5488074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5488178Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5488500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5488615Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5488621Z 2025-09-07T07:01:28.5488736Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5488953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5489020Z return mod(**inputs) 2025-09-07T07:01:28.5489358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5490348Z outputs = self.model( 2025-09-07T07:01:28.5490676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5490754Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5491100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5491177Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5491444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5491536Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5491850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5491969Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5492284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5492366Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5492379Z 2025-09-07T07:01:28.5492482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5492684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5492758Z return mod(**inputs) 2025-09-07T07:01:28.5493072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5493148Z outputs = self.model( 2025-09-07T07:01:28.5493487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5493563Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5493897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5493973Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5494201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5494281Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5494605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5494734Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5494737Z 2025-09-07T07:01:28.5494839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5495049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5495116Z return mod(**inputs) 2025-09-07T07:01:28.5495436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5495506Z outputs = self.model( 2025-09-07T07:01:28.5495817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5495896Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5496211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5496290Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5496513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5496595Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5496931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5497052Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5497276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5497346Z return self.act(input) 2025-09-07T07:01:28.5497349Z 2025-09-07T07:01:28.5497457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5497658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5497743Z return mod(**inputs) 2025-09-07T07:01:28.5498065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5498134Z outputs = self.model( 2025-09-07T07:01:28.5498459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5498534Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5498847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5498927Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5499154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5499243Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5499555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5499644Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5499648Z 2025-09-07T07:01:28.5499775Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5499976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5500051Z return mod(**inputs) 2025-09-07T07:01:28.5500381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5500458Z outputs = self.model( 2025-09-07T07:01:28.5500769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5500847Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5501209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5501284Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5501536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5501621Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5501955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5502061Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5502385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5502553Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5502559Z 2025-09-07T07:01:28.5502666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5502884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5502955Z return mod(**inputs) 2025-09-07T07:01:28.5503286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5503382Z outputs = self.model( 2025-09-07T07:01:28.5503727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5503812Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5504153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5504237Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5504489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5504573Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5504919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5505025Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5505389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5505477Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5505481Z 2025-09-07T07:01:28.5505663Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5505892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5505965Z return mod(**inputs) 2025-09-07T07:01:28.5506332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5506408Z outputs = self.model( 2025-09-07T07:01:28.5506777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5506870Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5507221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5507310Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5507551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5507643Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5507974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5508083Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5508421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5508516Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5508521Z 2025-09-07T07:01:28.5508617Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5508702Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5508796Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5508879Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5508988Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5509208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5509279Z return mod(**inputs) 2025-09-07T07:01:28.5509618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5509691Z outputs = self.model( 2025-09-07T07:01:28.5510025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5510129Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5510477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5510560Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5510803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5510886Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5511228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5511351Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5511685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5511790Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5512111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5512256Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5512260Z 2025-09-07T07:01:28.5512368Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5512587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5512656Z return mod(**inputs) 2025-09-07T07:01:28.5512994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5513069Z outputs = self.model( 2025-09-07T07:01:28.5513410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5513503Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5513842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5513941Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5514179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5514269Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5514601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5514706Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5515040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5515141Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5515460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5515577Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5515580Z 2025-09-07T07:01:28.5515697Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5515909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5515978Z return mod(**inputs) 2025-09-07T07:01:28.5516312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5516385Z outputs = self.model( 2025-09-07T07:01:28.5516718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5516794Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5517144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5517228Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5517464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5517556Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5517881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-09-07T07:01:28.5518021Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:01:28.5518348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5518434Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5518437Z 2025-09-07T07:01:28.5518554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5518763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5518840Z return mod(**inputs) 2025-09-07T07:01:28.5519171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5519242Z outputs = self.model( 2025-09-07T07:01:28.5519735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5519825Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5520990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5521160Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5521680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5521792Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5522251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5522385Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5522722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-09-07T07:01:28.5522892Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:01:28.5522904Z 2025-09-07T07:01:28.5523027Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5523249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5523335Z return mod(**inputs) 2025-09-07T07:01:28.5523691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5523789Z outputs = self.model( 2025-09-07T07:01:28.5524102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5524188Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5524540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5524616Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5524850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5524933Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5525250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5525394Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5525700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-09-07T07:01:28.5525790Z key_states = self.k_proj(current_states) 2025-09-07T07:01:28.5525795Z 2025-09-07T07:01:28.5525902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5526115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5526222Z return mod(**inputs) 2025-09-07T07:01:28.5526536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5526607Z outputs = self.model( 2025-09-07T07:01:28.5526968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5527052Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5527357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5527436Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5527659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5527736Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5528046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5528157Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5528486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-09-07T07:01:28.5528577Z value_states = self.v_proj(current_states) 2025-09-07T07:01:28.5528581Z 2025-09-07T07:01:28.5528670Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5528767Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5528844Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5528924Z cudagraph partition due to non gpu ops 2025-09-07T07:01:28.5529027Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5529237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5529303Z return mod(**inputs) 2025-09-07T07:01:28.5529611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5529686Z outputs = self.model( 2025-09-07T07:01:28.5529993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5530075Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5530381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5530451Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5530680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5530760Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5531079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5531185Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5531533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5531636Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5531943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:01:28.5532085Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:28.5532089Z 2025-09-07T07:01:28.5532192Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5532399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5532466Z return mod(**inputs) 2025-09-07T07:01:28.5532789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5532862Z outputs = self.model( 2025-09-07T07:01:28.5533169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5533249Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5533556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5533635Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5533856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5533937Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5534247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5534355Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5534660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-09-07T07:01:28.5534774Z attn_output, attn_weights = attention_interface( 2025-09-07T07:01:28.5535072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:01:28.5535196Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:01:28.5535201Z 2025-09-07T07:01:28.5535302Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5535508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5535573Z return mod(**inputs) 2025-09-07T07:01:28.5535883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5535952Z outputs = self.model( 2025-09-07T07:01:28.5536329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5536410Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5536720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5536796Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5537017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5537100Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5537403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-09-07T07:01:28.5537511Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:01:28.5537820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-09-07T07:01:28.5537901Z attn_output = self.out_proj(attn_output) 2025-09-07T07:01:28.5537922Z 2025-09-07T07:01:28.5538041Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5538233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5538300Z return mod(**inputs) 2025-09-07T07:01:28.5538603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5538667Z outputs = self.model( 2025-09-07T07:01:28.5538969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5539056Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5539354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5539423Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5539637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5539721Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5540015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5540140Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5540143Z 2025-09-07T07:01:28.5540241Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5540438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5540512Z return mod(**inputs) 2025-09-07T07:01:28.5540818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5540893Z outputs = self.model( 2025-09-07T07:01:28.5541213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5541297Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5541637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5541709Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5541933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5542011Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5542324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-09-07T07:01:28.5542442Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:01:28.5542653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:28.5542731Z return self.act(input) 2025-09-07T07:01:28.5542734Z 2025-09-07T07:01:28.5542837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5543045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5543110Z return mod(**inputs) 2025-09-07T07:01:28.5543419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-09-07T07:01:28.5543485Z outputs = self.model( 2025-09-07T07:01:28.5543789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-09-07T07:01:28.5543869Z decoder_outputs = self.decoder( 2025-09-07T07:01:28.5544171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-09-07T07:01:28.5544279Z layer_outputs = decoder_layer( 2025-09-07T07:01:28.5544507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:28.5544588Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:28.5544906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-09-07T07:01:28.5544991Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:01:28.5544994Z 2025-09-07T07:01:28.5545106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5545321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5545394Z return mod(**inputs) 2025-09-07T07:01:28.5545900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1393, in forward 2025-09-07T07:01:28.5546031Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-09-07T07:01:28.5546036Z 2025-09-07T07:01:28.5546151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:28.5546364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:28.5546444Z return mod(**inputs) 2025-09-07T07:01:28.5546788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1398, in forward 2025-09-07T07:01:28.5546967Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:01:28.5546981Z 2025-09-07T07:01:40.6035976Z Compilation time (from dynamo_timed): 22.951186128 2025-09-07T07:01:40.6050216Z pass 2025-09-07T07:01:40.6050794Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:40.6052214Z TIMING: _recursive_pre_grad_passes:0.01029 _recursive_joint_graph_passes:0.58603 _recursive_post_grad_passes:0.11809 async_compile.wait:0.79324 code_gen:11.40571 inductor_compile:13.91894 backend_compile:19.08298 gc:0.0026 entire_frame_compile:22.95119 total_wall_time:22.95119 2025-09-07T07:01:40.6053264Z STATS: call_* op count: 652 | FakeTensorMode.__torch_dispatch__:22573 | FakeTensor.__torch_dispatch__:7513 | ProxyTorchDispatchMode.__torch_dispatch__:8304 2025-09-07T07:01:40.6053839Z Dynamo produced 1 graphs covering 652 ops with 0 graph breaks (0 unique) 2025-09-07T07:01:43.5189150Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:01:43.5190085Z import pynvml # type: ignore[import] 2025-09-07T07:01:46.2841581Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:01:46.2842519Z from pkg_resources import resource_filename 2025-09-07T07:01:46.9631292Z 2025-09-07T07:01:48.2494396Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:01:48.2494722Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:01:48.2502032Z cpu eval CamemBert 2025-09-07T07:01:48.7906075Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:49.0629301Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:49.3238746Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:01:57.1543705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1544650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1545036Z return mod(**inputs) 2025-09-07T07:01:57.1545780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1546285Z outputs = self.roberta( 2025-09-07T07:01:57.1546748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-09-07T07:01:57.1547249Z embedding_output = self.embeddings( 2025-09-07T07:01:57.1547833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-09-07T07:01:57.1548443Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:01:57.1549140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1590, in create_position_ids_from_input_ids 2025-09-07T07:01:57.1549688Z mask = input_ids.ne(padding_idx).int() 2025-09-07T07:01:57.1549857Z 2025-09-07T07:01:57.1549949Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1550193Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1550421Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1550652Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1550870Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1551090Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1551309Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1551533Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1551749Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1551971Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1552191Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1552412Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1552733Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1553140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1553562Z return mod(**inputs) 2025-09-07T07:01:57.1553991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1554439Z outputs = self.roberta( 2025-09-07T07:01:57.1554882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-09-07T07:01:57.1555357Z embedding_output = self.embeddings( 2025-09-07T07:01:57.1555811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-09-07T07:01:57.1556397Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:01:57.1557070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-09-07T07:01:57.1557746Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:01:57.1558026Z 2025-09-07T07:01:57.1558149Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1558565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1558931Z return mod(**inputs) 2025-09-07T07:01:57.1559366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1559814Z outputs = self.roberta( 2025-09-07T07:01:57.1560271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-09-07T07:01:57.1560745Z embedding_output = self.embeddings( 2025-09-07T07:01:57.1561243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-09-07T07:01:57.1561846Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:01:57.1562522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-09-07T07:01:57.1563178Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:01:57.1563465Z 2025-09-07T07:01:57.1563589Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1563990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1564360Z return mod(**inputs) 2025-09-07T07:01:57.1564795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1565246Z outputs = self.roberta( 2025-09-07T07:01:57.1565693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1566150Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1566629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1567093Z layer_outputs = layer_module( 2025-09-07T07:01:57.1567485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1567911Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1568401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1568927Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1569387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1569842Z return func(*args, **kwargs) 2025-09-07T07:01:57.1570323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1570783Z self_outputs = self.self( 2025-09-07T07:01:57.1571194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1571633Z return func(*args, **kwargs) 2025-09-07T07:01:57.1572089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1572710Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1572996Z 2025-09-07T07:01:57.1573114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1573520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1573892Z return mod(**inputs) 2025-09-07T07:01:57.1574335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1574797Z outputs = self.roberta( 2025-09-07T07:01:57.1575225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1575681Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1576138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1576590Z layer_outputs = layer_module( 2025-09-07T07:01:57.1576965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1577396Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1577862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1578317Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1578743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1579153Z return func(*args, **kwargs) 2025-09-07T07:01:57.1579612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1580083Z self_outputs = self.self( 2025-09-07T07:01:57.1580496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1580927Z return func(*args, **kwargs) 2025-09-07T07:01:57.1581358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.1581809Z self.key(current_states) 2025-09-07T07:01:57.1581943Z 2025-09-07T07:01:57.1582061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1582461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1582817Z return mod(**inputs) 2025-09-07T07:01:57.1583246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1583696Z outputs = self.roberta( 2025-09-07T07:01:57.1584132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1584582Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1585042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1585497Z layer_outputs = layer_module( 2025-09-07T07:01:57.1586010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1586426Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1586895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1587338Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1587755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1588159Z return func(*args, **kwargs) 2025-09-07T07:01:57.1588588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1589021Z self_outputs = self.self( 2025-09-07T07:01:57.1589415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1589817Z return func(*args, **kwargs) 2025-09-07T07:01:57.1590253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.1590695Z self.value(current_states) 2025-09-07T07:01:57.1590818Z 2025-09-07T07:01:57.1590903Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1591153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1591526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1591874Z return mod(**inputs) 2025-09-07T07:01:57.1592289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1592739Z outputs = self.roberta( 2025-09-07T07:01:57.1593173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1593651Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1594102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1594536Z layer_outputs = layer_module( 2025-09-07T07:01:57.1594893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1595290Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1595757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1596225Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1596640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1597049Z return func(*args, **kwargs) 2025-09-07T07:01:57.1597494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1597917Z self_outputs = self.self( 2025-09-07T07:01:57.1598287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1598675Z return func(*args, **kwargs) 2025-09-07T07:01:57.1599082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.1599572Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.1599765Z 2025-09-07T07:01:57.1599879Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1600251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1600688Z return mod(**inputs) 2025-09-07T07:01:57.1601085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1601515Z outputs = self.roberta( 2025-09-07T07:01:57.1601910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1602320Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1602756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1603198Z layer_outputs = layer_module( 2025-09-07T07:01:57.1603578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1603971Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1604409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1604837Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1605228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1605612Z return func(*args, **kwargs) 2025-09-07T07:01:57.1606006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.1606483Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.1606972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.1607434Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1607589Z 2025-09-07T07:01:57.1607710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1608094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1608467Z return mod(**inputs) 2025-09-07T07:01:57.1608886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1609326Z outputs = self.roberta( 2025-09-07T07:01:57.1609751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1610206Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1610652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1611119Z layer_outputs = layer_module( 2025-09-07T07:01:57.1611501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1611894Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1612371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1612834Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1613278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1613707Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1614169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1614696Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1615193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.1615654Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1615804Z 2025-09-07T07:01:57.1615937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1616328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1616683Z return mod(**inputs) 2025-09-07T07:01:57.1617132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1617574Z outputs = self.roberta( 2025-09-07T07:01:57.1617986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1618427Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1618864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1619312Z layer_outputs = layer_module( 2025-09-07T07:01:57.1620045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1620444Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1620899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1621356Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1621808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1622241Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1622714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1623248Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1623740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.1624228Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.1624705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.1625074Z return self.act(input) 2025-09-07T07:01:57.1625203Z 2025-09-07T07:01:57.1625318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1625775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1626135Z return mod(**inputs) 2025-09-07T07:01:57.1626554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1627054Z outputs = self.roberta( 2025-09-07T07:01:57.1627478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1627925Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1628370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1628806Z layer_outputs = layer_module( 2025-09-07T07:01:57.1629190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1629592Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1630039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1630487Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1630931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1631346Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1631850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.1632389Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.1632919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.1633348Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1633497Z 2025-09-07T07:01:57.1633604Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1633977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1634310Z return mod(**inputs) 2025-09-07T07:01:57.1634702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1635119Z outputs = self.roberta( 2025-09-07T07:01:57.1635516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1635936Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1636348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1636756Z layer_outputs = layer_module( 2025-09-07T07:01:57.1637117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1637494Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1637912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1638336Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1638725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1639109Z return func(*args, **kwargs) 2025-09-07T07:01:57.1639514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1639949Z self_outputs = self.self( 2025-09-07T07:01:57.1640312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1640690Z return func(*args, **kwargs) 2025-09-07T07:01:57.1641095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1641693Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1642002Z 2025-09-07T07:01:57.1642120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1642508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1642879Z return mod(**inputs) 2025-09-07T07:01:57.1643295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1643712Z outputs = self.roberta( 2025-09-07T07:01:57.1644100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1644518Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1645217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1645633Z layer_outputs = layer_module( 2025-09-07T07:01:57.1645990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1646352Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1646771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1647218Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1647615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1648018Z return func(*args, **kwargs) 2025-09-07T07:01:57.1648416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1648828Z self_outputs = self.self( 2025-09-07T07:01:57.1649197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1649578Z return func(*args, **kwargs) 2025-09-07T07:01:57.1649978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.1650382Z self.key(current_states) 2025-09-07T07:01:57.1650507Z 2025-09-07T07:01:57.1650615Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1650982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1651310Z return mod(**inputs) 2025-09-07T07:01:57.1651701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1652112Z outputs = self.roberta( 2025-09-07T07:01:57.1652506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1652921Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1653359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1653779Z layer_outputs = layer_module( 2025-09-07T07:01:57.1654133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1654530Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1654966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1655431Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1655836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1656234Z return func(*args, **kwargs) 2025-09-07T07:01:57.1656641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1657099Z self_outputs = self.self( 2025-09-07T07:01:57.1657461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1657841Z return func(*args, **kwargs) 2025-09-07T07:01:57.1658245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.1658665Z self.value(current_states) 2025-09-07T07:01:57.1658784Z 2025-09-07T07:01:57.1658877Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1659129Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1659520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1659870Z return mod(**inputs) 2025-09-07T07:01:57.1660288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1660722Z outputs = self.roberta( 2025-09-07T07:01:57.1661143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1661614Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1662070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1662512Z layer_outputs = layer_module( 2025-09-07T07:01:57.1663672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1664092Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1664535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1664986Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1665400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1665878Z return func(*args, **kwargs) 2025-09-07T07:01:57.1666317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1666761Z self_outputs = self.self( 2025-09-07T07:01:57.1667155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1667554Z return func(*args, **kwargs) 2025-09-07T07:01:57.1667984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.1668489Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.1668693Z 2025-09-07T07:01:57.1668817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1669209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1669556Z return mod(**inputs) 2025-09-07T07:01:57.1669976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1670413Z outputs = self.roberta( 2025-09-07T07:01:57.1670830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1671295Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1671723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1672160Z layer_outputs = layer_module( 2025-09-07T07:01:57.1672540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1672938Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1673395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1673850Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1674260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1674642Z return func(*args, **kwargs) 2025-09-07T07:01:57.1675042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.1675512Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.1675993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.1676420Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1676562Z 2025-09-07T07:01:57.1676677Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1677052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1677377Z return mod(**inputs) 2025-09-07T07:01:57.1677796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1678214Z outputs = self.roberta( 2025-09-07T07:01:57.1678610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1679051Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1679464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1679888Z layer_outputs = layer_module( 2025-09-07T07:01:57.1680243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1680614Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1681025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1681477Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1681922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1682357Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1682834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1683355Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1683827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.1684258Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1684401Z 2025-09-07T07:01:57.1684517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1684887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1685218Z return mod(**inputs) 2025-09-07T07:01:57.1685620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1686062Z outputs = self.roberta( 2025-09-07T07:01:57.1686458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1686872Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1687288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1687706Z layer_outputs = layer_module( 2025-09-07T07:01:57.1688093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1688469Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1688902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1689338Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1689753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1690162Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1690614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1691102Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1691569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.1692034Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.1692431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.1692780Z return self.act(input) 2025-09-07T07:01:57.1692926Z 2025-09-07T07:01:57.1693036Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1693404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1693760Z return mod(**inputs) 2025-09-07T07:01:57.1694162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1694569Z outputs = self.roberta( 2025-09-07T07:01:57.1694966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1695384Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1695803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1696216Z layer_outputs = layer_module( 2025-09-07T07:01:57.1696567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1696943Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1697364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1697790Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1698195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1698605Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1699052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.1699561Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.1700037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.1700510Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1700673Z 2025-09-07T07:01:57.1700785Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1701189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1701552Z return mod(**inputs) 2025-09-07T07:01:57.1701980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1702432Z outputs = self.roberta( 2025-09-07T07:01:57.1702873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1703362Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1703817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1704286Z layer_outputs = layer_module( 2025-09-07T07:01:57.1704677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1705088Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1705557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1706113Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1706550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1707037Z return func(*args, **kwargs) 2025-09-07T07:01:57.1707439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1707854Z self_outputs = self.self( 2025-09-07T07:01:57.1708252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1708625Z return func(*args, **kwargs) 2025-09-07T07:01:57.1709045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1709599Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1709861Z 2025-09-07T07:01:57.1709974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1710338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1710663Z return mod(**inputs) 2025-09-07T07:01:57.1711052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1711460Z outputs = self.roberta( 2025-09-07T07:01:57.1711850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1712261Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1712681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1713166Z layer_outputs = layer_module( 2025-09-07T07:01:57.1713558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1713953Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1714387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1714855Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1715282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1715694Z return func(*args, **kwargs) 2025-09-07T07:01:57.1716162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1716594Z self_outputs = self.self( 2025-09-07T07:01:57.1716983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1717378Z return func(*args, **kwargs) 2025-09-07T07:01:57.1717807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.1718235Z self.key(current_states) 2025-09-07T07:01:57.1718388Z 2025-09-07T07:01:57.1718499Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1718896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1719256Z return mod(**inputs) 2025-09-07T07:01:57.1719820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1720266Z outputs = self.roberta( 2025-09-07T07:01:57.1720688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1721116Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1721522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1721941Z layer_outputs = layer_module( 2025-09-07T07:01:57.1722302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1722671Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1723089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1723563Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1723941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1724311Z return func(*args, **kwargs) 2025-09-07T07:01:57.1724733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1725149Z self_outputs = self.self( 2025-09-07T07:01:57.1725515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1725883Z return func(*args, **kwargs) 2025-09-07T07:01:57.1726290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.1726714Z self.value(current_states) 2025-09-07T07:01:57.1726835Z 2025-09-07T07:01:57.1726928Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1727181Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1727549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1727876Z return mod(**inputs) 2025-09-07T07:01:57.1728271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1728683Z outputs = self.roberta( 2025-09-07T07:01:57.1729065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1729475Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1729881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1730291Z layer_outputs = layer_module( 2025-09-07T07:01:57.1730635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1731021Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1731436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1731851Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1732228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1732591Z return func(*args, **kwargs) 2025-09-07T07:01:57.1732981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1733458Z self_outputs = self.self( 2025-09-07T07:01:57.1733816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1734183Z return func(*args, **kwargs) 2025-09-07T07:01:57.1734568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.1735036Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.1735230Z 2025-09-07T07:01:57.1735337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1735710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1736043Z return mod(**inputs) 2025-09-07T07:01:57.1736432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1736855Z outputs = self.roberta( 2025-09-07T07:01:57.1737251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1737661Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1738090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1738507Z layer_outputs = layer_module( 2025-09-07T07:01:57.1738889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1739262Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1739685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1740109Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1740510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1740891Z return func(*args, **kwargs) 2025-09-07T07:01:57.1741313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.1741800Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.1742265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.1742703Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1742862Z 2025-09-07T07:01:57.1742974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1743365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1743713Z return mod(**inputs) 2025-09-07T07:01:57.1744134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1744576Z outputs = self.roberta( 2025-09-07T07:01:57.1744995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1745450Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1745994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1746464Z layer_outputs = layer_module( 2025-09-07T07:01:57.1746860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1747259Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1747682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1748125Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1748543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1748950Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1749403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1749901Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1750357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.1750786Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1750932Z 2025-09-07T07:01:57.1751038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1751410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1751756Z return mod(**inputs) 2025-09-07T07:01:57.1752174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1752610Z outputs = self.roberta( 2025-09-07T07:01:57.1753042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1753482Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1753910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1754323Z layer_outputs = layer_module( 2025-09-07T07:01:57.1754677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1755060Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1755503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1755946Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1756380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1756808Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1757284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1757800Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1758297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.1758783Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.1759198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.1759576Z return self.act(input) 2025-09-07T07:01:57.1759698Z 2025-09-07T07:01:57.1759812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1760207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1760574Z return mod(**inputs) 2025-09-07T07:01:57.1760994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1761460Z outputs = self.roberta( 2025-09-07T07:01:57.1761886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1762328Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1762776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1763227Z layer_outputs = layer_module( 2025-09-07T07:01:57.1763617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1764035Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1764495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1764952Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1765392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1765815Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1766288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.1766827Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.1767335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.1767792Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1767942Z 2025-09-07T07:01:57.1768055Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1768465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1768816Z return mod(**inputs) 2025-09-07T07:01:57.1769239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1769657Z outputs = self.roberta( 2025-09-07T07:01:57.1770046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1770463Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1770876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1771292Z layer_outputs = layer_module( 2025-09-07T07:01:57.1771638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1772009Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1772426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1772856Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1773255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1773631Z return func(*args, **kwargs) 2025-09-07T07:01:57.1774033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1774445Z self_outputs = self.self( 2025-09-07T07:01:57.1774818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1775198Z return func(*args, **kwargs) 2025-09-07T07:01:57.1775592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1776148Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1776450Z 2025-09-07T07:01:57.1776559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1776938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1777265Z return mod(**inputs) 2025-09-07T07:01:57.1777664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1778082Z outputs = self.roberta( 2025-09-07T07:01:57.1778503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1778926Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1779341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1779771Z layer_outputs = layer_module( 2025-09-07T07:01:57.1780135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1780518Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1780948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1781376Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1781798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1782209Z return func(*args, **kwargs) 2025-09-07T07:01:57.1782642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1783080Z self_outputs = self.self( 2025-09-07T07:01:57.1783502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1783910Z return func(*args, **kwargs) 2025-09-07T07:01:57.1784365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.1784805Z self.key(current_states) 2025-09-07T07:01:57.1784930Z 2025-09-07T07:01:57.1785042Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1785438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1785872Z return mod(**inputs) 2025-09-07T07:01:57.1786308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1786751Z outputs = self.roberta( 2025-09-07T07:01:57.1787176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1787596Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1788013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1788424Z layer_outputs = layer_module( 2025-09-07T07:01:57.1788778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1789155Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1789565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1789991Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1790378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1790740Z return func(*args, **kwargs) 2025-09-07T07:01:57.1791132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1791561Z self_outputs = self.self( 2025-09-07T07:01:57.1791919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1792287Z return func(*args, **kwargs) 2025-09-07T07:01:57.1792681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.1793118Z self.value(current_states) 2025-09-07T07:01:57.1793242Z 2025-09-07T07:01:57.1793348Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1793593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1793948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1794280Z return mod(**inputs) 2025-09-07T07:01:57.1794668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1795078Z outputs = self.roberta( 2025-09-07T07:01:57.1795471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1795872Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1796274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1796680Z layer_outputs = layer_module( 2025-09-07T07:01:57.1797033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1797388Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1797796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1798235Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1798621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1799010Z return func(*args, **kwargs) 2025-09-07T07:01:57.1799400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1799805Z self_outputs = self.self( 2025-09-07T07:01:57.1800164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1800537Z return func(*args, **kwargs) 2025-09-07T07:01:57.1800930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.1801387Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.1801579Z 2025-09-07T07:01:57.1801685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1802045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1802366Z return mod(**inputs) 2025-09-07T07:01:57.1802743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1803150Z outputs = self.roberta( 2025-09-07T07:01:57.1803536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1803941Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1804345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1804747Z layer_outputs = layer_module( 2025-09-07T07:01:57.1805097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1805485Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1805899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1806313Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1806690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1807059Z return func(*args, **kwargs) 2025-09-07T07:01:57.1807459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.1807930Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.1808372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.1808790Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1808934Z 2025-09-07T07:01:57.1809038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1809396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1809722Z return mod(**inputs) 2025-09-07T07:01:57.1810096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1810496Z outputs = self.roberta( 2025-09-07T07:01:57.1810881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1811281Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1811680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1812076Z layer_outputs = layer_module( 2025-09-07T07:01:57.1812454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1812822Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1813257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1813676Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1814072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1814466Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1814903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1815385Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1815843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.1816250Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1816388Z 2025-09-07T07:01:57.1816492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1816850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1817174Z return mod(**inputs) 2025-09-07T07:01:57.1817546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1817947Z outputs = self.roberta( 2025-09-07T07:01:57.1818333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1818741Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1819140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1819698Z layer_outputs = layer_module( 2025-09-07T07:01:57.1820059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1820429Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1820841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1821252Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1821650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1822096Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1822546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1823043Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1823498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.1823964Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.1824353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.1824704Z return self.act(input) 2025-09-07T07:01:57.1824821Z 2025-09-07T07:01:57.1824936Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1825325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1825748Z return mod(**inputs) 2025-09-07T07:01:57.1826191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1826649Z outputs = self.roberta( 2025-09-07T07:01:57.1827131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1827583Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1828023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1828437Z layer_outputs = layer_module( 2025-09-07T07:01:57.1828796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1829189Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1829649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1830117Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1830560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1830985Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1831440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.1831979Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.1832493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.1832961Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1833111Z 2025-09-07T07:01:57.1833235Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1833624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1833984Z return mod(**inputs) 2025-09-07T07:01:57.1834382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1834788Z outputs = self.roberta( 2025-09-07T07:01:57.1835200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1835608Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1836017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1836430Z layer_outputs = layer_module( 2025-09-07T07:01:57.1836787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1837176Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1837645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1838104Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1838519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1838894Z return func(*args, **kwargs) 2025-09-07T07:01:57.1839284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1839746Z self_outputs = self.self( 2025-09-07T07:01:57.1840116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1840500Z return func(*args, **kwargs) 2025-09-07T07:01:57.1840900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1841460Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1841740Z 2025-09-07T07:01:57.1841849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1842248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1842584Z return mod(**inputs) 2025-09-07T07:01:57.1842998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1843423Z outputs = self.roberta( 2025-09-07T07:01:57.1843822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1844243Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1844658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1845068Z layer_outputs = layer_module( 2025-09-07T07:01:57.1845426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1845809Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1846233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1846670Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1847057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1847437Z return func(*args, **kwargs) 2025-09-07T07:01:57.1847844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1848288Z self_outputs = self.self( 2025-09-07T07:01:57.1848671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1849074Z return func(*args, **kwargs) 2025-09-07T07:01:57.1849515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.1849983Z self.key(current_states) 2025-09-07T07:01:57.1850109Z 2025-09-07T07:01:57.1850229Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1850620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1850956Z return mod(**inputs) 2025-09-07T07:01:57.1851352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1851767Z outputs = self.roberta( 2025-09-07T07:01:57.1852194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1852673Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1853121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1853576Z layer_outputs = layer_module( 2025-09-07T07:01:57.1853975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1854364Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1854810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1855262Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1855678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1856080Z return func(*args, **kwargs) 2025-09-07T07:01:57.1856499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1856936Z self_outputs = self.self( 2025-09-07T07:01:57.1857350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1857757Z return func(*args, **kwargs) 2025-09-07T07:01:57.1858215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.1858673Z self.value(current_states) 2025-09-07T07:01:57.1858806Z 2025-09-07T07:01:57.1858895Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1859156Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1859549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1859905Z return mod(**inputs) 2025-09-07T07:01:57.1860335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1860788Z outputs = self.roberta( 2025-09-07T07:01:57.1861224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1861672Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1862131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1862587Z layer_outputs = layer_module( 2025-09-07T07:01:57.1862976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1863386Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1863833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1864300Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1864719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1865130Z return func(*args, **kwargs) 2025-09-07T07:01:57.1865647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1866134Z self_outputs = self.self( 2025-09-07T07:01:57.1866547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1866980Z return func(*args, **kwargs) 2025-09-07T07:01:57.1867418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.1867930Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.1868166Z 2025-09-07T07:01:57.1868281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1868677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1869029Z return mod(**inputs) 2025-09-07T07:01:57.1869450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1869886Z outputs = self.roberta( 2025-09-07T07:01:57.1870310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1870753Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1871193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1871660Z layer_outputs = layer_module( 2025-09-07T07:01:57.1872035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1872441Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1872934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1873419Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1873836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1874264Z return func(*args, **kwargs) 2025-09-07T07:01:57.1874702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.1875217Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.1875722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.1876184Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1876346Z 2025-09-07T07:01:57.1876461Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1876847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1877218Z return mod(**inputs) 2025-09-07T07:01:57.1877640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1878075Z outputs = self.roberta( 2025-09-07T07:01:57.1878560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1878991Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1879404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1879821Z layer_outputs = layer_module( 2025-09-07T07:01:57.1880169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1880603Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1881025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1881478Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1881885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1882293Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1882746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1883241Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1883735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.1884152Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1884299Z 2025-09-07T07:01:57.1884405Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1884771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1885106Z return mod(**inputs) 2025-09-07T07:01:57.1885499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1885906Z outputs = self.roberta( 2025-09-07T07:01:57.1886302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1886715Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1887124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1887527Z layer_outputs = layer_module( 2025-09-07T07:01:57.1887882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1888271Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1888695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1889146Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1889554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1889959Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1890411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1890911Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1891379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.1891834Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.1892234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.1892593Z return self.act(input) 2025-09-07T07:01:57.1892713Z 2025-09-07T07:01:57.1892834Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1893225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1893573Z return mod(**inputs) 2025-09-07T07:01:57.1893992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1894438Z outputs = self.roberta( 2025-09-07T07:01:57.1894837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1895244Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1895659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1896098Z layer_outputs = layer_module( 2025-09-07T07:01:57.1896459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1896830Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1897247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1897677Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1898084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1898497Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1898943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.1899440Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.1899912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.1900341Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1900483Z 2025-09-07T07:01:57.1900597Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1900955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1901288Z return mod(**inputs) 2025-09-07T07:01:57.1901684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1902105Z outputs = self.roberta( 2025-09-07T07:01:57.1902520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1902949Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1903404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1903845Z layer_outputs = layer_module( 2025-09-07T07:01:57.1904246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1904642Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1905087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1905552Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1906232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1906658Z return func(*args, **kwargs) 2025-09-07T07:01:57.1907106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1907552Z self_outputs = self.self( 2025-09-07T07:01:57.1907950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1908360Z return func(*args, **kwargs) 2025-09-07T07:01:57.1908794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1909377Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1909674Z 2025-09-07T07:01:57.1909789Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1910181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1910540Z return mod(**inputs) 2025-09-07T07:01:57.1910964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1911427Z outputs = self.roberta( 2025-09-07T07:01:57.1911853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1912290Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1912730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1913167Z layer_outputs = layer_module( 2025-09-07T07:01:57.1913535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1913955Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1914411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1914878Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1915294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1915711Z return func(*args, **kwargs) 2025-09-07T07:01:57.1916139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1916579Z self_outputs = self.self( 2025-09-07T07:01:57.1916980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1917385Z return func(*args, **kwargs) 2025-09-07T07:01:57.1917817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.1918257Z self.key(current_states) 2025-09-07T07:01:57.1918385Z 2025-09-07T07:01:57.1918506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1918911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1919278Z return mod(**inputs) 2025-09-07T07:01:57.1919891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1920346Z outputs = self.roberta( 2025-09-07T07:01:57.1920773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1921210Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1921659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1922103Z layer_outputs = layer_module( 2025-09-07T07:01:57.1922483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1922888Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1923336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1923799Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1924215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1924619Z return func(*args, **kwargs) 2025-09-07T07:01:57.1925007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1925401Z self_outputs = self.self( 2025-09-07T07:01:57.1925750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1926116Z return func(*args, **kwargs) 2025-09-07T07:01:57.1926511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.1926911Z self.value(current_states) 2025-09-07T07:01:57.1927068Z 2025-09-07T07:01:57.1927150Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1927394Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1927766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1928073Z return mod(**inputs) 2025-09-07T07:01:57.1928449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1928841Z outputs = self.roberta( 2025-09-07T07:01:57.1929243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1929646Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1930028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1930425Z layer_outputs = layer_module( 2025-09-07T07:01:57.1930761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1931119Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1931515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1931910Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1932279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1932640Z return func(*args, **kwargs) 2025-09-07T07:01:57.1933022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1933407Z self_outputs = self.self( 2025-09-07T07:01:57.1933781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1934141Z return func(*args, **kwargs) 2025-09-07T07:01:57.1934540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.1934999Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.1935182Z 2025-09-07T07:01:57.1935285Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1935648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1935975Z return mod(**inputs) 2025-09-07T07:01:57.1936360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1936764Z outputs = self.roberta( 2025-09-07T07:01:57.1937143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1937549Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1937948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1938351Z layer_outputs = layer_module( 2025-09-07T07:01:57.1938694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1939047Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1939445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1939858Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1940235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1940595Z return func(*args, **kwargs) 2025-09-07T07:01:57.1940987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.1941482Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.1941945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.1942373Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1942515Z 2025-09-07T07:01:57.1942624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1942996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1943350Z return mod(**inputs) 2025-09-07T07:01:57.1943759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1944185Z outputs = self.roberta( 2025-09-07T07:01:57.1944582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1945014Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1945469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1945982Z layer_outputs = layer_module( 2025-09-07T07:01:57.1946357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1946776Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1947241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1947719Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1948170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1948565Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1949021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1949507Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1949959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.1950382Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1950518Z 2025-09-07T07:01:57.1950624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1950986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1951308Z return mod(**inputs) 2025-09-07T07:01:57.1951699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1952094Z outputs = self.roberta( 2025-09-07T07:01:57.1952486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1952893Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1953300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1953704Z layer_outputs = layer_module( 2025-09-07T07:01:57.1954043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1954408Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1954818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1955237Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1955637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1956050Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1956464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1956925Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1957364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.1957815Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.1958180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.1958512Z return self.act(input) 2025-09-07T07:01:57.1958627Z 2025-09-07T07:01:57.1958729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1959081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1959392Z return mod(**inputs) 2025-09-07T07:01:57.1959778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1960183Z outputs = self.roberta( 2025-09-07T07:01:57.1960570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1960985Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1961378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1961775Z layer_outputs = layer_module( 2025-09-07T07:01:57.1962111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1962482Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1962885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1963319Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1963722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1964119Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1964546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.1965022Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.1965479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.1965890Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1966026Z 2025-09-07T07:01:57.1966138Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1966491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1966559Z return mod(**inputs) 2025-09-07T07:01:57.1966834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1966908Z outputs = self.roberta( 2025-09-07T07:01:57.1967178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1967257Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1967525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1967599Z layer_outputs = layer_module( 2025-09-07T07:01:57.1967818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1967944Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1968217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1968297Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1968542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1968610Z return func(*args, **kwargs) 2025-09-07T07:01:57.1968879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1968972Z self_outputs = self.self( 2025-09-07T07:01:57.1969206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1969281Z return func(*args, **kwargs) 2025-09-07T07:01:57.1969548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1969751Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1969763Z 2025-09-07T07:01:57.1969865Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1970056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1970128Z return mod(**inputs) 2025-09-07T07:01:57.1970407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1970483Z outputs = self.roberta( 2025-09-07T07:01:57.1970751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1970838Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1971114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1971202Z layer_outputs = layer_module( 2025-09-07T07:01:57.1971425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1971500Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1971764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1971852Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1972088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1972164Z return func(*args, **kwargs) 2025-09-07T07:01:57.1972430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1972507Z self_outputs = self.self( 2025-09-07T07:01:57.1972743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1972810Z return func(*args, **kwargs) 2025-09-07T07:01:57.1973082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.1973149Z self.key(current_states) 2025-09-07T07:01:57.1973152Z 2025-09-07T07:01:57.1973258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1973452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1973517Z return mod(**inputs) 2025-09-07T07:01:57.1973794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1973861Z outputs = self.roberta( 2025-09-07T07:01:57.1974154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1974225Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1974494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1974563Z layer_outputs = layer_module( 2025-09-07T07:01:57.1974777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1974876Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1975141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1975224Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1975458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1975525Z return func(*args, **kwargs) 2025-09-07T07:01:57.1975798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1975866Z self_outputs = self.self( 2025-09-07T07:01:57.1976108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1976173Z return func(*args, **kwargs) 2025-09-07T07:01:57.1976444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.1976522Z self.value(current_states) 2025-09-07T07:01:57.1976525Z 2025-09-07T07:01:57.1976605Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.1976712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1976919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1976992Z return mod(**inputs) 2025-09-07T07:01:57.1977284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1977351Z outputs = self.roberta( 2025-09-07T07:01:57.1977621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1977692Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1977969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1978038Z layer_outputs = layer_module( 2025-09-07T07:01:57.1978254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1978337Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1978605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1978691Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1978929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1978995Z return func(*args, **kwargs) 2025-09-07T07:01:57.1979272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1979341Z self_outputs = self.self( 2025-09-07T07:01:57.1979586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1979653Z return func(*args, **kwargs) 2025-09-07T07:01:57.1979933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.1980085Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.1980089Z 2025-09-07T07:01:57.1980191Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1980402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1980468Z return mod(**inputs) 2025-09-07T07:01:57.1980750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1980817Z outputs = self.roberta( 2025-09-07T07:01:57.1981118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1981200Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1981471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1981553Z layer_outputs = layer_module( 2025-09-07T07:01:57.1981771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1981858Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1982131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1982211Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1982456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1982534Z return func(*args, **kwargs) 2025-09-07T07:01:57.1982823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.1982956Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.1983253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.1983349Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1983352Z 2025-09-07T07:01:57.1983472Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1983682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1983748Z return mod(**inputs) 2025-09-07T07:01:57.1984036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1984106Z outputs = self.roberta( 2025-09-07T07:01:57.1984387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1984467Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1984747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1984828Z layer_outputs = layer_module( 2025-09-07T07:01:57.1985054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1985133Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1985422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1985508Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1985861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1985948Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1986280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1986427Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1986762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.1986860Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1986864Z 2025-09-07T07:01:57.1986972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1987182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1987250Z return mod(**inputs) 2025-09-07T07:01:57.1987533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1987629Z outputs = self.roberta( 2025-09-07T07:01:57.1987942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1988025Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1988303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1988373Z layer_outputs = layer_module( 2025-09-07T07:01:57.1988605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1988683Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1988967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1989049Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1989314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1989393Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1989717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.1989849Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.1990148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.1990267Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.1990478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.1990547Z return self.act(input) 2025-09-07T07:01:57.1990558Z 2025-09-07T07:01:57.1990659Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1990857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1990927Z return mod(**inputs) 2025-09-07T07:01:57.1991204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1991279Z outputs = self.roberta( 2025-09-07T07:01:57.1991553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1991626Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1991903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1991972Z layer_outputs = layer_module( 2025-09-07T07:01:57.1992198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1992278Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1992549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.1992639Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.1992898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.1992998Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.1993306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.1993446Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.1993716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.1993814Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.1993817Z 2025-09-07T07:01:57.1993926Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1994123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1994195Z return mod(**inputs) 2025-09-07T07:01:57.1994475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1994542Z outputs = self.roberta( 2025-09-07T07:01:57.1994823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1994895Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1995170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1995240Z layer_outputs = layer_module( 2025-09-07T07:01:57.1995466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1995545Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1995831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1995924Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1996184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1996262Z return func(*args, **kwargs) 2025-09-07T07:01:57.1996531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.1996602Z self_outputs = self.self( 2025-09-07T07:01:57.1996848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1996919Z return func(*args, **kwargs) 2025-09-07T07:01:57.1997196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.1997403Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.1997408Z 2025-09-07T07:01:57.1997517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.1997717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.1997782Z return mod(**inputs) 2025-09-07T07:01:57.1998064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.1998131Z outputs = self.roberta( 2025-09-07T07:01:57.1998406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.1998479Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.1998749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.1998825Z layer_outputs = layer_module( 2025-09-07T07:01:57.1999042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.1999145Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.1999418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.1999498Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.1999748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.1999815Z return func(*args, **kwargs) 2025-09-07T07:01:57.2000106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2000176Z self_outputs = self.self( 2025-09-07T07:01:57.2000420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2000490Z return func(*args, **kwargs) 2025-09-07T07:01:57.2000763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.2000841Z self.key(current_states) 2025-09-07T07:01:57.2000845Z 2025-09-07T07:01:57.2000947Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2001150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2001216Z return mod(**inputs) 2025-09-07T07:01:57.2001491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2001567Z outputs = self.roberta( 2025-09-07T07:01:57.2001836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2001914Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2002211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2002284Z layer_outputs = layer_module( 2025-09-07T07:01:57.2002534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2002613Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2002894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2002977Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2003234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2003300Z return func(*args, **kwargs) 2025-09-07T07:01:57.2003570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2003649Z self_outputs = self.self( 2025-09-07T07:01:57.2003891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2003965Z return func(*args, **kwargs) 2025-09-07T07:01:57.2004236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.2004307Z self.value(current_states) 2025-09-07T07:01:57.2004311Z 2025-09-07T07:01:57.2004401Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.2004505Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2004714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2004779Z return mod(**inputs) 2025-09-07T07:01:57.2005057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2005155Z outputs = self.roberta( 2025-09-07T07:01:57.2005430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2005511Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2005791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2005870Z layer_outputs = layer_module( 2025-09-07T07:01:57.2006094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2006194Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2006475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2006557Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2006816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2006892Z return func(*args, **kwargs) 2025-09-07T07:01:57.2007186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2007273Z self_outputs = self.self( 2025-09-07T07:01:57.2007536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2007618Z return func(*args, **kwargs) 2025-09-07T07:01:57.2007909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.2008060Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.2008063Z 2025-09-07T07:01:57.2008174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2008404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2008486Z return mod(**inputs) 2025-09-07T07:01:57.2008802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2008883Z outputs = self.roberta( 2025-09-07T07:01:57.2009178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2009266Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2009548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2009623Z layer_outputs = layer_module( 2025-09-07T07:01:57.2009855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2009934Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2010219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2010303Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2010553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2010635Z return func(*args, **kwargs) 2025-09-07T07:01:57.2010926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.2011074Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.2011367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.2011457Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2011461Z 2025-09-07T07:01:57.2011579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2011819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2011897Z return mod(**inputs) 2025-09-07T07:01:57.2012194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2012267Z outputs = self.roberta( 2025-09-07T07:01:57.2012572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2012648Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2012965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2013040Z layer_outputs = layer_module( 2025-09-07T07:01:57.2013285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2013370Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2013661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2013760Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2014039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2014128Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2014453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2014577Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2014861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.2014969Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2014975Z 2025-09-07T07:01:57.2015088Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2015308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2015384Z return mod(**inputs) 2025-09-07T07:01:57.2015672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2015740Z outputs = self.roberta( 2025-09-07T07:01:57.2016023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2016098Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2016384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2016455Z layer_outputs = layer_module( 2025-09-07T07:01:57.2016681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2016770Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2017048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2017140Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2017404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2017489Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2017802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2017924Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2018211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.2018345Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.2018568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.2018641Z return self.act(input) 2025-09-07T07:01:57.2018645Z 2025-09-07T07:01:57.2018748Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2018962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2019030Z return mod(**inputs) 2025-09-07T07:01:57.2019340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2019409Z outputs = self.roberta( 2025-09-07T07:01:57.2019870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2019954Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2020237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2020323Z layer_outputs = layer_module( 2025-09-07T07:01:57.2020562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2020655Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2020956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2021049Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2021336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2021418Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2021795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.2021946Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.2022289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.2022379Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2022385Z 2025-09-07T07:01:57.2022494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2022715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2022790Z return mod(**inputs) 2025-09-07T07:01:57.2023097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2023170Z outputs = self.roberta( 2025-09-07T07:01:57.2023480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2023569Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2023874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2023957Z layer_outputs = layer_module( 2025-09-07T07:01:57.2024213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2024297Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2024607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2024698Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2024970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2025046Z return func(*args, **kwargs) 2025-09-07T07:01:57.2025379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2025458Z self_outputs = self.self( 2025-09-07T07:01:57.2025779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2025868Z return func(*args, **kwargs) 2025-09-07T07:01:57.2026171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.2026445Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.2026450Z 2025-09-07T07:01:57.2026559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2026786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2026860Z return mod(**inputs) 2025-09-07T07:01:57.2027163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2027247Z outputs = self.roberta( 2025-09-07T07:01:57.2027553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2027638Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2027932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2028011Z layer_outputs = layer_module( 2025-09-07T07:01:57.2028260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2028345Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2028680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2028771Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2029049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2029134Z return func(*args, **kwargs) 2025-09-07T07:01:57.2029432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2029516Z self_outputs = self.self( 2025-09-07T07:01:57.2029775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2029861Z return func(*args, **kwargs) 2025-09-07T07:01:57.2030157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.2030232Z self.key(current_states) 2025-09-07T07:01:57.2030237Z 2025-09-07T07:01:57.2030357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2030570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2030648Z return mod(**inputs) 2025-09-07T07:01:57.2030947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2031018Z outputs = self.roberta( 2025-09-07T07:01:57.2031330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2031408Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2031713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2031788Z layer_outputs = layer_module( 2025-09-07T07:01:57.2032023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2032124Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2032389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2032476Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2032712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2032787Z return func(*args, **kwargs) 2025-09-07T07:01:57.2033063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2033151Z self_outputs = self.self( 2025-09-07T07:01:57.2033398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2033467Z return func(*args, **kwargs) 2025-09-07T07:01:57.2033754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.2033826Z self.value(current_states) 2025-09-07T07:01:57.2033831Z 2025-09-07T07:01:57.2033912Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.2034023Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2034220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2034293Z return mod(**inputs) 2025-09-07T07:01:57.2034572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2034640Z outputs = self.roberta( 2025-09-07T07:01:57.2034919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2035010Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2035289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2035373Z layer_outputs = layer_module( 2025-09-07T07:01:57.2035601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2035679Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2035946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2036036Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2036278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2036352Z return func(*args, **kwargs) 2025-09-07T07:01:57.2036632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2036705Z self_outputs = self.self( 2025-09-07T07:01:57.2036962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2037029Z return func(*args, **kwargs) 2025-09-07T07:01:57.2037295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.2037424Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.2037427Z 2025-09-07T07:01:57.2037533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2037727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2037790Z return mod(**inputs) 2025-09-07T07:01:57.2038070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2038155Z outputs = self.roberta( 2025-09-07T07:01:57.2038435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2038508Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2038776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2038851Z layer_outputs = layer_module( 2025-09-07T07:01:57.2039069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2039171Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2039438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2039520Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2039765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2039835Z return func(*args, **kwargs) 2025-09-07T07:01:57.2040114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.2040240Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.2040519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.2040602Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2040607Z 2025-09-07T07:01:57.2040710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2040913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2040978Z return mod(**inputs) 2025-09-07T07:01:57.2041274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2041345Z outputs = self.roberta( 2025-09-07T07:01:57.2041633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2041711Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2041980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2042057Z layer_outputs = layer_module( 2025-09-07T07:01:57.2042279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2042365Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2042631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2042716Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2042988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2043065Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2043366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2043485Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2043748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.2043835Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2043838Z 2025-09-07T07:01:57.2043938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2044139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2044205Z return mod(**inputs) 2025-09-07T07:01:57.2044497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2044564Z outputs = self.roberta( 2025-09-07T07:01:57.2044826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2044903Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2045166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2045257Z layer_outputs = layer_module( 2025-09-07T07:01:57.2045474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2045550Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2045834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2045917Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2046177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2046253Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2046566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2046686Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2046962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.2047085Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.2047300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.2047395Z return self.act(input) 2025-09-07T07:01:57.2047399Z 2025-09-07T07:01:57.2047501Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2047726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2047799Z return mod(**inputs) 2025-09-07T07:01:57.2048065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2048139Z outputs = self.roberta( 2025-09-07T07:01:57.2048402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2048482Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2048747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2048816Z layer_outputs = layer_module( 2025-09-07T07:01:57.2049039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2049116Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2049391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2049472Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2049730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2049812Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2050118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.2050261Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.2050533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.2050655Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2050658Z 2025-09-07T07:01:57.2050763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2050964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2051036Z return mod(**inputs) 2025-09-07T07:01:57.2051309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2051401Z outputs = self.roberta( 2025-09-07T07:01:57.2051683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2051754Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2052040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2052111Z layer_outputs = layer_module( 2025-09-07T07:01:57.2052349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2052430Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2052717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2052810Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2053069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2053145Z return func(*args, **kwargs) 2025-09-07T07:01:57.2053428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2053506Z self_outputs = self.self( 2025-09-07T07:01:57.2053781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2053855Z return func(*args, **kwargs) 2025-09-07T07:01:57.2054170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.2054379Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.2054382Z 2025-09-07T07:01:57.2054493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2054693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2054758Z return mod(**inputs) 2025-09-07T07:01:57.2055040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2055109Z outputs = self.roberta( 2025-09-07T07:01:57.2055389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2055463Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2055741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2055812Z layer_outputs = layer_module( 2025-09-07T07:01:57.2056031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2056116Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2056387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2056477Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2056719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2056806Z return func(*args, **kwargs) 2025-09-07T07:01:57.2057088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2057158Z self_outputs = self.self( 2025-09-07T07:01:57.2057404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2057472Z return func(*args, **kwargs) 2025-09-07T07:01:57.2057751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.2057837Z self.key(current_states) 2025-09-07T07:01:57.2057840Z 2025-09-07T07:01:57.2057943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2058147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2058213Z return mod(**inputs) 2025-09-07T07:01:57.2058502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2058571Z outputs = self.roberta( 2025-09-07T07:01:57.2058844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2058921Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2059192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2059267Z layer_outputs = layer_module( 2025-09-07T07:01:57.2059484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2059562Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2059856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2059940Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2060200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2060270Z return func(*args, **kwargs) 2025-09-07T07:01:57.2060571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2060648Z self_outputs = self.self( 2025-09-07T07:01:57.2060909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2060991Z return func(*args, **kwargs) 2025-09-07T07:01:57.2061288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.2061370Z self.value(current_states) 2025-09-07T07:01:57.2061374Z 2025-09-07T07:01:57.2061463Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.2061577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2061810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2061881Z return mod(**inputs) 2025-09-07T07:01:57.2062198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2062272Z outputs = self.roberta( 2025-09-07T07:01:57.2062576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2062664Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2062974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2063060Z layer_outputs = layer_module( 2025-09-07T07:01:57.2063308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2063422Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2063727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2063819Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2064095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2064170Z return func(*args, **kwargs) 2025-09-07T07:01:57.2064503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2064581Z self_outputs = self.self( 2025-09-07T07:01:57.2064847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2064933Z return func(*args, **kwargs) 2025-09-07T07:01:57.2065243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.2065398Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.2065401Z 2025-09-07T07:01:57.2065514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2065819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2065897Z return mod(**inputs) 2025-09-07T07:01:57.2066202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2066288Z outputs = self.roberta( 2025-09-07T07:01:57.2066589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2066698Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2067005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2067134Z layer_outputs = layer_module( 2025-09-07T07:01:57.2067388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2067473Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2067781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2067871Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2068138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2068222Z return func(*args, **kwargs) 2025-09-07T07:01:57.2068523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.2068672Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.2068972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.2069073Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2069077Z 2025-09-07T07:01:57.2069189Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2069405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2069488Z return mod(**inputs) 2025-09-07T07:01:57.2069794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2069877Z outputs = self.roberta( 2025-09-07T07:01:57.2070205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2070304Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2070613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2070689Z layer_outputs = layer_module( 2025-09-07T07:01:57.2070939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2071027Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2071336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2071445Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2071709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2071791Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2072086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2072213Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2072477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.2072556Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2072566Z 2025-09-07T07:01:57.2072665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2072856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2072929Z return mod(**inputs) 2025-09-07T07:01:57.2073197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2073269Z outputs = self.roberta( 2025-09-07T07:01:57.2073548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2073623Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2073919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2073989Z layer_outputs = layer_module( 2025-09-07T07:01:57.2074210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2074286Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2074554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2074644Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2074904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2074987Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2075288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2075413Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2075679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.2075788Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.2076003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.2076072Z return self.act(input) 2025-09-07T07:01:57.2076076Z 2025-09-07T07:01:57.2076181Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2076374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2076455Z return mod(**inputs) 2025-09-07T07:01:57.2076740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2076807Z outputs = self.roberta( 2025-09-07T07:01:57.2077077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2077144Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2077414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2077501Z layer_outputs = layer_module( 2025-09-07T07:01:57.2077712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2077796Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2078059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2078146Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2078397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2078472Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2078771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.2078904Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.2079180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.2079259Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2079262Z 2025-09-07T07:01:57.2079368Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2079579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2079645Z return mod(**inputs) 2025-09-07T07:01:57.2079937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2080005Z outputs = self.roberta( 2025-09-07T07:01:57.2080275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2080344Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2080612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2080689Z layer_outputs = layer_module( 2025-09-07T07:01:57.2080910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2080994Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2081258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2081339Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2081579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2081646Z return func(*args, **kwargs) 2025-09-07T07:01:57.2081917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2081985Z self_outputs = self.self( 2025-09-07T07:01:57.2082226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2082291Z return func(*args, **kwargs) 2025-09-07T07:01:57.2082556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.2082786Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.2082791Z 2025-09-07T07:01:57.2082888Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2083088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2083152Z return mod(**inputs) 2025-09-07T07:01:57.2083420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2083506Z outputs = self.roberta( 2025-09-07T07:01:57.2083773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2083849Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2084116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2084190Z layer_outputs = layer_module( 2025-09-07T07:01:57.2084403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2084477Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2084749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2084826Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2085064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2085133Z return func(*args, **kwargs) 2025-09-07T07:01:57.2085395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2086365Z self_outputs = self.self( 2025-09-07T07:01:57.2086623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2086696Z return func(*args, **kwargs) 2025-09-07T07:01:57.2086982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.2087053Z self.key(current_states) 2025-09-07T07:01:57.2087066Z 2025-09-07T07:01:57.2087170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2087367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2087444Z return mod(**inputs) 2025-09-07T07:01:57.2087716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2087790Z outputs = self.roberta( 2025-09-07T07:01:57.2088069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2088140Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2088416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2088485Z layer_outputs = layer_module( 2025-09-07T07:01:57.2088708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2088784Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2089048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2089140Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2089379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2089455Z return func(*args, **kwargs) 2025-09-07T07:01:57.2089749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2089828Z self_outputs = self.self( 2025-09-07T07:01:57.2090071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2090139Z return func(*args, **kwargs) 2025-09-07T07:01:57.2090421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.2090512Z self.value(current_states) 2025-09-07T07:01:57.2090516Z 2025-09-07T07:01:57.2090606Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.2090709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2090908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2090983Z return mod(**inputs) 2025-09-07T07:01:57.2091263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2091338Z outputs = self.roberta( 2025-09-07T07:01:57.2091613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2091684Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2091964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2092034Z layer_outputs = layer_module( 2025-09-07T07:01:57.2092261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2092339Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2092633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2092717Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2092975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2093052Z return func(*args, **kwargs) 2025-09-07T07:01:57.2093320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2093395Z self_outputs = self.self( 2025-09-07T07:01:57.2093632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2093701Z return func(*args, **kwargs) 2025-09-07T07:01:57.2093978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.2094111Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.2094116Z 2025-09-07T07:01:57.2094226Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2094423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2094496Z return mod(**inputs) 2025-09-07T07:01:57.2094771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2094837Z outputs = self.roberta( 2025-09-07T07:01:57.2095116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2095188Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2095463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2095531Z layer_outputs = layer_module( 2025-09-07T07:01:57.2095750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2095853Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2096128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2096217Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2096461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2096529Z return func(*args, **kwargs) 2025-09-07T07:01:57.2096828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.2096957Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.2097234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.2097318Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2097322Z 2025-09-07T07:01:57.2097429Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2097626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2097692Z return mod(**inputs) 2025-09-07T07:01:57.2097978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2098045Z outputs = self.roberta( 2025-09-07T07:01:57.2098325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2098397Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2098703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2098810Z layer_outputs = layer_module( 2025-09-07T07:01:57.2099031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2099134Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2099407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2099496Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2099757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2099834Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2100145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2100265Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2100546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.2100628Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2100632Z 2025-09-07T07:01:57.2100735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2100940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2101005Z return mod(**inputs) 2025-09-07T07:01:57.2101284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2101351Z outputs = self.roberta( 2025-09-07T07:01:57.2101672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2101745Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2102026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2102131Z layer_outputs = layer_module( 2025-09-07T07:01:57.2102358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2102443Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2102726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2102808Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2103082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2103179Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2103498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2103622Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2103915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.2104030Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.2104249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.2104329Z return self.act(input) 2025-09-07T07:01:57.2104333Z 2025-09-07T07:01:57.2104436Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2104653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2104719Z return mod(**inputs) 2025-09-07T07:01:57.2105003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2105100Z outputs = self.roberta( 2025-09-07T07:01:57.2105384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2105466Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2105887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2105964Z layer_outputs = layer_module( 2025-09-07T07:01:57.2106198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2106278Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2106569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2106654Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2106926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2107005Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2107338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.2107491Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.2107797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.2107893Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2107899Z 2025-09-07T07:01:57.2108009Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2108247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2108319Z return mod(**inputs) 2025-09-07T07:01:57.2108619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2108720Z outputs = self.roberta( 2025-09-07T07:01:57.2109086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2109165Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2109447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2109519Z layer_outputs = layer_module( 2025-09-07T07:01:57.2109773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2109877Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2110185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2110284Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2110536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2110617Z return func(*args, **kwargs) 2025-09-07T07:01:57.2110900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2110980Z self_outputs = self.self( 2025-09-07T07:01:57.2111227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2111305Z return func(*args, **kwargs) 2025-09-07T07:01:57.2111601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-09-07T07:01:57.2111822Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:01:57.2111826Z 2025-09-07T07:01:57.2111961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2112180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2112258Z return mod(**inputs) 2025-09-07T07:01:57.2112572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2112646Z outputs = self.roberta( 2025-09-07T07:01:57.2112952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2113031Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2113336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2113410Z layer_outputs = layer_module( 2025-09-07T07:01:57.2113658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2113744Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2114043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2114133Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2114377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2114453Z return func(*args, **kwargs) 2025-09-07T07:01:57.2114729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2114802Z self_outputs = self.self( 2025-09-07T07:01:57.2115052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2115122Z return func(*args, **kwargs) 2025-09-07T07:01:57.2115429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-09-07T07:01:57.2115524Z self.key(current_states) 2025-09-07T07:01:57.2115527Z 2025-09-07T07:01:57.2115646Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2115861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2115932Z return mod(**inputs) 2025-09-07T07:01:57.2116236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2116324Z outputs = self.roberta( 2025-09-07T07:01:57.2116622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2116698Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2116996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2117080Z layer_outputs = layer_module( 2025-09-07T07:01:57.2117319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2117413Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2117707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2117794Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2118062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2118139Z return func(*args, **kwargs) 2025-09-07T07:01:57.2118436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2118511Z self_outputs = self.self( 2025-09-07T07:01:57.2118800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2118876Z return func(*args, **kwargs) 2025-09-07T07:01:57.2119188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-09-07T07:01:57.2119273Z self.value(current_states) 2025-09-07T07:01:57.2119277Z 2025-09-07T07:01:57.2119365Z cudagraph partition due to non gpu ops 2025-09-07T07:01:57.2119484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2119825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2119904Z return mod(**inputs) 2025-09-07T07:01:57.2120212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2120286Z outputs = self.roberta( 2025-09-07T07:01:57.2120590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2120670Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2120966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2121051Z layer_outputs = layer_module( 2025-09-07T07:01:57.2121291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2121383Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2121681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2121781Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2122041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2122166Z return func(*args, **kwargs) 2025-09-07T07:01:57.2122470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-09-07T07:01:57.2122543Z self_outputs = self.self( 2025-09-07T07:01:57.2122855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2122926Z return func(*args, **kwargs) 2025-09-07T07:01:57.2123206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-09-07T07:01:57.2123384Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:01:57.2123388Z 2025-09-07T07:01:57.2123499Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2123742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2123815Z return mod(**inputs) 2025-09-07T07:01:57.2124118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2124201Z outputs = self.roberta( 2025-09-07T07:01:57.2124496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2124598Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2124899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2124984Z layer_outputs = layer_module( 2025-09-07T07:01:57.2125225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2125310Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2125641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-09-07T07:01:57.2125731Z self_attention_outputs = self.attention( 2025-09-07T07:01:57.2126023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:01:57.2126099Z return func(*args, **kwargs) 2025-09-07T07:01:57.2126393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-09-07T07:01:57.2126542Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:01:57.2126835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-09-07T07:01:57.2126932Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2126936Z 2025-09-07T07:01:57.2127046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2127267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2127340Z return mod(**inputs) 2025-09-07T07:01:57.2127636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2127717Z outputs = self.roberta( 2025-09-07T07:01:57.2128019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2128103Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2128395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2128474Z layer_outputs = layer_module( 2025-09-07T07:01:57.2128717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2128815Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2129165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2129298Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2129625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2129930Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2130276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2130465Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2130758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-09-07T07:01:57.2130931Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2130935Z 2025-09-07T07:01:57.2131049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2131350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2131437Z return mod(**inputs) 2025-09-07T07:01:57.2131735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2131873Z outputs = self.roberta( 2025-09-07T07:01:57.2132172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2132309Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2132625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2132745Z layer_outputs = layer_module( 2025-09-07T07:01:57.2133009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2133112Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2133442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2133564Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2149946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2150194Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2150614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-09-07T07:01:57.2150772Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:01:57.2151101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-09-07T07:01:57.2151236Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:01:57.2151480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:01:57.2151571Z return self.act(input) 2025-09-07T07:01:57.2151582Z 2025-09-07T07:01:57.2151707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2151942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2152017Z return mod(**inputs) 2025-09-07T07:01:57.2152328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-09-07T07:01:57.2152824Z outputs = self.roberta( 2025-09-07T07:01:57.2153262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-09-07T07:01:57.2153712Z encoder_outputs = self.encoder( 2025-09-07T07:01:57.2154158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-09-07T07:01:57.2154711Z layer_outputs = layer_module( 2025-09-07T07:01:57.2155111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:01:57.2155506Z return super().__call__(*args, **kwargs) 2025-09-07T07:01:57.2155929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-09-07T07:01:57.2156359Z layer_output = apply_chunking_to_forward( 2025-09-07T07:01:57.2156810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:01:57.2157205Z return forward_fn(*input_tensors) 2025-09-07T07:01:57.2157652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-09-07T07:01:57.2158158Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:01:57.2158639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-09-07T07:01:57.2159071Z hidden_states = self.dense(hidden_states) 2025-09-07T07:01:57.2159225Z 2025-09-07T07:01:57.2159335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2159711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2160050Z return mod(**inputs) 2025-09-07T07:01:57.2160452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-09-07T07:01:57.2160966Z prediction_scores = self.lm_head(sequence_output) 2025-09-07T07:01:57.2161456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 756, in forward 2025-09-07T07:01:57.2161873Z x = self.dense(features) 2025-09-07T07:01:57.2162004Z 2025-09-07T07:01:57.2162111Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2162513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2162845Z return mod(**inputs) 2025-09-07T07:01:57.2163241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-09-07T07:01:57.2163694Z prediction_scores = self.lm_head(sequence_output) 2025-09-07T07:01:57.2164139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 761, in forward 2025-09-07T07:01:57.2164558Z x = self.decoder(x) 2025-09-07T07:01:57.2164670Z 2025-09-07T07:01:57.2164786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:01:57.2165179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:01:57.2165515Z return mod(**inputs) 2025-09-07T07:01:57.2165911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1059, in forward 2025-09-07T07:01:57.2166454Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:01:57.2166696Z 2025-09-07T07:02:08.6527916Z Compilation time (from dynamo_timed): 17.966802344 2025-09-07T07:02:08.6607591Z pass 2025-09-07T07:02:08.6608135Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:02:08.6609154Z TIMING: _recursive_pre_grad_passes:0.00718 _recursive_joint_graph_passes:0.38173 _recursive_post_grad_passes:0.07791 async_compile.wait:0.85237 code_gen:10.6869 inductor_compile:11.97569 backend_compile:15.19463 gc:0.00054 entire_frame_compile:17.9668 total_wall_time:17.9668 2025-09-07T07:02:08.6610123Z STATS: call_* op count: 297 | FakeTensorMode.__torch_dispatch__:12430 | FakeTensor.__torch_dispatch__:4399 | ProxyTorchDispatchMode.__torch_dispatch__:4530 2025-09-07T07:02:08.6610978Z Dynamo produced 1 graphs covering 297 ops with 0 graph breaks (0 unique) 2025-09-07T07:02:11.3967172Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:02:11.3968139Z import pynvml # type: ignore[import] 2025-09-07T07:02:14.2277033Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:02:14.2278559Z from pkg_resources import resource_filename 2025-09-07T07:02:14.9305512Z 2025-09-07T07:02:23.9095588Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:02:23.9100634Z loading model: 0it [00:08, ?it/s] 2025-09-07T07:02:23.9121521Z cpu eval DebertaV2ForMaskedLM 2025-09-07T07:02:24.0535976Z Compilation time (from dynamo_timed): 0 2025-09-07T07:02:24.0541251Z pass_due_to_skip 2025-09-07T07:02:24.0542241Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:02:24.0542640Z TIMING: total_wall_time:0 2025-09-07T07:02:24.0542840Z STATS: call_* op count: 0 2025-09-07T07:02:24.0543134Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-09-07T07:02:26.1026586Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:02:26.1027787Z import pynvml # type: ignore[import] 2025-09-07T07:02:28.9359795Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:02:28.9360812Z from pkg_resources import resource_filename 2025-09-07T07:02:29.6127770Z 2025-09-07T07:02:36.9213678Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:02:36.9213979Z loading model: 0it [00:07, ?it/s] 2025-09-07T07:02:36.9243863Z cpu eval DebertaV2ForQuestionAnswering 2025-09-07T07:02:40.2245159Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:02:41.7672454Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:02:43.1189925Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:02:59.0882198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0882957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0883582Z return mod(**inputs) 2025-09-07T07:02:59.0884312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0885061Z outputs = self.deberta( 2025-09-07T07:02:59.0885775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0886510Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0887011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0887466Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0888184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0888575Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0889079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0889783Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0890549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0891044Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0891466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.0892002Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.0892263Z 2025-09-07T07:02:59.0892386Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0892758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0893105Z return mod(**inputs) 2025-09-07T07:02:59.0893509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0894130Z outputs = self.deberta( 2025-09-07T07:02:59.0894818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0895311Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0895749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0896216Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0896652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0897248Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0898085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0898631Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0899106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0899556Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0900206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.0901104Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.0901534Z 2025-09-07T07:02:59.0901725Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0902380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0902946Z return mod(**inputs) 2025-09-07T07:02:59.0903638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0904361Z outputs = self.deberta( 2025-09-07T07:02:59.0905074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0905931Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0906679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0907427Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0908087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0908777Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0909499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0910255Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0911000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0911741Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0912463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.0913449Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.0914479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.0915378Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.0915725Z 2025-09-07T07:02:59.0915916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0916570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0917150Z return mod(**inputs) 2025-09-07T07:02:59.0917846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0918803Z outputs = self.deberta( 2025-09-07T07:02:59.0919503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0920618Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0921456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0922257Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0922947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0923601Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0924354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0925102Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0925899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0926663Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0927436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.0928328Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.0928625Z 2025-09-07T07:02:59.0928748Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0929144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0929488Z return mod(**inputs) 2025-09-07T07:02:59.0929910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0930361Z outputs = self.deberta( 2025-09-07T07:02:59.0930778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0931268Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0931952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0932693Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0933405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0934057Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0934803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0935591Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0936398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0937209Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0937978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.0939020Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.0939526Z 2025-09-07T07:02:59.0939723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0940427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0941057Z return mod(**inputs) 2025-09-07T07:02:59.0941785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0942545Z outputs = self.deberta( 2025-09-07T07:02:59.0943327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0944151Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0944903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0945807Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0946594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0947290Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0948112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0948913Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0949698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0950485Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0951182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.0951762Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.0952026Z 2025-09-07T07:02:59.0952149Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0952607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0952964Z return mod(**inputs) 2025-09-07T07:02:59.0953385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0953838Z outputs = self.deberta( 2025-09-07T07:02:59.0954256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0954690Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0955122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0955575Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0955977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0956400Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0956948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0957736Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0958394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0958847Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0959309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.0959877Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.0960480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.0961271Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.0961626Z 2025-09-07T07:02:59.0961777Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0962168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0962517Z return mod(**inputs) 2025-09-07T07:02:59.0962945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0963390Z outputs = self.deberta( 2025-09-07T07:02:59.0963807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0964238Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0964799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0965592Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0966189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0966588Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0967031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0967498Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0967957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0968409Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0968855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.0969368Z context_layer = torch.bmm( 2025-09-07T07:02:59.0969586Z 2025-09-07T07:02:59.0969763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0970435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0970875Z return mod(**inputs) 2025-09-07T07:02:59.0971287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0972007Z outputs = self.deberta( 2025-09-07T07:02:59.0972736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0973463Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0973905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0974646Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0975392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0976062Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0976825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0977597Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0978389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.0979236Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.0980042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.0981044Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.0981523Z 2025-09-07T07:02:59.0981699Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0982368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0982980Z return mod(**inputs) 2025-09-07T07:02:59.0983666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0984416Z outputs = self.deberta( 2025-09-07T07:02:59.0985126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0985996Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0986744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0987523Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.0988225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.0988893Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.0989679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.0990479Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.0991266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.0992094Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.0992936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.0993703Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.0993960Z 2025-09-07T07:02:59.0994134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.0994802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.0995399Z return mod(**inputs) 2025-09-07T07:02:59.0996097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.0996849Z outputs = self.deberta( 2025-09-07T07:02:59.0997558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.0998326Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.0999071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.0999837Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1000485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1001139Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1001919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1002776Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1003553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1004281Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1004528Z 2025-09-07T07:02:59.1004708Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1005408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1006015Z return mod(**inputs) 2025-09-07T07:02:59.1006724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1007458Z outputs = self.deberta( 2025-09-07T07:02:59.1008146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1008891Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1009602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1010384Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1011030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1011666Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1012424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1013282Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1014167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1014967Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1015698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1016326Z return self.act(input) 2025-09-07T07:02:59.1016526Z 2025-09-07T07:02:59.1016713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1017355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1017950Z return mod(**inputs) 2025-09-07T07:02:59.1018667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1019405Z outputs = self.deberta( 2025-09-07T07:02:59.1020299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1021055Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1021789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1022569Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1023255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1023923Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1024666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1025528Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1026474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1027254Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1027592Z 2025-09-07T07:02:59.1027781Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1028441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1029037Z return mod(**inputs) 2025-09-07T07:02:59.1029747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1030499Z outputs = self.deberta( 2025-09-07T07:02:59.1031213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1032030Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1032773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1033600Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1034293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1034982Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1035701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1036490Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1037243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1037995Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1038714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1039638Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1040135Z 2025-09-07T07:02:59.1040313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1040957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1041592Z return mod(**inputs) 2025-09-07T07:02:59.1042299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1043044Z outputs = self.deberta( 2025-09-07T07:02:59.1043730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1044475Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1045182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1045944Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1046633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1047252Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1047913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1048582Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1049254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1049924Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1050604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1051460Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1051846Z 2025-09-07T07:02:59.1052009Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1052662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1053197Z return mod(**inputs) 2025-09-07T07:02:59.1053940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1054742Z outputs = self.deberta( 2025-09-07T07:02:59.1055510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1056310Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1057153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1058005Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1058752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1059434Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1060216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1060929Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1061619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1062282Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1062983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1063850Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1064829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1065809Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1066143Z 2025-09-07T07:02:59.1066365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1066994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1067536Z return mod(**inputs) 2025-09-07T07:02:59.1068187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1068875Z outputs = self.deberta( 2025-09-07T07:02:59.1069552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1070266Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1070965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1071705Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1072350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1072980Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1073685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1074422Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1075164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1075876Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1076575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1077469Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1077978Z 2025-09-07T07:02:59.1078147Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1078741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1079284Z return mod(**inputs) 2025-09-07T07:02:59.1079959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1080670Z outputs = self.deberta( 2025-09-07T07:02:59.1081358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1082118Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1082825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1083569Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1084219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1084854Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1085573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1086325Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1087075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1087810Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1088543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1089580Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1090084Z 2025-09-07T07:02:59.1090274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1090950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1091514Z return mod(**inputs) 2025-09-07T07:02:59.1092205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1092912Z outputs = self.deberta( 2025-09-07T07:02:59.1093602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1094323Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1095032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1095771Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1096433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1097075Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1097838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1098617Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1099397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1100190Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1100963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1101960Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1102412Z 2025-09-07T07:02:59.1102592Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1103310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1103880Z return mod(**inputs) 2025-09-07T07:02:59.1104549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1105266Z outputs = self.deberta( 2025-09-07T07:02:59.1106055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1106809Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1107531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1108313Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1108975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1109647Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1110422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1111161Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1111900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1112602Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1113288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1114217Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1115288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1116190Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1116524Z 2025-09-07T07:02:59.1116742Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1117415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1118030Z return mod(**inputs) 2025-09-07T07:02:59.1118760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1119498Z outputs = self.deberta( 2025-09-07T07:02:59.1120425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1121208Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1121994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1122783Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1123473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1124125Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1124876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1125727Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1126475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1127221Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1127954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1128692Z context_layer = torch.bmm( 2025-09-07T07:02:59.1129032Z 2025-09-07T07:02:59.1129206Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1129875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1130464Z return mod(**inputs) 2025-09-07T07:02:59.1131169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1131928Z outputs = self.deberta( 2025-09-07T07:02:59.1132602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1133399Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1134143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1134970Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1135682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1136358Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1137095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1137921Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1138744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1139532Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1140329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1141363Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1141906Z 2025-09-07T07:02:59.1142109Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1142807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1143476Z return mod(**inputs) 2025-09-07T07:02:59.1144246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1145041Z outputs = self.deberta( 2025-09-07T07:02:59.1145916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1146752Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1147517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1148322Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1149039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1149739Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1150550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1151377Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1152218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1153046Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1153895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1154648Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1154898Z 2025-09-07T07:02:59.1155075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1155745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1156390Z return mod(**inputs) 2025-09-07T07:02:59.1157100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1157841Z outputs = self.deberta( 2025-09-07T07:02:59.1158565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1159329Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1160095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1160870Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1161545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1162219Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1162978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1163827Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1164662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1165387Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1165641Z 2025-09-07T07:02:59.1165821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1166469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1167028Z return mod(**inputs) 2025-09-07T07:02:59.1167714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1168513Z outputs = self.deberta( 2025-09-07T07:02:59.1169183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1170075Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1170682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1171362Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1172000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1172621Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1173340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1174143Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1174950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1175729Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1176416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1177026Z return self.act(input) 2025-09-07T07:02:59.1177215Z 2025-09-07T07:02:59.1177405Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1178044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1178601Z return mod(**inputs) 2025-09-07T07:02:59.1179273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1179991Z outputs = self.deberta( 2025-09-07T07:02:59.1180689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1181510Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1182247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1183023Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1183707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1184384Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1185210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1186314Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1187241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1188033Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1188279Z 2025-09-07T07:02:59.1188457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1189120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1189725Z return mod(**inputs) 2025-09-07T07:02:59.1190448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1191190Z outputs = self.deberta( 2025-09-07T07:02:59.1191906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1192662Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1193425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1194294Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1194995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1195749Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1196544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1197382Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1198218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1199033Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1199851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1200905Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1201400Z 2025-09-07T07:02:59.1201603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1202311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1202934Z return mod(**inputs) 2025-09-07T07:02:59.1203702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1204497Z outputs = self.deberta( 2025-09-07T07:02:59.1205249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1206046Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1206831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1207642Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1208352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1209106Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1209907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1210752Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1211596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1212428Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1213272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1214276Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1214753Z 2025-09-07T07:02:59.1214955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1215672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1216297Z return mod(**inputs) 2025-09-07T07:02:59.1217077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1217869Z outputs = self.deberta( 2025-09-07T07:02:59.1218639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1219438Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1220468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1221288Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1222104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1222805Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1223667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1224509Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1225358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1226257Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1227102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1228175Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1229320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1230316Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1230698Z 2025-09-07T07:02:59.1230902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1231632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1232275Z return mod(**inputs) 2025-09-07T07:02:59.1233041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1233854Z outputs = self.deberta( 2025-09-07T07:02:59.1234645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1235463Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1236280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1237217Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1237953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1238639Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1239380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1240164Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1240944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1241756Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1242514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1243529Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1244031Z 2025-09-07T07:02:59.1244227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1244915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1245525Z return mod(**inputs) 2025-09-07T07:02:59.1246276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1247062Z outputs = self.deberta( 2025-09-07T07:02:59.1247815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1248615Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1249334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1250143Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1250844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1251579Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1252368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1253195Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1254012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1254808Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1255603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1256682Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1257225Z 2025-09-07T07:02:59.1257427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1258138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1258772Z return mod(**inputs) 2025-09-07T07:02:59.1259515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1260302Z outputs = self.deberta( 2025-09-07T07:02:59.1261040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1261838Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1262618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1263424Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1264201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1264922Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1265860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1266729Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1267570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1268425Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1269213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1270253Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1270742Z 2025-09-07T07:02:59.1270953Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1271646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1272282Z return mod(**inputs) 2025-09-07T07:02:59.1273034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1273827Z outputs = self.deberta( 2025-09-07T07:02:59.1274557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1275360Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1276145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1276974Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1277730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1278394Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1279218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1280061Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1280838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1281605Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1282388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1283385Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1284439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1285429Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1285787Z 2025-09-07T07:02:59.1285994Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1286686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1287320Z return mod(**inputs) 2025-09-07T07:02:59.1288071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1288880Z outputs = self.deberta( 2025-09-07T07:02:59.1289584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1290344Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1291083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1291931Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1292653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1293305Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1294096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1294943Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1295784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1296619Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1297403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1298201Z context_layer = torch.bmm( 2025-09-07T07:02:59.1298439Z 2025-09-07T07:02:59.1298641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1299358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1300000Z return mod(**inputs) 2025-09-07T07:02:59.1300760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1301554Z outputs = self.deberta( 2025-09-07T07:02:59.1302324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1303107Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1303894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1304772Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1305485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1306283Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1307141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1307985Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1308802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1309610Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1310424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1311402Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1311855Z 2025-09-07T07:02:59.1312041Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1312712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1313306Z return mod(**inputs) 2025-09-07T07:02:59.1314018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1314779Z outputs = self.deberta( 2025-09-07T07:02:59.1315508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1316312Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1317051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1317821Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1318486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1319230Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1320162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1320980Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1321819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1322673Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1323601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1324410Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1324676Z 2025-09-07T07:02:59.1324882Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1325603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1326258Z return mod(**inputs) 2025-09-07T07:02:59.1327054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1327851Z outputs = self.deberta( 2025-09-07T07:02:59.1328623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1329409Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1330195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1331017Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1331733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1332504Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1333304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1334254Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1335168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1336020Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1336300Z 2025-09-07T07:02:59.1336499Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1337201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1337843Z return mod(**inputs) 2025-09-07T07:02:59.1338608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1339432Z outputs = self.deberta( 2025-09-07T07:02:59.1340199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1340995Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1341799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1342665Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1343413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1344132Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1344964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1345990Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1346908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1347890Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1348660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1349338Z return self.act(input) 2025-09-07T07:02:59.1349565Z 2025-09-07T07:02:59.1349766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1350493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1351174Z return mod(**inputs) 2025-09-07T07:02:59.1351943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1352751Z outputs = self.deberta( 2025-09-07T07:02:59.1353535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1354331Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1355116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1355917Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1356625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1357337Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1358140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1359045Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1359949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1360824Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1361091Z 2025-09-07T07:02:59.1361299Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1362036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1362657Z return mod(**inputs) 2025-09-07T07:02:59.1363427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1364232Z outputs = self.deberta( 2025-09-07T07:02:59.1364996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1365804Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1366581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1367398Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1368117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1368832Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1369629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1370494Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1371315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1372113Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1372927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1373906Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1374433Z 2025-09-07T07:02:59.1374635Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1375347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1375971Z return mod(**inputs) 2025-09-07T07:02:59.1376730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1377523Z outputs = self.deberta( 2025-09-07T07:02:59.1378283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1379126Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1379923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1380736Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1381456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1382181Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1383027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1383862Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1384705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1385539Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1386484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1387512Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1387962Z 2025-09-07T07:02:59.1388237Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1388956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1389599Z return mod(**inputs) 2025-09-07T07:02:59.1390422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1391242Z outputs = self.deberta( 2025-09-07T07:02:59.1392028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1392838Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1393640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1394484Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1395238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1395976Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1396800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1397662Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1398518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1399358Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1400152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1401219Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1402360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1403411Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1403775Z 2025-09-07T07:02:59.1403989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1404684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1405299Z return mod(**inputs) 2025-09-07T07:02:59.1406056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1406856Z outputs = self.deberta( 2025-09-07T07:02:59.1408903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1409688Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1410468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1411294Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1412030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1412737Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1413548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1414374Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1415205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1416008Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1416800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1417915Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1418458Z 2025-09-07T07:02:59.1418662Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1419405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1420294Z return mod(**inputs) 2025-09-07T07:02:59.1421039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1421825Z outputs = self.deberta( 2025-09-07T07:02:59.1422583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1423385Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1424167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1424987Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1425812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1426535Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1427362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1428191Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1429017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1429838Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1430662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1431742Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1432378Z 2025-09-07T07:02:59.1432592Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1433317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1433965Z return mod(**inputs) 2025-09-07T07:02:59.1434747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1435573Z outputs = self.deberta( 2025-09-07T07:02:59.1436333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1437163Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1437943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1438778Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1439511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1440166Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1440934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1441730Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1442539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1443344Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1444097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1445136Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1445634Z 2025-09-07T07:02:59.1445890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1446606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1447286Z return mod(**inputs) 2025-09-07T07:02:59.1448011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1448773Z outputs = self.deberta( 2025-09-07T07:02:59.1449500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1450266Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1451051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1451867Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1452576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1453236Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1454003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1454821Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1455669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1456473Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1457279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1458315Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1459416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1460466Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1460839Z 2025-09-07T07:02:59.1461040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1461763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1462405Z return mod(**inputs) 2025-09-07T07:02:59.1463169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1464025Z outputs = self.deberta( 2025-09-07T07:02:59.1464814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1465745Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1466572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1467385Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1468110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1468805Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1469601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1470435Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1471261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1472066Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1472855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1473693Z context_layer = torch.bmm( 2025-09-07T07:02:59.1473919Z 2025-09-07T07:02:59.1474119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1474901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1475527Z return mod(**inputs) 2025-09-07T07:02:59.1476298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1477091Z outputs = self.deberta( 2025-09-07T07:02:59.1477838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1478611Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1479357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1480134Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1480846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1481539Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1482337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1483173Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1483942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1484742Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1485527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1486564Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1487059Z 2025-09-07T07:02:59.1487306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1488017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1488632Z return mod(**inputs) 2025-09-07T07:02:59.1489383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1490158Z outputs = self.deberta( 2025-09-07T07:02:59.1490920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1491743Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1492477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1493299Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1494008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1494710Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1495522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1496347Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1497192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1498087Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1498974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1499803Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1500079Z 2025-09-07T07:02:59.1500278Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1501053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1501702Z return mod(**inputs) 2025-09-07T07:02:59.1502513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1503322Z outputs = self.deberta( 2025-09-07T07:02:59.1504111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1504933Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1505863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1506707Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1507441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1508148Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1508956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1509889Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1510789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1511623Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1511914Z 2025-09-07T07:02:59.1512114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1512840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1513493Z return mod(**inputs) 2025-09-07T07:02:59.1514266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1515091Z outputs = self.deberta( 2025-09-07T07:02:59.1515960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1516807Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1517612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1518423Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1519135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1520174Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1520982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1521865Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1522758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1523635Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1524389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1525054Z return self.act(input) 2025-09-07T07:02:59.1525265Z 2025-09-07T07:02:59.1525471Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1526158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1526789Z return mod(**inputs) 2025-09-07T07:02:59.1527550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1528341Z outputs = self.deberta( 2025-09-07T07:02:59.1529179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1529978Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1530812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1531648Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1532357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1533062Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1533856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1534801Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1535674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1536461Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1536712Z 2025-09-07T07:02:59.1536895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1537557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1538150Z return mod(**inputs) 2025-09-07T07:02:59.1538859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1539599Z outputs = self.deberta( 2025-09-07T07:02:59.1540312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1541073Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1541848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1542670Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1543436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1544137Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1544943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1545900Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1546738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1547587Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1548364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1549334Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1549792Z 2025-09-07T07:02:59.1549985Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1550654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1551239Z return mod(**inputs) 2025-09-07T07:02:59.1551949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1552695Z outputs = self.deberta( 2025-09-07T07:02:59.1553409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1554161Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1554902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1555688Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1556426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1557091Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1557868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1558670Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1559453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1560218Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1560988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1561952Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1562400Z 2025-09-07T07:02:59.1562583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1563246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1563832Z return mod(**inputs) 2025-09-07T07:02:59.1564544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1565341Z outputs = self.deberta( 2025-09-07T07:02:59.1566097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1566870Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1567615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1568387Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1569065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1569756Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1570515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1571308Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1572097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1572858Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1573648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1574717Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1575835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1576823Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1577199Z 2025-09-07T07:02:59.1577398Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1578105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1578743Z return mod(**inputs) 2025-09-07T07:02:59.1579448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1580191Z outputs = self.deberta( 2025-09-07T07:02:59.1580950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1581751Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1582571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1583422Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1584164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1584871Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1585762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1586631Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1587499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1588274Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1589025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1590042Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1590537Z 2025-09-07T07:02:59.1590726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1591382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1591984Z return mod(**inputs) 2025-09-07T07:02:59.1592725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1593473Z outputs = self.deberta( 2025-09-07T07:02:59.1594201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1595005Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1595796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1596600Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1597353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1598034Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1598819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1599663Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1600475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1601318Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1602113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1603192Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1603724Z 2025-09-07T07:02:59.1603914Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1604624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1605244Z return mod(**inputs) 2025-09-07T07:02:59.1605988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1606789Z outputs = self.deberta( 2025-09-07T07:02:59.1607546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1608368Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1609150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1609954Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1610662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1611316Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1612128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1612913Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1613683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1614466Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1615254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1616278Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1616737Z 2025-09-07T07:02:59.1616937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1617588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1618176Z return mod(**inputs) 2025-09-07T07:02:59.1618879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1619868Z outputs = self.deberta( 2025-09-07T07:02:59.1620598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1621355Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1622112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1622887Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1623560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1624307Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1625067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1625976Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1626812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1627603Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1628441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1629435Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1630537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1631531Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1631900Z 2025-09-07T07:02:59.1632118Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1632818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1633452Z return mod(**inputs) 2025-09-07T07:02:59.1634215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1635017Z outputs = self.deberta( 2025-09-07T07:02:59.1635768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1636569Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1637407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1638243Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1638985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1639635Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1640392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1641182Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1641983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1642746Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1643501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1644262Z context_layer = torch.bmm( 2025-09-07T07:02:59.1644482Z 2025-09-07T07:02:59.1644669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1645080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1645207Z return mod(**inputs) 2025-09-07T07:02:59.1645772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1645903Z outputs = self.deberta( 2025-09-07T07:02:59.1646454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1646583Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1647139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1647306Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1647762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1647892Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1648404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1648570Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1649076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1649238Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1649751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1650107Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1650116Z 2025-09-07T07:02:59.1650303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1650664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1650782Z return mod(**inputs) 2025-09-07T07:02:59.1651304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1651427Z outputs = self.deberta( 2025-09-07T07:02:59.1651943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1652075Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1652587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1652732Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1653172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1653308Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1653880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1654046Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1654591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1654819Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1655397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1655571Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1655579Z 2025-09-07T07:02:59.1655772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1656186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1656295Z return mod(**inputs) 2025-09-07T07:02:59.1656845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1656976Z outputs = self.deberta( 2025-09-07T07:02:59.1657545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1657687Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1658259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1658414Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1658871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1659019Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1659622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1659854Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1660431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1660581Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1660588Z 2025-09-07T07:02:59.1660788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1661234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1661355Z return mod(**inputs) 2025-09-07T07:02:59.1661933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1662057Z outputs = self.deberta( 2025-09-07T07:02:59.1662612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1662758Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1663323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1663492Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1663950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1664093Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1664665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1664896Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1665491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1665805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1666281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1666405Z return self.act(input) 2025-09-07T07:02:59.1666413Z 2025-09-07T07:02:59.1667593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1668015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1668133Z return mod(**inputs) 2025-09-07T07:02:59.1668720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1668839Z outputs = self.deberta( 2025-09-07T07:02:59.1669394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1669517Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1670032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1670185Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1670595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1670734Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1671238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1671473Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1671995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1672167Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1672174Z 2025-09-07T07:02:59.1672362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1672721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1672838Z return mod(**inputs) 2025-09-07T07:02:59.1673372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1673484Z outputs = self.deberta( 2025-09-07T07:02:59.1673988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1674136Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1674643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1674782Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1675190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1675327Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1675834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1676004Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1676521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1676651Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1677146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1677522Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1677532Z 2025-09-07T07:02:59.1677719Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1678115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1678233Z return mod(**inputs) 2025-09-07T07:02:59.1678745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1678853Z outputs = self.deberta( 2025-09-07T07:02:59.1679367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1679484Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1679990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1680133Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1680544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1680674Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1681194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1681369Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1681880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1682013Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1682518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1682839Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1682846Z 2025-09-07T07:02:59.1683034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1683425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1683537Z return mod(**inputs) 2025-09-07T07:02:59.1684042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1684161Z outputs = self.deberta( 2025-09-07T07:02:59.1684672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1684819Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1685374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1685530Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1685977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1686126Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1686621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1686785Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1687289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1687423Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1687946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1688298Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1688908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1689149Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1689157Z 2025-09-07T07:02:59.1689374Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1689732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1689848Z return mod(**inputs) 2025-09-07T07:02:59.1690360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1690474Z outputs = self.deberta( 2025-09-07T07:02:59.1690992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1691110Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1691637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1691782Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1692197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1692326Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1692827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1692993Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1693511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1693645Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1694156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1694548Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1694594Z 2025-09-07T07:02:59.1694779Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1695148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1695263Z return mod(**inputs) 2025-09-07T07:02:59.1695782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1695900Z outputs = self.deberta( 2025-09-07T07:02:59.1696444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1696570Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1697114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1697270Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1697694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1697827Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1698338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1698503Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1699017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1699156Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1699675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1700106Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1700116Z 2025-09-07T07:02:59.1700310Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1700725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1700846Z return mod(**inputs) 2025-09-07T07:02:59.1701400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1701527Z outputs = self.deberta( 2025-09-07T07:02:59.1702064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1702196Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1702746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1702903Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1703351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1703491Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1704049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1704214Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1704771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1704921Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1705485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1705962Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1705979Z 2025-09-07T07:02:59.1706223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1706632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1706743Z return mod(**inputs) 2025-09-07T07:02:59.1707313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1707446Z outputs = self.deberta( 2025-09-07T07:02:59.1708009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1708175Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1708705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1708856Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1709296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1709434Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1709986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1710150Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1710686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1710827Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1711367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1711734Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1712385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1712638Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1712645Z 2025-09-07T07:02:59.1712860Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1713255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1713356Z return mod(**inputs) 2025-09-07T07:02:59.1713863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1713985Z outputs = self.deberta( 2025-09-07T07:02:59.1714507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1714644Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1715186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1715345Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1715806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1715946Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1716503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1716659Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1717174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1717311Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1717823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1717952Z context_layer = torch.bmm( 2025-09-07T07:02:59.1717998Z 2025-09-07T07:02:59.1718177Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1718475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1718576Z return mod(**inputs) 2025-09-07T07:02:59.1719084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1719197Z outputs = self.deberta( 2025-09-07T07:02:59.1719901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1720125Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1720495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1720598Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1720836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1720917Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1721202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1721300Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1721581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1721671Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1721950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1722156Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1722195Z 2025-09-07T07:02:59.1722310Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1722525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1722621Z return mod(**inputs) 2025-09-07T07:02:59.1722907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1722986Z outputs = self.deberta( 2025-09-07T07:02:59.1723261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1723344Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1723618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1723707Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1723947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1724031Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1724318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1724419Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1724717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1724844Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1725140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1725237Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1725241Z 2025-09-07T07:02:59.1725356Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1725614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1725684Z return mod(**inputs) 2025-09-07T07:02:59.1725986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1726077Z outputs = self.deberta( 2025-09-07T07:02:59.1726352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1726435Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1726735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1726835Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1727071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1727158Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1727458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1727593Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1727892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1727983Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1727987Z 2025-09-07T07:02:59.1728106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1728322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1728392Z return mod(**inputs) 2025-09-07T07:02:59.1728695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1728788Z outputs = self.deberta( 2025-09-07T07:02:59.1729087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1729180Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1729470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1729566Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1729790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1729880Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1730153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1730276Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1730562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1730680Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1730906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1730979Z return self.act(input) 2025-09-07T07:02:59.1730983Z 2025-09-07T07:02:59.1731094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1731299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1731367Z return mod(**inputs) 2025-09-07T07:02:59.1731661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1731733Z outputs = self.deberta( 2025-09-07T07:02:59.1732038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1732134Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1732426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1732525Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1732761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1732853Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1733153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1733323Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1733614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1733702Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1733706Z 2025-09-07T07:02:59.1733822Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1734038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1734115Z return mod(**inputs) 2025-09-07T07:02:59.1734411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1734482Z outputs = self.deberta( 2025-09-07T07:02:59.1734781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1734860Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1735155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1735264Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1735506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1735608Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1735898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1736005Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1736294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1736387Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1736679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1736888Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1736902Z 2025-09-07T07:02:59.1737013Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1737227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1737304Z return mod(**inputs) 2025-09-07T07:02:59.1737599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1737678Z outputs = self.deberta( 2025-09-07T07:02:59.1737969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1738048Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1738346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1738439Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1738684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1738786Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1739076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1739183Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1739473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1739563Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1739899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1740102Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1740106Z 2025-09-07T07:02:59.1740217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1740432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1740510Z return mod(**inputs) 2025-09-07T07:02:59.1740806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1740884Z outputs = self.deberta( 2025-09-07T07:02:59.1741174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1741252Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1741549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1741640Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1741918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1742008Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1742319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1742418Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1742712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1742804Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1743104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1743323Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1743675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1743830Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1743845Z 2025-09-07T07:02:59.1743962Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1744184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1744265Z return mod(**inputs) 2025-09-07T07:02:59.1744571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1744654Z outputs = self.deberta( 2025-09-07T07:02:59.1744954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1745033Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1745343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1745469Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1745899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1745996Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1746294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1746408Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1746709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1746825Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1747125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1747384Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1747390Z 2025-09-07T07:02:59.1747502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1747716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1747796Z return mod(**inputs) 2025-09-07T07:02:59.1748098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1748179Z outputs = self.deberta( 2025-09-07T07:02:59.1748481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1748560Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1748868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1748978Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1749259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1749365Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1749669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1749770Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1750067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1750159Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1750458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1750702Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1750707Z 2025-09-07T07:02:59.1750820Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1751042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1751121Z return mod(**inputs) 2025-09-07T07:02:59.1751424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1751507Z outputs = self.deberta( 2025-09-07T07:02:59.1751804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1751894Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1752192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1752287Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1752559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1752646Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1752959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1753060Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1753364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1753474Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1753772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1753993Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1753998Z 2025-09-07T07:02:59.1754113Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1754339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1754413Z return mod(**inputs) 2025-09-07T07:02:59.1754707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1754784Z outputs = self.deberta( 2025-09-07T07:02:59.1755057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1755137Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1755418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1755510Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1755770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1755857Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1756171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1756265Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1756547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1756625Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1756902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1757102Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1757421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1757566Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1757570Z 2025-09-07T07:02:59.1757676Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1757884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1757952Z return mod(**inputs) 2025-09-07T07:02:59.1758237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1758314Z outputs = self.deberta( 2025-09-07T07:02:59.1758591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1758672Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1758955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1759059Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1759291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1759373Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1759653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1759746Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1760035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1760121Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1760395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1760478Z context_layer = torch.bmm( 2025-09-07T07:02:59.1760482Z 2025-09-07T07:02:59.1760587Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1760798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1760866Z return mod(**inputs) 2025-09-07T07:02:59.1761148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1761224Z outputs = self.deberta( 2025-09-07T07:02:59.1761499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1761580Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1761855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1761959Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1762191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1762271Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1762565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1762659Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1762942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1763023Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1763299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1763501Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1763506Z 2025-09-07T07:02:59.1763612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1763823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1763893Z return mod(**inputs) 2025-09-07T07:02:59.1764173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1764249Z outputs = self.deberta( 2025-09-07T07:02:59.1764535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1764622Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1764910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1765012Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1765251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1765354Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1765653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1765751Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1766049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1766172Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1766485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1766575Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1766578Z 2025-09-07T07:02:59.1766681Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1766892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1766958Z return mod(**inputs) 2025-09-07T07:02:59.1767246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1767314Z outputs = self.deberta( 2025-09-07T07:02:59.1767589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1767669Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1767946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1768039Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1768267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1768363Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1768656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1768800Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1769075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1769157Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1769161Z 2025-09-07T07:02:59.1769267Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1769466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1769530Z return mod(**inputs) 2025-09-07T07:02:59.1769811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1769881Z outputs = self.deberta( 2025-09-07T07:02:59.1770155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1770226Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1770496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1770589Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1770807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1770893Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1771167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1771294Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1771576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1771729Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1771950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1772020Z return self.act(input) 2025-09-07T07:02:59.1772024Z 2025-09-07T07:02:59.1772131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1772328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1772409Z return mod(**inputs) 2025-09-07T07:02:59.1772690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1772756Z outputs = self.deberta( 2025-09-07T07:02:59.1773029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1773100Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1773374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1773459Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1773677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1773763Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1774028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1774169Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1774458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1774541Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1774544Z 2025-09-07T07:02:59.1774652Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1774869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1774943Z return mod(**inputs) 2025-09-07T07:02:59.1775235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1775314Z outputs = self.deberta( 2025-09-07T07:02:59.1775609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1775689Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1775990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1776081Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1776328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1776416Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1776718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1776822Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1777100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1777188Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1777469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1777666Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1777688Z 2025-09-07T07:02:59.1777793Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1777994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1778069Z return mod(**inputs) 2025-09-07T07:02:59.1778340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1778414Z outputs = self.deberta( 2025-09-07T07:02:59.1778682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1778769Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1779046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1779132Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1779362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1779443Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1779714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1779814Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1780081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1780168Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1780437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1780624Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1780627Z 2025-09-07T07:02:59.1780746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1780949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1781040Z return mod(**inputs) 2025-09-07T07:02:59.1781321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1781396Z outputs = self.deberta( 2025-09-07T07:02:59.1781671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1781752Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1782050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1782143Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1782390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1782476Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1782783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1782881Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1783180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1783270Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1783571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1783780Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1784125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1784295Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1784300Z 2025-09-07T07:02:59.1784413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1784628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1784708Z return mod(**inputs) 2025-09-07T07:02:59.1785016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1785113Z outputs = self.deberta( 2025-09-07T07:02:59.1785426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1785504Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1785950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1786056Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1786312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1786402Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1786722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1786825Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1787135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1787244Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1787544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1787810Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1787816Z 2025-09-07T07:02:59.1787930Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1788174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1788256Z return mod(**inputs) 2025-09-07T07:02:59.1788560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1788643Z outputs = self.deberta( 2025-09-07T07:02:59.1788948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1789036Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1789337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1789430Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1789678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1789764Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1790073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1790171Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1790470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1790562Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1790852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1791085Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1791107Z 2025-09-07T07:02:59.1791219Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1791438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1791508Z return mod(**inputs) 2025-09-07T07:02:59.1791802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1791883Z outputs = self.deberta( 2025-09-07T07:02:59.1792185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1792289Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1792582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1792676Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1792923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1793009Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1793306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1793404Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1793703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1793786Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1794074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1794287Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1794318Z 2025-09-07T07:02:59.1794429Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1794648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1794737Z return mod(**inputs) 2025-09-07T07:02:59.1795038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1795129Z outputs = self.deberta( 2025-09-07T07:02:59.1795402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1795486Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1795760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1795857Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1796097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1796183Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1796482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1796587Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1796867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1796945Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1797223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1797422Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1797747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1797920Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1797924Z 2025-09-07T07:02:59.1798037Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1798259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1798329Z return mod(**inputs) 2025-09-07T07:02:59.1798626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1798724Z outputs = self.deberta( 2025-09-07T07:02:59.1799015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1799102Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1799397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1799497Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1799747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1799827Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1800107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1800200Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1800484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1800562Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1800839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1800937Z context_layer = torch.bmm( 2025-09-07T07:02:59.1800941Z 2025-09-07T07:02:59.1801045Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1801282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1801349Z return mod(**inputs) 2025-09-07T07:02:59.1801636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1801714Z outputs = self.deberta( 2025-09-07T07:02:59.1801992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1802073Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1802357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1802454Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1802680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1802764Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1803047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1803139Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1803417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1803496Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1803769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1803976Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1803981Z 2025-09-07T07:02:59.1804103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1804314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1804383Z return mod(**inputs) 2025-09-07T07:02:59.1804671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1804739Z outputs = self.deberta( 2025-09-07T07:02:59.1805016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1805118Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1805408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1805505Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1805744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1805830Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1806127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1806226Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1806520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1806645Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1806947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1807036Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1807039Z 2025-09-07T07:02:59.1807155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1807382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1807448Z return mod(**inputs) 2025-09-07T07:02:59.1807747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1807817Z outputs = self.deberta( 2025-09-07T07:02:59.1808092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1808173Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1808456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1808550Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1808775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1808870Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1809159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1809292Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1809587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1809674Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1809677Z 2025-09-07T07:02:59.1809793Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1810007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1810078Z return mod(**inputs) 2025-09-07T07:02:59.1810377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1810471Z outputs = self.deberta( 2025-09-07T07:02:59.1810776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1810861Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1811144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1811231Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1811461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1811573Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1811867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1812006Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1812302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1812427Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1812670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1812747Z return self.act(input) 2025-09-07T07:02:59.1812751Z 2025-09-07T07:02:59.1812867Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1813083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1813163Z return mod(**inputs) 2025-09-07T07:02:59.1813463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1813538Z outputs = self.deberta( 2025-09-07T07:02:59.1813864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1813945Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1814272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1814365Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1814607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1814700Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1814992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1815163Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1815458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1815555Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1815559Z 2025-09-07T07:02:59.1815669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1815882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1815963Z return mod(**inputs) 2025-09-07T07:02:59.1816260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1816340Z outputs = self.deberta( 2025-09-07T07:02:59.1816630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1816710Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1817004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1817096Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1817367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1817453Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1817759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1817861Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1818159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1818275Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1818567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1818780Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1818785Z 2025-09-07T07:02:59.1818896Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1819112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1819191Z return mod(**inputs) 2025-09-07T07:02:59.1819489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1819741Z outputs = self.deberta( 2025-09-07T07:02:59.1820144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1820236Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1820537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1820630Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1820934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1821024Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1821366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1821467Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1821769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1821860Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1822151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1822354Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1822359Z 2025-09-07T07:02:59.1822469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1822690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1822764Z return mod(**inputs) 2025-09-07T07:02:59.1823067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1823151Z outputs = self.deberta( 2025-09-07T07:02:59.1823452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1823538Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1823840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1823932Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1824179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1824293Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1824595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1824695Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1824999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1825081Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1825381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1825690Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1826047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1826201Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1826205Z 2025-09-07T07:02:59.1826317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1826532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1826609Z return mod(**inputs) 2025-09-07T07:02:59.1826912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1826995Z outputs = self.deberta( 2025-09-07T07:02:59.1827298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1827382Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1827709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1827804Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1828069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1828157Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1828457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1828557Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1828850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1828944Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1829235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1829474Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1829480Z 2025-09-07T07:02:59.1829589Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1829823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1829892Z return mod(**inputs) 2025-09-07T07:02:59.1830199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1830280Z outputs = self.deberta( 2025-09-07T07:02:59.1830573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1830657Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1830949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1831064Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1831308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1831395Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1831692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1831791Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1832091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1832189Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1832474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1832693Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1832697Z 2025-09-07T07:02:59.1832801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1833012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1833078Z return mod(**inputs) 2025-09-07T07:02:59.1833356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1833434Z outputs = self.deberta( 2025-09-07T07:02:59.1833706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1833789Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1834069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1834184Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1834409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1834489Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1834793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1834887Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1835164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1835243Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1835530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1835745Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1835750Z 2025-09-07T07:02:59.1835861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1836086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1836152Z return mod(**inputs) 2025-09-07T07:02:59.1836440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1836509Z outputs = self.deberta( 2025-09-07T07:02:59.1836783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1836865Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1837138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1837232Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1837464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1837566Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1837848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1837942Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1838223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1838300Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1838596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1838789Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1839106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1839250Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1839254Z 2025-09-07T07:02:59.1839357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1839569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1839633Z return mod(**inputs) 2025-09-07T07:02:59.1839918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1839988Z outputs = self.deberta( 2025-09-07T07:02:59.1840261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1840339Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1840634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1840731Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1840970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1841053Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1841338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1841431Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1841713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1841793Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1842077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1842151Z context_layer = torch.bmm( 2025-09-07T07:02:59.1842155Z 2025-09-07T07:02:59.1842259Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1842469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1842535Z return mod(**inputs) 2025-09-07T07:02:59.1842822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1842891Z outputs = self.deberta( 2025-09-07T07:02:59.1843163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1843246Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1843524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1843620Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1843874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1843957Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1844239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1844330Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1844613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1844726Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1845009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1845202Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1845208Z 2025-09-07T07:02:59.1845312Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1845522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1845589Z return mod(**inputs) 2025-09-07T07:02:59.1845876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1845944Z outputs = self.deberta( 2025-09-07T07:02:59.1846220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1846303Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1846579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1846673Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1846911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1847012Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1847294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1847386Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1847665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1847781Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1848062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1848145Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1848148Z 2025-09-07T07:02:59.1848261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1848465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1848530Z return mod(**inputs) 2025-09-07T07:02:59.1848826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1848893Z outputs = self.deberta( 2025-09-07T07:02:59.1849167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1849237Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1849510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1849602Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1849818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1849925Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1850200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1850322Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1850606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1850689Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1850693Z 2025-09-07T07:02:59.1850802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1851023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1851096Z return mod(**inputs) 2025-09-07T07:02:59.1851376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1851444Z outputs = self.deberta( 2025-09-07T07:02:59.1851726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1851802Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1852085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1852171Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1852403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1852493Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1852761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1852886Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1853181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1853306Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1853536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1853609Z return self.act(input) 2025-09-07T07:02:59.1853612Z 2025-09-07T07:02:59.1853726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1853932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1854006Z return mod(**inputs) 2025-09-07T07:02:59.1854285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1854354Z outputs = self.deberta( 2025-09-07T07:02:59.1854637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1854712Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1855005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1855096Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1855339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1855425Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1855716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1855869Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1856164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1856280Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1856284Z 2025-09-07T07:02:59.1856394Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1856611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1856689Z return mod(**inputs) 2025-09-07T07:02:59.1856984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1857062Z outputs = self.deberta( 2025-09-07T07:02:59.1857352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1857455Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1857752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1857845Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1858091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1858179Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1858475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1858574Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1858866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1858958Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1859248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1859477Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1859483Z 2025-09-07T07:02:59.1859596Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1859836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1859908Z return mod(**inputs) 2025-09-07T07:02:59.1860210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1860291Z outputs = self.deberta( 2025-09-07T07:02:59.1860584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1860671Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1860964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1861056Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1861301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1861389Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1861692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1861792Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1862094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1862177Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1862479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1862690Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1862694Z 2025-09-07T07:02:59.1862808Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1863054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1863129Z return mod(**inputs) 2025-09-07T07:02:59.1863434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1863516Z outputs = self.deberta( 2025-09-07T07:02:59.1863818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1863924Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1864235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1864336Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1864585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1864676Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1864984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1865086Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1865395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1865478Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1865852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1866079Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1866452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1866614Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1866618Z 2025-09-07T07:02:59.1866757Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1866999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1867069Z return mod(**inputs) 2025-09-07T07:02:59.1867369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1867454Z outputs = self.deberta( 2025-09-07T07:02:59.1867747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1867833Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1868144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1868237Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1868487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1868573Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1868875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1868973Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1869273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1869357Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1869650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1869889Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1869916Z 2025-09-07T07:02:59.1870028Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1870254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1870324Z return mod(**inputs) 2025-09-07T07:02:59.1870621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1870702Z outputs = self.deberta( 2025-09-07T07:02:59.1871009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1871092Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1871385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1871487Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1871724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1871811Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1872111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1872210Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1872507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1872590Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1872882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1873130Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1873136Z 2025-09-07T07:02:59.1873248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1873486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1873556Z return mod(**inputs) 2025-09-07T07:02:59.1873858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1873929Z outputs = self.deberta( 2025-09-07T07:02:59.1874220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1874304Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1874599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1874698Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1874934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1875019Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1875316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1875414Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1875709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1875791Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1876090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1876295Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1876300Z 2025-09-07T07:02:59.1876444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1876666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1876736Z return mod(**inputs) 2025-09-07T07:02:59.1877040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1877112Z outputs = self.deberta( 2025-09-07T07:02:59.1877405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1877507Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1877797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1877896Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1878137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1878232Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1878529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1878627Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1878926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1879006Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1879307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1879511Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1879854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1879992Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1879996Z 2025-09-07T07:02:59.1880122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1880349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1880418Z return mod(**inputs) 2025-09-07T07:02:59.1880731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1880803Z outputs = self.deberta( 2025-09-07T07:02:59.1881082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1881163Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1881440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1881534Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1881765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1881857Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1882148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1882244Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1882564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1882646Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1882946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1883041Z context_layer = torch.bmm( 2025-09-07T07:02:59.1883045Z 2025-09-07T07:02:59.1883157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1883386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1883455Z return mod(**inputs) 2025-09-07T07:02:59.1883774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1883843Z outputs = self.deberta( 2025-09-07T07:02:59.1884129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1884220Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1884505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1884603Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1884837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1884925Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1885212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1885311Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1885608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1885692Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1885991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1886193Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1886216Z 2025-09-07T07:02:59.1886340Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1886561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1886650Z return mod(**inputs) 2025-09-07T07:02:59.1886957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1887025Z outputs = self.deberta( 2025-09-07T07:02:59.1887306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1887381Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1887656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1887750Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1887976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1888065Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1888344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1888448Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1888720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1888838Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1889124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1889207Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1889210Z 2025-09-07T07:02:59.1889323Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1889539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1889605Z return mod(**inputs) 2025-09-07T07:02:59.1889901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1889968Z outputs = self.deberta( 2025-09-07T07:02:59.1890252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1890327Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1890630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1890728Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1890963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1891058Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1891357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1891492Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1891781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1891867Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1891870Z 2025-09-07T07:02:59.1891986Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1892200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1892276Z return mod(**inputs) 2025-09-07T07:02:59.1892597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1892672Z outputs = self.deberta( 2025-09-07T07:02:59.1892972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1893065Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1893365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1893457Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1893703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1893786Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1894060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1894189Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1894465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1894591Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1894806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1894877Z return self.act(input) 2025-09-07T07:02:59.1894888Z 2025-09-07T07:02:59.1894992Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1895193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1895269Z return mod(**inputs) 2025-09-07T07:02:59.1895555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1895631Z outputs = self.deberta( 2025-09-07T07:02:59.1895911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1896004Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1896286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1896372Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1896610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1896694Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1896985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1897151Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1897445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1897542Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1897546Z 2025-09-07T07:02:59.1897656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1897879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1897948Z return mod(**inputs) 2025-09-07T07:02:59.1898244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1898323Z outputs = self.deberta( 2025-09-07T07:02:59.1898614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1898695Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1898972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1899074Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1899312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1899410Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1899699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1899800Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1900097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1900183Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1900473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1900690Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1900695Z 2025-09-07T07:02:59.1900803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1901024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1901093Z return mod(**inputs) 2025-09-07T07:02:59.1901388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1901467Z outputs = self.deberta( 2025-09-07T07:02:59.1901756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1901843Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1902134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1902232Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1902468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1902575Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1902874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1902974Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1903272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1903374Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1903665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1903868Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1903872Z 2025-09-07T07:02:59.1903982Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1904209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1904280Z return mod(**inputs) 2025-09-07T07:02:59.1904585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1904656Z outputs = self.deberta( 2025-09-07T07:02:59.1904952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1905038Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1905335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1905434Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1905773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1905869Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1906207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1906311Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1906620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1906706Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1907017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1907224Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1907570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1907726Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1907730Z 2025-09-07T07:02:59.1907844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1908072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1908145Z return mod(**inputs) 2025-09-07T07:02:59.1908449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1908534Z outputs = self.deberta( 2025-09-07T07:02:59.1908831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1908916Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1909214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1909335Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1909586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1909672Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1909983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1910082Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1910390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1910499Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1910798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1911042Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1911047Z 2025-09-07T07:02:59.1911161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1911401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1911467Z return mod(**inputs) 2025-09-07T07:02:59.1911752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1911821Z outputs = self.deberta( 2025-09-07T07:02:59.1912098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1912182Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1912458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1912575Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1912800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1912895Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1913178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1913268Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1913549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1913628Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1913909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1914122Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1914127Z 2025-09-07T07:02:59.1914231Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1914444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1914509Z return mod(**inputs) 2025-09-07T07:02:59.1914798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1914867Z outputs = self.deberta( 2025-09-07T07:02:59.1915151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1915225Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1915501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1915596Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1915834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1915921Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1916195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1916289Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1916572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1916679Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1916961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1917157Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1917162Z 2025-09-07T07:02:59.1917272Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1917473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1917544Z return mod(**inputs) 2025-09-07T07:02:59.1917834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1917901Z outputs = self.deberta( 2025-09-07T07:02:59.1918183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1918259Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1918532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1918628Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1918868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1918958Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1919250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1919352Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1919899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1920001Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1920290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1920482Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1920810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1920946Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1920950Z 2025-09-07T07:02:59.1921057Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1921269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1921337Z return mod(**inputs) 2025-09-07T07:02:59.1921625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1921696Z outputs = self.deberta( 2025-09-07T07:02:59.1921983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1922057Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1922334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1922486Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1922716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1922803Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1923079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1923171Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1923479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1923556Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1923837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1923911Z context_layer = torch.bmm( 2025-09-07T07:02:59.1923914Z 2025-09-07T07:02:59.1924023Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1924226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1924293Z return mod(**inputs) 2025-09-07T07:02:59.1924592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1924664Z outputs = self.deberta( 2025-09-07T07:02:59.1924961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1925041Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1925347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1925473Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1925712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1925828Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1926131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1926239Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1926541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1926624Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1926930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1927124Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1927130Z 2025-09-07T07:02:59.1927243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1927447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1927515Z return mod(**inputs) 2025-09-07T07:02:59.1927804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1927873Z outputs = self.deberta( 2025-09-07T07:02:59.1928219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1928297Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1928614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1928708Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1928948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1929063Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1929365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1929472Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1929771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1929895Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1930216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1930301Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1930305Z 2025-09-07T07:02:59.1930417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1930624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1930701Z return mod(**inputs) 2025-09-07T07:02:59.1931002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1931073Z outputs = self.deberta( 2025-09-07T07:02:59.1931378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1931454Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1931771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1931863Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1932119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1932215Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1932573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1932711Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1933014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1933110Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1933114Z 2025-09-07T07:02:59.1933222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1933438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1933514Z return mod(**inputs) 2025-09-07T07:02:59.1933822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1933901Z outputs = self.deberta( 2025-09-07T07:02:59.1934204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1934283Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1934591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1934685Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1934929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1935015Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1935307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1935441Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1935734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1935886Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1936116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1936197Z return self.act(input) 2025-09-07T07:02:59.1936200Z 2025-09-07T07:02:59.1936312Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1936528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1936622Z return mod(**inputs) 2025-09-07T07:02:59.1936923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1937000Z outputs = self.deberta( 2025-09-07T07:02:59.1937294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1937372Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1937672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1937762Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1938008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1938093Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1938393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1938539Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1938850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1938948Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1938952Z 2025-09-07T07:02:59.1939061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1939299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1939371Z return mod(**inputs) 2025-09-07T07:02:59.1939668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1939746Z outputs = self.deberta( 2025-09-07T07:02:59.1940038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1940121Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1940413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1940511Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1940749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1940835Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1941132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1941231Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1941525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1941608Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1941897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1942109Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1942138Z 2025-09-07T07:02:59.1942249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1942474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1942544Z return mod(**inputs) 2025-09-07T07:02:59.1942854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1942925Z outputs = self.deberta( 2025-09-07T07:02:59.1943229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1943334Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1943629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1943729Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1943967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1944053Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1944356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1944456Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1944757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1944842Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1945143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1945338Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1945341Z 2025-09-07T07:02:59.1945471Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1945775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1945880Z return mod(**inputs) 2025-09-07T07:02:59.1946199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1946273Z outputs = self.deberta( 2025-09-07T07:02:59.1946572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1946661Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1946961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1947069Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1947295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1947385Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1947663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1947757Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1948045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1948124Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1948409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1948601Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1948922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1949111Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1949115Z 2025-09-07T07:02:59.1949220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1949445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1949510Z return mod(**inputs) 2025-09-07T07:02:59.1949801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1949885Z outputs = self.deberta( 2025-09-07T07:02:59.1950164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1950245Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1950524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1950619Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1950845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1950926Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1951209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1951301Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1951589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1951670Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1951957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1952187Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1952192Z 2025-09-07T07:02:59.1952298Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1952552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1952620Z return mod(**inputs) 2025-09-07T07:02:59.1952906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1952973Z outputs = self.deberta( 2025-09-07T07:02:59.1953262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1953334Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1953610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1953706Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1953935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1954026Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1954304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1954397Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1954682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1954761Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1955043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1955255Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1955274Z 2025-09-07T07:02:59.1955385Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1955592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1955659Z return mod(**inputs) 2025-09-07T07:02:59.1955947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1956015Z outputs = self.deberta( 2025-09-07T07:02:59.1956302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1956391Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1956669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1956763Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1956990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1957076Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1957356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1957456Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1957739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1957816Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1958103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1958304Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1958326Z 2025-09-07T07:02:59.1958446Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1958661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1958746Z return mod(**inputs) 2025-09-07T07:02:59.1959053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1959127Z outputs = self.deberta( 2025-09-07T07:02:59.1959426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1959515Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1959798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1959886Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1960112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1960201Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1960483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1960584Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1960874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1960956Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1961258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.1961462Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1961807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1961967Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1961973Z 2025-09-07T07:02:59.1962089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1962304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1962375Z return mod(**inputs) 2025-09-07T07:02:59.1962676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1962768Z outputs = self.deberta( 2025-09-07T07:02:59.1963069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1963148Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1963440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1963541Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1963783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1963874Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1964168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1964273Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1964566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1964648Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1964969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.1965048Z context_layer = torch.bmm( 2025-09-07T07:02:59.1965051Z 2025-09-07T07:02:59.1965171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1965410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1965483Z return mod(**inputs) 2025-09-07T07:02:59.1965788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1965860Z outputs = self.deberta( 2025-09-07T07:02:59.1966169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1966248Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1966554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1966646Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1966891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1966985Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1967283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1967389Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1967685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1967769Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1968073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.1968280Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.1968305Z 2025-09-07T07:02:59.1968426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1968642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1968718Z return mod(**inputs) 2025-09-07T07:02:59.1969017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1969090Z outputs = self.deberta( 2025-09-07T07:02:59.1969390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1969485Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1969787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1969880Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1970118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1970215Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1970506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1970612Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1970903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.1971036Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.1971332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.1971421Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1971425Z 2025-09-07T07:02:59.1971561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1971778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1971855Z return mod(**inputs) 2025-09-07T07:02:59.1972173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1972248Z outputs = self.deberta( 2025-09-07T07:02:59.1972549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1972625Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1972933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1973023Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1973269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1973355Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1973648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1973787Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1974077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.1974172Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1974176Z 2025-09-07T07:02:59.1974284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1974500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1974578Z return mod(**inputs) 2025-09-07T07:02:59.1974871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1974967Z outputs = self.deberta( 2025-09-07T07:02:59.1975263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1975351Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1975628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1975714Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1975949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1976043Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1976327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.1976452Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.1976746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.1976878Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.1977108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.1977190Z return self.act(input) 2025-09-07T07:02:59.1977193Z 2025-09-07T07:02:59.1977301Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1977523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1977595Z return mod(**inputs) 2025-09-07T07:02:59.1977892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1977972Z outputs = self.deberta( 2025-09-07T07:02:59.1978277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1978371Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1978660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1978748Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1978982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1979062Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1979348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.1979489Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.1979811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.1979933Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.1979939Z 2025-09-07T07:02:59.1980093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1980324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1980395Z return mod(**inputs) 2025-09-07T07:02:59.1980698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1980772Z outputs = self.deberta( 2025-09-07T07:02:59.1981066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1981152Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1981545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1981680Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1982068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1982177Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1982564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1982668Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1983086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1983210Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1983510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1983714Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1983719Z 2025-09-07T07:02:59.1983832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1984063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1984135Z return mod(**inputs) 2025-09-07T07:02:59.1984454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1984527Z outputs = self.deberta( 2025-09-07T07:02:59.1984824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1984911Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1985210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1985311Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1985648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1985765Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1986085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1986190Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1986499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1986588Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1986896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.1987098Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.1987102Z 2025-09-07T07:02:59.1987225Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1987448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1987522Z return mod(**inputs) 2025-09-07T07:02:59.1987850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1987922Z outputs = self.deberta( 2025-09-07T07:02:59.1988228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1988309Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1988602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1988702Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1988942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1989071Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1989506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1989633Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1990106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1990211Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1990701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.1990995Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.1991537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.1991734Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.1991739Z 2025-09-07T07:02:59.1991889Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1992217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1992304Z return mod(**inputs) 2025-09-07T07:02:59.1992783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1992874Z outputs = self.deberta( 2025-09-07T07:02:59.1993338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1993437Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1993902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1994035Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1994306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1994400Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1994758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1994859Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1995166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1995253Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1995562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1995804Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1995810Z 2025-09-07T07:02:59.1995929Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1996151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1996223Z return mod(**inputs) 2025-09-07T07:02:59.1996543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1996617Z outputs = self.deberta( 2025-09-07T07:02:59.1996930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.1997010Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.1997314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.1997439Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.1997686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.1997783Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.1998080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.1998189Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.1998491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.1998594Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.1998901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.1999136Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.1999141Z 2025-09-07T07:02:59.1999263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.1999483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.1999555Z return mod(**inputs) 2025-09-07T07:02:59.1999870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.1999945Z outputs = self.deberta( 2025-09-07T07:02:59.2000253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2000333Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2000642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2000757Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2001005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2001117Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2001418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2001525Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2001836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2001919Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2002220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2002427Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2002434Z 2025-09-07T07:02:59.2002556Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2002775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2002856Z return mod(**inputs) 2025-09-07T07:02:59.2003174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2003246Z outputs = self.deberta( 2025-09-07T07:02:59.2003543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2003622Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2003916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2004007Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2004244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2004354Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2004655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2004766Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2005065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2005154Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2005470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2005677Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2006034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2006185Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2006191Z 2025-09-07T07:02:59.2006322Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2006535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2006606Z return mod(**inputs) 2025-09-07T07:02:59.2006914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2006990Z outputs = self.deberta( 2025-09-07T07:02:59.2007299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2007377Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2007709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2007805Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2008067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2008160Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2008453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2008556Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2008870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2008952Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2009265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2009342Z context_layer = torch.bmm( 2025-09-07T07:02:59.2009345Z 2025-09-07T07:02:59.2009461Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2009690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2009766Z return mod(**inputs) 2025-09-07T07:02:59.2010074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2010147Z outputs = self.deberta( 2025-09-07T07:02:59.2010445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2010523Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2010834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2010927Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2011183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2011278Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2011577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2011680Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2011980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2012084Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2012380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2012592Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2012597Z 2025-09-07T07:02:59.2012712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2012938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2013015Z return mod(**inputs) 2025-09-07T07:02:59.2013322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2013393Z outputs = self.deberta( 2025-09-07T07:02:59.2013703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2013782Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2014085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2014176Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2014443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2014533Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2014866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2014979Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2015290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2015427Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2015729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2015822Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2015833Z 2025-09-07T07:02:59.2015947Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2016168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2016246Z return mod(**inputs) 2025-09-07T07:02:59.2016565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2016647Z outputs = self.deberta( 2025-09-07T07:02:59.2016955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2017033Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2017339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2017434Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2017686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2017792Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2018105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2018246Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2018557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2018655Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2018658Z 2025-09-07T07:02:59.2018770Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2019022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2019093Z return mod(**inputs) 2025-09-07T07:02:59.2019407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2019503Z outputs = self.deberta( 2025-09-07T07:02:59.2020070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2020171Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2020476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2020571Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2020829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2020920Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2021228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2021360Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2021721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2021878Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2022115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2022201Z return self.act(input) 2025-09-07T07:02:59.2022205Z 2025-09-07T07:02:59.2022318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2022546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2022619Z return mod(**inputs) 2025-09-07T07:02:59.2022922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2023003Z outputs = self.deberta( 2025-09-07T07:02:59.2023310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2023399Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2023698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2023792Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2024043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2024131Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2024439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2024589Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2024896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2025011Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2025015Z 2025-09-07T07:02:59.2025126Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2025354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2025426Z return mod(**inputs) 2025-09-07T07:02:59.2025844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2025923Z outputs = self.deberta( 2025-09-07T07:02:59.2026252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2026338Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2026637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2026739Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2026984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2027081Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2027380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2027482Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2027788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2027876Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2028180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2028405Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2028411Z 2025-09-07T07:02:59.2028533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2028771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2028843Z return mod(**inputs) 2025-09-07T07:02:59.2029167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2029240Z outputs = self.deberta( 2025-09-07T07:02:59.2029545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2029624Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2029924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2030025Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2030273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2030369Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2030667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2030770Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2031078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2031163Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2031468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2031667Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2031670Z 2025-09-07T07:02:59.2031784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2032005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2032073Z return mod(**inputs) 2025-09-07T07:02:59.2032363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2032432Z outputs = self.deberta( 2025-09-07T07:02:59.2032713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2032801Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2033081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2033176Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2033404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2033495Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2033773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2033873Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2034149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2034226Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2034510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2034697Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2035039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2035178Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2035182Z 2025-09-07T07:02:59.2035315Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2035532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2035602Z return mod(**inputs) 2025-09-07T07:02:59.2035918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2035988Z outputs = self.deberta( 2025-09-07T07:02:59.2036279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2036350Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2036627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2036723Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2036950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2037037Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2037309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2037408Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2037684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2037762Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2038048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2038268Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2038289Z 2025-09-07T07:02:59.2038402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2038604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2038671Z return mod(**inputs) 2025-09-07T07:02:59.2038967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2039035Z outputs = self.deberta( 2025-09-07T07:02:59.2039332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2039405Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2039687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2039774Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2039997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2040089Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2040367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2040469Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2040746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2040826Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2041109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2041337Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2041342Z 2025-09-07T07:02:59.2041456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2041675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2041749Z return mod(**inputs) 2025-09-07T07:02:59.2042029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2042097Z outputs = self.deberta( 2025-09-07T07:02:59.2042380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2042454Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2042752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2042846Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2043087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2043180Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2043471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2043575Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2043870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2043958Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2044248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2044454Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2044485Z 2025-09-07T07:02:59.2044603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2044817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2044895Z return mod(**inputs) 2025-09-07T07:02:59.2045193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2045265Z outputs = self.deberta( 2025-09-07T07:02:59.2045561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2045655Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2045955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2046048Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2046295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2046382Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2046677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2046783Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2047078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2047167Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2047462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2047665Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2048029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2048174Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2048192Z 2025-09-07T07:02:59.2048312Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2048527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2048604Z return mod(**inputs) 2025-09-07T07:02:59.2048904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2048978Z outputs = self.deberta( 2025-09-07T07:02:59.2049279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2049358Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2049658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2049752Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2049998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2050092Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2050382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2050486Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2050780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2050871Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2051169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2051263Z context_layer = torch.bmm( 2025-09-07T07:02:59.2051267Z 2025-09-07T07:02:59.2051386Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2051598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2051677Z return mod(**inputs) 2025-09-07T07:02:59.2051977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2052048Z outputs = self.deberta( 2025-09-07T07:02:59.2052355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2052449Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2052747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2052839Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2053086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2053173Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2053466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2053572Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2053868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2053958Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2054249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2054477Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2054489Z 2025-09-07T07:02:59.2054600Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2054845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2054923Z return mod(**inputs) 2025-09-07T07:02:59.2055237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2055316Z outputs = self.deberta( 2025-09-07T07:02:59.2055607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2055685Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2055987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2056078Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2056325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2056412Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2056706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2056811Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2057106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2057242Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2057539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2057637Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2057640Z 2025-09-07T07:02:59.2057752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2057986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2058063Z return mod(**inputs) 2025-09-07T07:02:59.2058364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2058443Z outputs = self.deberta( 2025-09-07T07:02:59.2058737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2058812Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2059133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2059224Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2059469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2059554Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2059852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2059982Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2060271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2060368Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2060372Z 2025-09-07T07:02:59.2060478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2060702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2060771Z return mod(**inputs) 2025-09-07T07:02:59.2061092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2061173Z outputs = self.deberta( 2025-09-07T07:02:59.2061480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2061566Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2061865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2061962Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2062199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2062285Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2062585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2062714Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2063019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2063143Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2063374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2063455Z return self.act(input) 2025-09-07T07:02:59.2063459Z 2025-09-07T07:02:59.2063569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2063790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2063863Z return mod(**inputs) 2025-09-07T07:02:59.2064169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2064241Z outputs = self.deberta( 2025-09-07T07:02:59.2064536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2064639Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2064932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2065031Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2065269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2065352Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2065748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2065900Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2066209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2066303Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2066307Z 2025-09-07T07:02:59.2066428Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2066650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2066726Z return mod(**inputs) 2025-09-07T07:02:59.2067046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2067119Z outputs = self.deberta( 2025-09-07T07:02:59.2067420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2067497Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2067792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2067914Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2068156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2068278Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2068581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2068685Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2069000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2069088Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2069394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2069605Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2069610Z 2025-09-07T07:02:59.2069729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2069947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2070019Z return mod(**inputs) 2025-09-07T07:02:59.2070332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2070406Z outputs = self.deberta( 2025-09-07T07:02:59.2070714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2070796Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2071097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2071196Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2071464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2071560Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2071867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2071974Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2072275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2072377Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2072687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2072888Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2072892Z 2025-09-07T07:02:59.2073015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2073235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2073317Z return mod(**inputs) 2025-09-07T07:02:59.2073624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2073698Z outputs = self.deberta( 2025-09-07T07:02:59.2074005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2074089Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2074404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2074498Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2074778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2074875Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2075183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2075292Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2075584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2075662Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2075947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2076137Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2076464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2076599Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2076603Z 2025-09-07T07:02:59.2076714Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2076914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2076979Z return mod(**inputs) 2025-09-07T07:02:59.2077270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2077340Z outputs = self.deberta( 2025-09-07T07:02:59.2077623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2077695Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2077981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2078086Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2078312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2078400Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2078675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2078772Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2079046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2079141Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2079424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2079641Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2079646Z 2025-09-07T07:02:59.2079759Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2079960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2080034Z return mod(**inputs) 2025-09-07T07:02:59.2080313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2080383Z outputs = self.deberta( 2025-09-07T07:02:59.2080670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2080741Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2081043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2081131Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2081359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2081463Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2081741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2081841Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2082119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2082207Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2082486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2082702Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2082706Z 2025-09-07T07:02:59.2082819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2083025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2083098Z return mod(**inputs) 2025-09-07T07:02:59.2083380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2083449Z outputs = self.deberta( 2025-09-07T07:02:59.2083735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2083811Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2084100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2084190Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2084442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2084527Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2084806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2084905Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2085183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2085285Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2085560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2085765Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2085779Z 2025-09-07T07:02:59.2085888Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2086104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2086182Z return mod(**inputs) 2025-09-07T07:02:59.2086479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2086558Z outputs = self.deberta( 2025-09-07T07:02:59.2086857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2086930Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2087233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2087321Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2087567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2087653Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2087946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2088047Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2088321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2088406Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2088680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2088879Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2089197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2089334Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2089339Z 2025-09-07T07:02:59.2089449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2089649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2089721Z return mod(**inputs) 2025-09-07T07:02:59.2090006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2090082Z outputs = self.deberta( 2025-09-07T07:02:59.2090363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2090436Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2090732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2090837Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2091070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2091151Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2091425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2091527Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2091819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2091904Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2092180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2092253Z context_layer = torch.bmm( 2025-09-07T07:02:59.2092263Z 2025-09-07T07:02:59.2092366Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2092569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2092642Z return mod(**inputs) 2025-09-07T07:02:59.2092922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2092997Z outputs = self.deberta( 2025-09-07T07:02:59.2093273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2093346Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2093629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2093744Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2093979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2094075Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2094354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2094456Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2094731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2094817Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2095096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2095305Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2095310Z 2025-09-07T07:02:59.2095420Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2095634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2095710Z return mod(**inputs) 2025-09-07T07:02:59.2096012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2096087Z outputs = self.deberta( 2025-09-07T07:02:59.2096364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2096439Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2096725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2096811Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2097045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2097151Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2097462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2097561Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2097862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2097993Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2098303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2098398Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2098401Z 2025-09-07T07:02:59.2098512Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2098736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2098806Z return mod(**inputs) 2025-09-07T07:02:59.2099111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2099190Z outputs = self.deberta( 2025-09-07T07:02:59.2099490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2099572Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2099873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2099965Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2100227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2100318Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2100640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2100770Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2101071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2101165Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2101169Z 2025-09-07T07:02:59.2101281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2101503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2101573Z return mod(**inputs) 2025-09-07T07:02:59.2101883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2101957Z outputs = self.deberta( 2025-09-07T07:02:59.2102261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2102344Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2102633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2102732Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2102976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2103063Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2103362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2103487Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2103806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2103928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2104164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2104239Z return self.act(input) 2025-09-07T07:02:59.2104243Z 2025-09-07T07:02:59.2104351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2104572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2104661Z return mod(**inputs) 2025-09-07T07:02:59.2104973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2105045Z outputs = self.deberta( 2025-09-07T07:02:59.2105348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2105437Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2105820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2105927Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2106178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2106275Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2106579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2106728Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2107065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2107156Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2107160Z 2025-09-07T07:02:59.2107277Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2107509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2107581Z return mod(**inputs) 2025-09-07T07:02:59.2107888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2107962Z outputs = self.deberta( 2025-09-07T07:02:59.2108264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2108342Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2108642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2108738Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2108977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2109072Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2131455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2131730Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2132093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2132205Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2132510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2132718Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2132895Z 2025-09-07T07:02:59.2133030Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2133252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2133328Z return mod(**inputs) 2025-09-07T07:02:59.2133631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2133709Z outputs = self.deberta( 2025-09-07T07:02:59.2134006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2134130Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2134429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2134536Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2134771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2134867Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2135151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2135269Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2135567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2135656Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2135964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2136153Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2136158Z 2025-09-07T07:02:59.2136317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2136532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2136642Z return mod(**inputs) 2025-09-07T07:02:59.2136928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2137003Z outputs = self.deberta( 2025-09-07T07:02:59.2137288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2137367Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2137652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2137747Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2137980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2138075Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2138355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2138464Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2138739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2138818Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2139098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2139296Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2139616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2139787Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2139791Z 2025-09-07T07:02:59.2139905Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2140119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2140188Z return mod(**inputs) 2025-09-07T07:02:59.2140482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2140587Z outputs = self.deberta( 2025-09-07T07:02:59.2140874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2140954Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2141226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2141321Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2141551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2141634Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2141919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2142013Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2142297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2142376Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2142661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2142901Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2142906Z 2025-09-07T07:02:59.2143038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2143269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2143339Z return mod(**inputs) 2025-09-07T07:02:59.2143645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2143719Z outputs = self.deberta( 2025-09-07T07:02:59.2144020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2144099Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2144397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2144501Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2144740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2144837Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2145127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2145228Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2145527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2145698Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2146006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2146237Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2146262Z 2025-09-07T07:02:59.2146385Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2146605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2146677Z return mod(**inputs) 2025-09-07T07:02:59.2146985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2147059Z outputs = self.deberta( 2025-09-07T07:02:59.2147359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2147458Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2147756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2147854Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2148080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2148171Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2148453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2148551Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2148823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2148899Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2149173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2149378Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2149383Z 2025-09-07T07:02:59.2149494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2149708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2149776Z return mod(**inputs) 2025-09-07T07:02:59.2150054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2150120Z outputs = self.deberta( 2025-09-07T07:02:59.2150396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2150470Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2150750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2150835Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2151057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2151144Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2151418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2151517Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2151791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2151868Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2152155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2152348Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2152684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2152833Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2152839Z 2025-09-07T07:02:59.2152948Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2153149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2153513Z return mod(**inputs) 2025-09-07T07:02:59.2153921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2154363Z outputs = self.deberta( 2025-09-07T07:02:59.2154792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2155238Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2155686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2156121Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2156501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2156869Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2157299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2157741Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2158180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2158598Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2159042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2159461Z context_layer = torch.bmm( 2025-09-07T07:02:59.2159583Z 2025-09-07T07:02:59.2159754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2161199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2161546Z return mod(**inputs) 2025-09-07T07:02:59.2161946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2162377Z outputs = self.deberta( 2025-09-07T07:02:59.2162781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2163204Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2163609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2164040Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2164430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2164811Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2165237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2165672Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2166114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2166541Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2166970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2167524Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2167796Z 2025-09-07T07:02:59.2167905Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2168283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2168630Z return mod(**inputs) 2025-09-07T07:02:59.2169033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2169460Z outputs = self.deberta( 2025-09-07T07:02:59.2169850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2170289Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2170699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2171129Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2171505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2171879Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2172298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2172734Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2173172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2173630Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2174090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2174510Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2174653Z 2025-09-07T07:02:59.2174788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2175165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2175512Z return mod(**inputs) 2025-09-07T07:02:59.2175956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2176408Z outputs = self.deberta( 2025-09-07T07:02:59.2176830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2177267Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2177662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2178081Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2178450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2178811Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2179214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2179668Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2180115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2180530Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2180670Z 2025-09-07T07:02:59.2180784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2181144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2181475Z return mod(**inputs) 2025-09-07T07:02:59.2181872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2182330Z outputs = self.deberta( 2025-09-07T07:02:59.2182721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2183139Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2183553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2183983Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2184379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2184788Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2185224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2185833Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2186357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2186848Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2187246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2187623Z return self.act(input) 2025-09-07T07:02:59.2187748Z 2025-09-07T07:02:59.2187873Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2188265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2188606Z return mod(**inputs) 2025-09-07T07:02:59.2189025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2189460Z outputs = self.deberta( 2025-09-07T07:02:59.2189903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2190338Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2190787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2191244Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2191645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2192044Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2192482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2192990Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2193495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2193947Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2194097Z 2025-09-07T07:02:59.2194216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2194605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2194958Z return mod(**inputs) 2025-09-07T07:02:59.2195389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2195829Z outputs = self.deberta( 2025-09-07T07:02:59.2196246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2196677Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2197114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2197585Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2197965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2198340Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2198757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2199204Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2199645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2200089Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2200504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2201042Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2201304Z 2025-09-07T07:02:59.2201408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2201775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2202101Z return mod(**inputs) 2025-09-07T07:02:59.2202487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2202913Z outputs = self.deberta( 2025-09-07T07:02:59.2203309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2203727Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2204143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2204591Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2204973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2205342Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2205790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2206230Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2206661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2207090Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2207513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2208046Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2208285Z 2025-09-07T07:02:59.2208407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2208784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2209123Z return mod(**inputs) 2025-09-07T07:02:59.2209527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2209948Z outputs = self.deberta( 2025-09-07T07:02:59.2210344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2210770Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2211191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2211649Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2212058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2212465Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2212913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2213361Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2213797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2214240Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2214698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2215268Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2215881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2216503Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2216710Z 2025-09-07T07:02:59.2216836Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2217221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2217581Z return mod(**inputs) 2025-09-07T07:02:59.2218009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2218462Z outputs = self.deberta( 2025-09-07T07:02:59.2218876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2219327Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2220239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2220744Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2221173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2221571Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2222018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2222489Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2222949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2223403Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2223837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2224437Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2224739Z 2025-09-07T07:02:59.2224861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2225263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2225686Z return mod(**inputs) 2025-09-07T07:02:59.2226149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2226601Z outputs = self.deberta( 2025-09-07T07:02:59.2227044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2227486Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2227920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2228423Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2228822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2229191Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2229606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2230039Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2230473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2230942Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2231355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2231914Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2232184Z 2025-09-07T07:02:59.2232292Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2232672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2233005Z return mod(**inputs) 2025-09-07T07:02:59.2233400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2233824Z outputs = self.deberta( 2025-09-07T07:02:59.2234214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2234635Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2235044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2235498Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2235880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2236286Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2236716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2237152Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2237588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2238012Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2238433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2238980Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2239233Z 2025-09-07T07:02:59.2239350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2239731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2240061Z return mod(**inputs) 2025-09-07T07:02:59.2240523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2240924Z outputs = self.deberta( 2025-09-07T07:02:59.2241307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2241713Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2242108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2242526Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2242898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2243293Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2243703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2244132Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2244560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2244979Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2245452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2246034Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2246647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2247166Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2247372Z 2025-09-07T07:02:59.2247482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2247863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2248177Z return mod(**inputs) 2025-09-07T07:02:59.2248558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2248963Z outputs = self.deberta( 2025-09-07T07:02:59.2249349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2249760Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2250197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2250612Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2250998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2251360Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2251772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2252196Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2252617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2253028Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2253434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2253842Z context_layer = torch.bmm( 2025-09-07T07:02:59.2253968Z 2025-09-07T07:02:59.2254072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2254434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2254757Z return mod(**inputs) 2025-09-07T07:02:59.2255141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2255552Z outputs = self.deberta( 2025-09-07T07:02:59.2255955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2256371Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2256790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2257204Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2257596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2257955Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2258359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2258780Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2259189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2259616Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2260021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2260548Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2260791Z 2025-09-07T07:02:59.2260902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2261260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2261592Z return mod(**inputs) 2025-09-07T07:02:59.2261977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2262385Z outputs = self.deberta( 2025-09-07T07:02:59.2262776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2263192Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2263605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2264035Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2264430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2264807Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2265283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2265823Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2266307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2266824Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2267308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2267761Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2267928Z 2025-09-07T07:02:59.2268038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2268413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2268737Z return mod(**inputs) 2025-09-07T07:02:59.2269139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2269552Z outputs = self.deberta( 2025-09-07T07:02:59.2269949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2270362Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2270766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2271194Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2271574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2271973Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2272396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2272860Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2273327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2273756Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2273898Z 2025-09-07T07:02:59.2274031Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2274400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2274737Z return mod(**inputs) 2025-09-07T07:02:59.2275154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2275596Z outputs = self.deberta( 2025-09-07T07:02:59.2276003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2276414Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2276850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2277308Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2277688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2278062Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2278475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2278958Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2279422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2279894Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2280286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2280633Z return self.act(input) 2025-09-07T07:02:59.2280756Z 2025-09-07T07:02:59.2280863Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2281240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2281584Z return mod(**inputs) 2025-09-07T07:02:59.2281972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2282390Z outputs = self.deberta( 2025-09-07T07:02:59.2282789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2283208Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2283636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2284094Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2284485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2284860Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2285278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2285760Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2286227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2286682Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2286831Z 2025-09-07T07:02:59.2286939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2287314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2287644Z return mod(**inputs) 2025-09-07T07:02:59.2288034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2288446Z outputs = self.deberta( 2025-09-07T07:02:59.2288867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2289284Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2289700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2290134Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2290515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2290893Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2291313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2291745Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2292182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2292605Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2293033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2293622Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2293898Z 2025-09-07T07:02:59.2294005Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2294393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2294731Z return mod(**inputs) 2025-09-07T07:02:59.2295127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2295539Z outputs = self.deberta( 2025-09-07T07:02:59.2295956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2296406Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2296848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2297295Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2297666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2298038Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2298463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2298899Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2299333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2299759Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2300178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2300714Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2300959Z 2025-09-07T07:02:59.2301113Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2301506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2301865Z return mod(**inputs) 2025-09-07T07:02:59.2302288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2302731Z outputs = self.deberta( 2025-09-07T07:02:59.2303154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2303615Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2304050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2304515Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2304919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2305320Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2305914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2306411Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2306893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2307369Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2307848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2308424Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2309060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2309618Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2309824Z 2025-09-07T07:02:59.2309968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2310362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2310710Z return mod(**inputs) 2025-09-07T07:02:59.2311130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2311579Z outputs = self.deberta( 2025-09-07T07:02:59.2312013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2312464Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2312893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2313350Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2313756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2314149Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2314600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2315072Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2315555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2316004Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2316441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2317070Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2317378Z 2025-09-07T07:02:59.2317492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2317888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2318250Z return mod(**inputs) 2025-09-07T07:02:59.2318685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2319114Z outputs = self.deberta( 2025-09-07T07:02:59.2319708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2320289Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2320732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2321188Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2321585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2321977Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2322427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2322891Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2323341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2323789Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2324226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2324871Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2325165Z 2025-09-07T07:02:59.2325291Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2325712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2326071Z return mod(**inputs) 2025-09-07T07:02:59.2326495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2326942Z outputs = self.deberta( 2025-09-07T07:02:59.2327368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2327806Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2328246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2328703Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2329121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2329512Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2329947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2330408Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2330870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2331321Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2331740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2332277Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2332561Z 2025-09-07T07:02:59.2332669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2333042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2333379Z return mod(**inputs) 2025-09-07T07:02:59.2333765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2334175Z outputs = self.deberta( 2025-09-07T07:02:59.2334567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2335008Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2335418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2335843Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2336233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2336622Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2337068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2337524Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2337983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2338402Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2338820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2339352Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2339947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2340459Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2340675Z 2025-09-07T07:02:59.2340783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2341177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2341541Z return mod(**inputs) 2025-09-07T07:02:59.2341960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2342409Z outputs = self.deberta( 2025-09-07T07:02:59.2342805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2343227Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2343650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2344116Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2344521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2344922Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2345365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2345946Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2346417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2346886Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2347336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2347828Z context_layer = torch.bmm( 2025-09-07T07:02:59.2347963Z 2025-09-07T07:02:59.2348087Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2348485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2348861Z return mod(**inputs) 2025-09-07T07:02:59.2349292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2349747Z outputs = self.deberta( 2025-09-07T07:02:59.2350169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2350662Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2351106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2351577Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2351994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2352391Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2352852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2353326Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2353797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2354255Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2354704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2355312Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2355592Z 2025-09-07T07:02:59.2355709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2356130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2356491Z return mod(**inputs) 2025-09-07T07:02:59.2356902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2357316Z outputs = self.deberta( 2025-09-07T07:02:59.2357712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2358131Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2358532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2358961Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2359338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2359706Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2360121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2360545Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2360979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2361442Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2361919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2362378Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2362521Z 2025-09-07T07:02:59.2362628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2363024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2363364Z return mod(**inputs) 2025-09-07T07:02:59.2363770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2364193Z outputs = self.deberta( 2025-09-07T07:02:59.2364613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2365055Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2365485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2365913Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2366286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2366380Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2366660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2366793Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2367065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2367157Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2367161Z 2025-09-07T07:02:59.2367270Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2367475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2367552Z return mod(**inputs) 2025-09-07T07:02:59.2367855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2367935Z outputs = self.deberta( 2025-09-07T07:02:59.2368232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2368308Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2368595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2368684Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2368919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2369001Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2369283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2369408Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2369684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2369806Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2370027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2370106Z return self.act(input) 2025-09-07T07:02:59.2370109Z 2025-09-07T07:02:59.2370214Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2370418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2370495Z return mod(**inputs) 2025-09-07T07:02:59.2370777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2370853Z outputs = self.deberta( 2025-09-07T07:02:59.2371132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2371232Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2371508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2371597Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2371830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2371911Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2372218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2372354Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2372631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2372724Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2372729Z 2025-09-07T07:02:59.2372836Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2373050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2373116Z return mod(**inputs) 2025-09-07T07:02:59.2373404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2373472Z outputs = self.deberta( 2025-09-07T07:02:59.2373756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2373837Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2374129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2374229Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2374466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2374566Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2374871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2374971Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2375270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2375357Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2375651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2375868Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2375873Z 2025-09-07T07:02:59.2375985Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2376211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2376278Z return mod(**inputs) 2025-09-07T07:02:59.2376563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2376631Z outputs = self.deberta( 2025-09-07T07:02:59.2376906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2376988Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2377277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2377379Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2377636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2377720Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2378020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2378120Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2378421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2378523Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2378821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2379016Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2379021Z 2025-09-07T07:02:59.2379133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2379354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2379427Z return mod(**inputs) 2025-09-07T07:02:59.2379730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2379802Z outputs = self.deberta( 2025-09-07T07:02:59.2380107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2380186Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2380479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2380582Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2380834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2380929Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2381237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2381337Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2381638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2381721Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2382023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2382226Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2382573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2382721Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2382725Z 2025-09-07T07:02:59.2382838Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2383062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2383132Z return mod(**inputs) 2025-09-07T07:02:59.2383435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2383509Z outputs = self.deberta( 2025-09-07T07:02:59.2383801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2383886Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2384181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2384308Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2384550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2384641Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2384934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2385033Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2385350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2385433Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2385817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2386059Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2386063Z 2025-09-07T07:02:59.2386176Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2386397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2386469Z return mod(**inputs) 2025-09-07T07:02:59.2386773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2386845Z outputs = self.deberta( 2025-09-07T07:02:59.2387146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2387224Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2387537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2387641Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2387895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2387990Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2388283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2388384Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2388688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2388774Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2389074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2389303Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2389309Z 2025-09-07T07:02:59.2389434Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2389637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2389703Z return mod(**inputs) 2025-09-07T07:02:59.2389995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2390064Z outputs = self.deberta( 2025-09-07T07:02:59.2390347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2390421Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2390702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2390804Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2391063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2391157Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2391448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2391554Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2391846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2391948Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2392245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2392453Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2392458Z 2025-09-07T07:02:59.2392579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2392780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2392848Z return mod(**inputs) 2025-09-07T07:02:59.2393136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2393205Z outputs = self.deberta( 2025-09-07T07:02:59.2393489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2393563Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2393847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2393934Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2394174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2394269Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2394579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2394687Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2394983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2395064Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2395366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2395575Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2395921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2396067Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2396073Z 2025-09-07T07:02:59.2396193Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2396404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2396474Z return mod(**inputs) 2025-09-07T07:02:59.2396790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2396861Z outputs = self.deberta( 2025-09-07T07:02:59.2397150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2397223Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2397503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2397617Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2397843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2397934Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2398210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2398312Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2398601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2398679Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2398965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2399040Z context_layer = torch.bmm( 2025-09-07T07:02:59.2399044Z 2025-09-07T07:02:59.2399158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2399371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2399442Z return mod(**inputs) 2025-09-07T07:02:59.2399746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2399820Z outputs = self.deberta( 2025-09-07T07:02:59.2400126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2400204Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2400501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2400612Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2400855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2400972Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2401264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2401369Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2401660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2401743Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2402040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2402253Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2402258Z 2025-09-07T07:02:59.2402379Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2402592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2402669Z return mod(**inputs) 2025-09-07T07:02:59.2402966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2403038Z outputs = self.deberta( 2025-09-07T07:02:59.2403337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2403417Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2403718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2403811Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2404056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2404171Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2404471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2404582Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2404882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2405054Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2405366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2405455Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2405459Z 2025-09-07T07:02:59.2405578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2405791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2405866Z return mod(**inputs) 2025-09-07T07:02:59.2406165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2406237Z outputs = self.deberta( 2025-09-07T07:02:59.2406542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2406619Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2406923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2407016Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2407276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2407363Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2407679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2407816Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2408118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2408214Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2408217Z 2025-09-07T07:02:59.2408329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2408543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2408619Z return mod(**inputs) 2025-09-07T07:02:59.2408918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2409001Z outputs = self.deberta( 2025-09-07T07:02:59.2409296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2409379Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2409679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2409772Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2410019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2410104Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2410404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2410532Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2410844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2410981Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2411214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2411295Z return self.act(input) 2025-09-07T07:02:59.2411299Z 2025-09-07T07:02:59.2411408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2411631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2411720Z return mod(**inputs) 2025-09-07T07:02:59.2412012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2412091Z outputs = self.deberta( 2025-09-07T07:02:59.2412382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2412468Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2412761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2412854Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2413100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2413184Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2413480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2413623Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2413938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2414029Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2414033Z 2025-09-07T07:02:59.2414143Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2414384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2414456Z return mod(**inputs) 2025-09-07T07:02:59.2414770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2414843Z outputs = self.deberta( 2025-09-07T07:02:59.2415142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2415226Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2415523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2415623Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2415861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2415946Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2416251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2416350Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2416662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2416746Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2417044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2417253Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2417273Z 2025-09-07T07:02:59.2417383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2417603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2417673Z return mod(**inputs) 2025-09-07T07:02:59.2417973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2418045Z outputs = self.deberta( 2025-09-07T07:02:59.2418347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2418442Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2418735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2418847Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2419074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2419161Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2419439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2419713Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2420144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2420241Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2420545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2420740Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2420794Z 2025-09-07T07:02:59.2420917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2421132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2421229Z return mod(**inputs) 2025-09-07T07:02:59.2421545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2421620Z outputs = self.deberta( 2025-09-07T07:02:59.2421929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2422011Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2422323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2422428Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2422689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2422784Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2423088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2423197Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2423499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2423581Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2423892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2424101Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2424466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2424641Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2424646Z 2025-09-07T07:02:59.2424760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2424991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2425065Z return mod(**inputs) 2025-09-07T07:02:59.2425382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2425482Z outputs = self.deberta( 2025-09-07T07:02:59.2425857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2425940Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2426257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2426361Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2426610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2426706Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2427015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2427113Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2427434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2427517Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2427828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2428065Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2428069Z 2025-09-07T07:02:59.2428196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2428401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2428467Z return mod(**inputs) 2025-09-07T07:02:59.2428763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2428831Z outputs = self.deberta( 2025-09-07T07:02:59.2429122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2429196Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2429485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2429582Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2429812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2429900Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2430186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2430286Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2430568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2430647Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2430938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2431155Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2431174Z 2025-09-07T07:02:59.2431287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2431488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2431555Z return mod(**inputs) 2025-09-07T07:02:59.2431852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2431922Z outputs = self.deberta( 2025-09-07T07:02:59.2432216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2432307Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2432594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2432684Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2432910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2432998Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2433279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2433377Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2433726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2433809Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2434092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2434317Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2434322Z 2025-09-07T07:02:59.2434443Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2434677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2434757Z return mod(**inputs) 2025-09-07T07:02:59.2435053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2435125Z outputs = self.deberta( 2025-09-07T07:02:59.2435429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2435508Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2435810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2435902Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2436143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2436246Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2436522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2436624Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2436902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2436992Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2437288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2437499Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2437845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2438017Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2438022Z 2025-09-07T07:02:59.2438142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2438358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2438435Z return mod(**inputs) 2025-09-07T07:02:59.2438735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2438830Z outputs = self.deberta( 2025-09-07T07:02:59.2439140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2439217Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2439527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2439622Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2439867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2439960Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2440260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2440366Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2440669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2440761Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2441076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2441156Z context_layer = torch.bmm( 2025-09-07T07:02:59.2441160Z 2025-09-07T07:02:59.2441280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2441511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2441589Z return mod(**inputs) 2025-09-07T07:02:59.2441888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2441958Z outputs = self.deberta( 2025-09-07T07:02:59.2442258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2442336Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2442634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2442727Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2442967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2443062Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2443355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2443464Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2443756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2443847Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2444140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2444343Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2444368Z 2025-09-07T07:02:59.2444486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2444702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2444780Z return mod(**inputs) 2025-09-07T07:02:59.2445079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2445150Z outputs = self.deberta( 2025-09-07T07:02:59.2445454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2445549Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2445848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2445939Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2446185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2446272Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2446565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2446670Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2446962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2447096Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2447389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2447477Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2447490Z 2025-09-07T07:02:59.2447616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2447836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2447912Z return mod(**inputs) 2025-09-07T07:02:59.2448240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2448323Z outputs = self.deberta( 2025-09-07T07:02:59.2448633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2448705Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2448997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2449086Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2449319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2449400Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2449675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2449805Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2450082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2450175Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2450178Z 2025-09-07T07:02:59.2450284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2450496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2450561Z return mod(**inputs) 2025-09-07T07:02:59.2450840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2450933Z outputs = self.deberta( 2025-09-07T07:02:59.2451210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2451292Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2451569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2451656Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2451890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2451985Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2452268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2452389Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2452684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2452798Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2453012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2453092Z return self.act(input) 2025-09-07T07:02:59.2453095Z 2025-09-07T07:02:59.2453196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2453405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2453474Z return mod(**inputs) 2025-09-07T07:02:59.2453752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2453826Z outputs = self.deberta( 2025-09-07T07:02:59.2454112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2454193Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2454485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2454582Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2454808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2454891Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2455177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2455313Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2455600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2455686Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2455689Z 2025-09-07T07:02:59.2455793Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2456005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2456070Z return mod(**inputs) 2025-09-07T07:02:59.2456362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2456431Z outputs = self.deberta( 2025-09-07T07:02:59.2456717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2456790Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2457068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2457180Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2457410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2457499Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2457774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2457867Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2458151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2458255Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2458539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2458731Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2458736Z 2025-09-07T07:02:59.2458847Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2459050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2459117Z return mod(**inputs) 2025-09-07T07:02:59.2459406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2459473Z outputs = self.deberta( 2025-09-07T07:02:59.2459757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2459831Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2460108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2460221Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2460447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2460534Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2460832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2460938Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2461228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2461311Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2461611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2461804Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2461809Z 2025-09-07T07:02:59.2461929Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2462139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2462210Z return mod(**inputs) 2025-09-07T07:02:59.2462516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2462590Z outputs = self.deberta( 2025-09-07T07:02:59.2462888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2462966Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2463263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2463355Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2463593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2463702Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2463992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2464098Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2464389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2464470Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2464793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2464996Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2465344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2465490Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2465494Z 2025-09-07T07:02:59.2465689Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2465915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2465987Z return mod(**inputs) 2025-09-07T07:02:59.2466302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2466378Z outputs = self.deberta( 2025-09-07T07:02:59.2466679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2466760Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2467074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2467181Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2467437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2467532Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2467836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2467942Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2468245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2468327Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2468651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2468883Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2468887Z 2025-09-07T07:02:59.2469008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2469228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2469305Z return mod(**inputs) 2025-09-07T07:02:59.2469619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2469692Z outputs = self.deberta( 2025-09-07T07:02:59.2470000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2470077Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2470383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2470495Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2470747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2470842Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2471135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2471240Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2471532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2471636Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2471940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2472166Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2472171Z 2025-09-07T07:02:59.2472290Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2472517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2472596Z return mod(**inputs) 2025-09-07T07:02:59.2472907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2472980Z outputs = self.deberta( 2025-09-07T07:02:59.2473283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2473362Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2473664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2473777Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2474018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2474126Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2474418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2474523Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2474814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2474906Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2475197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2475402Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2475407Z 2025-09-07T07:02:59.2475526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2475745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2475823Z return mod(**inputs) 2025-09-07T07:02:59.2476126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2476204Z outputs = self.deberta( 2025-09-07T07:02:59.2476495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2476572Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2476866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2476957Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2477203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2477308Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2477597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2477696Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2477972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2478057Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2478348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2478547Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2478868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2479005Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2479010Z 2025-09-07T07:02:59.2479122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2479323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2479395Z return mod(**inputs) 2025-09-07T07:02:59.2479681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2479751Z outputs = self.deberta( 2025-09-07T07:02:59.2480036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2480108Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2480408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2480501Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2480764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2480850Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2481139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2481247Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2481539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2481630Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2481928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2482006Z context_layer = torch.bmm( 2025-09-07T07:02:59.2482009Z 2025-09-07T07:02:59.2482127Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2482347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2482425Z return mod(**inputs) 2025-09-07T07:02:59.2482723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2482802Z outputs = self.deberta( 2025-09-07T07:02:59.2483104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2483179Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2483480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2483574Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2483850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2483938Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2484227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2484342Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2484619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2484724Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2484995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2485196Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2485201Z 2025-09-07T07:02:59.2485306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2485511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2485589Z return mod(**inputs) 2025-09-07T07:02:59.2485887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2485965Z outputs = self.deberta( 2025-09-07T07:02:59.2486256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2486334Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2486633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2486724Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2486987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2487073Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2487389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2487492Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2487771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2487899Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2488178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2488268Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2488271Z 2025-09-07T07:02:59.2488377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2488582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2488655Z return mod(**inputs) 2025-09-07T07:02:59.2488937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2489014Z outputs = self.deberta( 2025-09-07T07:02:59.2489296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2489377Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2489658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2489746Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2489983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2490077Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2490360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2490481Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2490754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2490846Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2490850Z 2025-09-07T07:02:59.2490971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2491191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2491260Z return mod(**inputs) 2025-09-07T07:02:59.2491559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2491639Z outputs = self.deberta( 2025-09-07T07:02:59.2491936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2492020Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2492328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2492420Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2492644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2492725Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2493006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2493143Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2493423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2493554Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2493773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2493852Z return self.act(input) 2025-09-07T07:02:59.2493855Z 2025-09-07T07:02:59.2493957Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2494172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2494243Z return mod(**inputs) 2025-09-07T07:02:59.2494544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2494616Z outputs = self.deberta( 2025-09-07T07:02:59.2494910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2494994Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2495289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2495388Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2495627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2495712Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2496012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2496154Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2496455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2496560Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2496564Z 2025-09-07T07:02:59.2496682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2496898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2496967Z return mod(**inputs) 2025-09-07T07:02:59.2497271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2497342Z outputs = self.deberta( 2025-09-07T07:02:59.2497657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2497734Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2498031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2498132Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2498373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2498465Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2498761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2498868Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2499166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2499251Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2499551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2499773Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2499779Z 2025-09-07T07:02:59.2499896Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2500127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2500199Z return mod(**inputs) 2025-09-07T07:02:59.2500507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2500580Z outputs = self.deberta( 2025-09-07T07:02:59.2500875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2500954Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2501252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2501346Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2501582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2501677Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2501965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2502070Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2502363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2502446Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2502744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2502938Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2502943Z 2025-09-07T07:02:59.2503078Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2503292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2503369Z return mod(**inputs) 2025-09-07T07:02:59.2503664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2503736Z outputs = self.deberta( 2025-09-07T07:02:59.2504036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2504129Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2504432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2504525Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2504767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2504860Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2505154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2505258Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2505557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2505721Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2506035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2506242Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2506617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2506783Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2506787Z 2025-09-07T07:02:59.2506931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2507147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2507225Z return mod(**inputs) 2025-09-07T07:02:59.2507527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2507599Z outputs = self.deberta( 2025-09-07T07:02:59.2507886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2507959Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2508251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2508341Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2508575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2508664Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2508938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2509039Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2509321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2509398Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2509681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2509935Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2509939Z 2025-09-07T07:02:59.2510050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2510250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2510322Z return mod(**inputs) 2025-09-07T07:02:59.2510625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2510697Z outputs = self.deberta( 2025-09-07T07:02:59.2511015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2511092Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2511394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2511487Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2511735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2511828Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2512118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2512223Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2512517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2512607Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2512898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2513142Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2513148Z 2025-09-07T07:02:59.2513266Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2513505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2513581Z return mod(**inputs) 2025-09-07T07:02:59.2513863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2513937Z outputs = self.deberta( 2025-09-07T07:02:59.2514224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2514302Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2514602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2514697Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2514945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2515031Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2515334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2515443Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2515743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2515838Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2516138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2516356Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2516377Z 2025-09-07T07:02:59.2516487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2516715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2516792Z return mod(**inputs) 2025-09-07T07:02:59.2517098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2517178Z outputs = self.deberta( 2025-09-07T07:02:59.2517481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2517576Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2517878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2517970Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2518219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2518305Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2518606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2518705Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2518997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2519099Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2519377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2519748Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2520241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2520421Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2520435Z 2025-09-07T07:02:59.2520543Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2520748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2520825Z return mod(**inputs) 2025-09-07T07:02:59.2521108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2521188Z outputs = self.deberta( 2025-09-07T07:02:59.2521464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2521537Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2521820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2521909Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2522145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2522225Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2522502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2522603Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2522878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2522962Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2523238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2523345Z context_layer = torch.bmm( 2025-09-07T07:02:59.2523349Z 2025-09-07T07:02:59.2523454Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2523658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2523732Z return mod(**inputs) 2025-09-07T07:02:59.2524009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2524082Z outputs = self.deberta( 2025-09-07T07:02:59.2524382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2524453Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2524747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2524842Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2525089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2525176Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2525475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2525573Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2525863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2525954Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2526245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2526472Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2526477Z 2025-09-07T07:02:59.2526587Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2526822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2526900Z return mod(**inputs) 2025-09-07T07:02:59.2527200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2527286Z outputs = self.deberta( 2025-09-07T07:02:59.2527559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2527638Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2527907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2527992Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2528224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2528302Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2528580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2528670Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2528943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2529071Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2529348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2529439Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2529443Z 2025-09-07T07:02:59.2529547Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2529772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2529838Z return mod(**inputs) 2025-09-07T07:02:59.2530120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2530197Z outputs = self.deberta( 2025-09-07T07:02:59.2530484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2530561Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2530846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2530930Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2531155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2531235Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2531509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2531625Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2531897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2531977Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2531981Z 2025-09-07T07:02:59.2532081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2532283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2532346Z return mod(**inputs) 2025-09-07T07:02:59.2532977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2533050Z outputs = self.deberta( 2025-09-07T07:02:59.2533334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2533415Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2533692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2533785Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2534010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2534092Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2534380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2534502Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2534785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2534902Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2535132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2535208Z return self.act(input) 2025-09-07T07:02:59.2535212Z 2025-09-07T07:02:59.2535322Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2535550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2535623Z return mod(**inputs) 2025-09-07T07:02:59.2535926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2535997Z outputs = self.deberta( 2025-09-07T07:02:59.2536291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2536396Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2536688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2536785Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2537033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2537119Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2537413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2537548Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2537836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2537921Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2537925Z 2025-09-07T07:02:59.2538037Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2538239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2538305Z return mod(**inputs) 2025-09-07T07:02:59.2538592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2538659Z outputs = self.deberta( 2025-09-07T07:02:59.2538946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2539019Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2539319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2539409Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2539640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2539745Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2540020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2540122Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2540395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2540473Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2540756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2540949Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2540954Z 2025-09-07T07:02:59.2541065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2541265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2541337Z return mod(**inputs) 2025-09-07T07:02:59.2541613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2541679Z outputs = self.deberta( 2025-09-07T07:02:59.2541961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2542034Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2542311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2542399Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2542641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2542730Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2543005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2543103Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2543393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2543499Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2543788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2543980Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2543986Z 2025-09-07T07:02:59.2544105Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2544317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2544392Z return mod(**inputs) 2025-09-07T07:02:59.2544688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2544759Z outputs = self.deberta( 2025-09-07T07:02:59.2545060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2545138Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2545437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2545528Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2545873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2545966Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2546275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2546386Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2546678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2546773Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2547065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2547256Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2547581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2547717Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2547721Z 2025-09-07T07:02:59.2547834Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2548035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2548110Z return mod(**inputs) 2025-09-07T07:02:59.2548393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2548464Z outputs = self.deberta( 2025-09-07T07:02:59.2548747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2548819Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2549104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2549209Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2549438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2549526Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2549804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2549904Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2550195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2550281Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2550556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2550774Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2550778Z 2025-09-07T07:02:59.2550890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2551092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2551165Z return mod(**inputs) 2025-09-07T07:02:59.2551445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2551519Z outputs = self.deberta( 2025-09-07T07:02:59.2551793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2551864Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2552169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2552260Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2552510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2552591Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2552863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2552964Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2553237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2553325Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2553604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2553830Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2553835Z 2025-09-07T07:02:59.2553941Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2554147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2554225Z return mod(**inputs) 2025-09-07T07:02:59.2554509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2554588Z outputs = self.deberta( 2025-09-07T07:02:59.2554868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2554945Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2555232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2555324Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2555575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2555656Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2555941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2556033Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2556308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2556418Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2556694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2556898Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2556903Z 2025-09-07T07:02:59.2557005Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2557209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2557282Z return mod(**inputs) 2025-09-07T07:02:59.2557563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2557636Z outputs = self.deberta( 2025-09-07T07:02:59.2557909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2557990Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2558268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2558354Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2558601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2558685Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2558993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2559081Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2559339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2559420Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2559677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2559864Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2560164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2560302Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2560306Z 2025-09-07T07:02:59.2560410Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2560610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2560681Z return mod(**inputs) 2025-09-07T07:02:59.2560953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2561027Z outputs = self.deberta( 2025-09-07T07:02:59.2561295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2561368Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2561645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2561747Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2561977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2562055Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2562332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2562421Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2562712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2562797Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2563075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2563155Z context_layer = torch.bmm( 2025-09-07T07:02:59.2563158Z 2025-09-07T07:02:59.2563260Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2563471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2563540Z return mod(**inputs) 2025-09-07T07:02:59.2563813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2563886Z outputs = self.deberta( 2025-09-07T07:02:59.2564174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2564256Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2564540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2564643Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2564879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2564977Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2565260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2565352Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2565628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2565715Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2565991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2566192Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2566196Z 2025-09-07T07:02:59.2566300Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2566511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2566578Z return mod(**inputs) 2025-09-07T07:02:59.2566858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2566934Z outputs = self.deberta( 2025-09-07T07:02:59.2567209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2567291Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2567572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2567660Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2567912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2567992Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2568290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2568380Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2568657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2568789Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2569064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2569153Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2569156Z 2025-09-07T07:02:59.2569254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2569454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2569518Z return mod(**inputs) 2025-09-07T07:02:59.2569783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2569855Z outputs = self.deberta( 2025-09-07T07:02:59.2570116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2570194Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2570456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2570547Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2570777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2570857Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2571149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2571269Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2571540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2571622Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2571625Z 2025-09-07T07:02:59.2571731Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2571937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2572003Z return mod(**inputs) 2025-09-07T07:02:59.2572286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2572354Z outputs = self.deberta( 2025-09-07T07:02:59.2572630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2572712Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2572972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2573062Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2573278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2573363Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2573625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2573740Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2574031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2574148Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2574371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2574442Z return self.act(input) 2025-09-07T07:02:59.2574445Z 2025-09-07T07:02:59.2574555Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2574770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2574857Z return mod(**inputs) 2025-09-07T07:02:59.2575162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2575237Z outputs = self.deberta( 2025-09-07T07:02:59.2575541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2575620Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2575920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2576016Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2576243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2576330Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2576608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2576744Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2577050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2577142Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2577146Z 2025-09-07T07:02:59.2577278Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2577492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2577569Z return mod(**inputs) 2025-09-07T07:02:59.2577867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2577939Z outputs = self.deberta( 2025-09-07T07:02:59.2578246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2578322Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2578624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2578717Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2578960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2579052Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2579342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2579449Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2579742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2579835Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2580129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2580333Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2580361Z 2025-09-07T07:02:59.2580481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2580694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2580777Z return mod(**inputs) 2025-09-07T07:02:59.2581076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2581148Z outputs = self.deberta( 2025-09-07T07:02:59.2581463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2581560Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2581860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2581956Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2582207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2582292Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2582585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2582695Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2582998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2583089Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2583380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-09-07T07:02:59.2583575Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2583603Z 2025-09-07T07:02:59.2583716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2583934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2584029Z return mod(**inputs) 2025-09-07T07:02:59.2584326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2584406Z outputs = self.deberta( 2025-09-07T07:02:59.2584711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2584789Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2585089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2585180Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2585426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2585514Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2585892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2586009Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2586311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2586406Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2586709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-09-07T07:02:59.2586928Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-09-07T07:02:59.2587275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2587450Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2587457Z 2025-09-07T07:02:59.2587592Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2587811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2587888Z return mod(**inputs) 2025-09-07T07:02:59.2588195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2588294Z outputs = self.deberta( 2025-09-07T07:02:59.2588599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2588677Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2588991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2589086Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2589339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2589424Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2589723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2589831Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2590133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2590222Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2590537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2590776Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2590781Z 2025-09-07T07:02:59.2590908Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2591125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2591202Z return mod(**inputs) 2025-09-07T07:02:59.2591503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2591584Z outputs = self.deberta( 2025-09-07T07:02:59.2591876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2591951Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2592252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2592346Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2592593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2592677Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2592990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2593089Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2593383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2593471Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2593747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-09-07T07:02:59.2593967Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-09-07T07:02:59.2593985Z 2025-09-07T07:02:59.2594091Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2594299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2594373Z return mod(**inputs) 2025-09-07T07:02:59.2594665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2594744Z outputs = self.deberta( 2025-09-07T07:02:59.2595035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2595136Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2595438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2595537Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2595795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2595886Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2596199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2596300Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2596608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2596702Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2596994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2597233Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2597238Z 2025-09-07T07:02:59.2597344Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2597570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2597638Z return mod(**inputs) 2025-09-07T07:02:59.2597934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2598016Z outputs = self.deberta( 2025-09-07T07:02:59.2598311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2598397Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2598693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2598787Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2599040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2599128Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2599429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2599527Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2599829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2599914Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2600208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-09-07T07:02:59.2600426Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-09-07T07:02:59.2600774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-09-07T07:02:59.2600950Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-09-07T07:02:59.2600954Z 2025-09-07T07:02:59.2601066Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2601304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2601375Z return mod(**inputs) 2025-09-07T07:02:59.2601694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2601793Z outputs = self.deberta( 2025-09-07T07:02:59.2602109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2602196Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2602495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2602591Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2602844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2602930Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2603235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2603332Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2603633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2603723Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2604043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-09-07T07:02:59.2604132Z context_layer = torch.bmm( 2025-09-07T07:02:59.2604137Z 2025-09-07T07:02:59.2604248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2604504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2604578Z return mod(**inputs) 2025-09-07T07:02:59.2604881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2604962Z outputs = self.deberta( 2025-09-07T07:02:59.2605261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2605347Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2605654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2605750Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2605999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2606087Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2606395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2606495Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2606800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-09-07T07:02:59.2606886Z self_output, att_matrix = self.self( 2025-09-07T07:02:59.2607185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-09-07T07:02:59.2607405Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-09-07T07:02:59.2607424Z 2025-09-07T07:02:59.2607538Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2607777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2607849Z return mod(**inputs) 2025-09-07T07:02:59.2608156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2608238Z outputs = self.deberta( 2025-09-07T07:02:59.2608540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2608642Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2608944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2609048Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2609306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2609393Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2609704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-09-07T07:02:59.2609805Z attention_output, att_matrix = self.attention( 2025-09-07T07:02:59.2610113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-09-07T07:02:59.2610240Z attention_output = self.output(self_output, query_states) 2025-09-07T07:02:59.2610542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-09-07T07:02:59.2610642Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2610646Z 2025-09-07T07:02:59.2610775Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2611005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2611075Z return mod(**inputs) 2025-09-07T07:02:59.2611447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2611523Z outputs = self.deberta( 2025-09-07T07:02:59.2611827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2611916Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2612220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2612324Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2612586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2612674Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2612986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2613120Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2613429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-09-07T07:02:59.2613520Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2613524Z 2025-09-07T07:02:59.2613645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2613867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2613937Z return mod(**inputs) 2025-09-07T07:02:59.2614270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2614363Z outputs = self.deberta( 2025-09-07T07:02:59.2614673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2614752Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2615056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2615159Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2615417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2615532Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2615835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-09-07T07:02:59.2615973Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:02:59.2616278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-09-07T07:02:59.2616403Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:02:59.2616644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:02:59.2616719Z return self.act(input) 2025-09-07T07:02:59.2616723Z 2025-09-07T07:02:59.2616843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2617074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2617148Z return mod(**inputs) 2025-09-07T07:02:59.2617473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-09-07T07:02:59.2617546Z outputs = self.deberta( 2025-09-07T07:02:59.2617872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-09-07T07:02:59.2617953Z encoder_outputs = self.encoder( 2025-09-07T07:02:59.2618290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-09-07T07:02:59.2618385Z output_states, attn_weights = layer_module( 2025-09-07T07:02:59.2618630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:02:59.2618722Z return super().__call__(*args, **kwargs) 2025-09-07T07:02:59.2619017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-09-07T07:02:59.2619168Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:02:59.2619464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-09-07T07:02:59.2619695Z hidden_states = self.dense(hidden_states) 2025-09-07T07:02:59.2619702Z 2025-09-07T07:02:59.2619889Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2620151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2620232Z return mod(**inputs) 2025-09-07T07:02:59.2620546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1244, in forward 2025-09-07T07:02:59.2620645Z logits = self.qa_outputs(sequence_output) 2025-09-07T07:02:59.2620651Z 2025-09-07T07:02:59.2620763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2620991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2621071Z return mod(**inputs) 2025-09-07T07:02:59.2621388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1262, in forward 2025-09-07T07:02:59.2621564Z start_loss = loss_fct(start_logits, start_positions) 2025-09-07T07:02:59.2621568Z 2025-09-07T07:02:59.2621682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:02:59.2621900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:02:59.2621978Z return mod(**inputs) 2025-09-07T07:02:59.2622282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1263, in forward 2025-09-07T07:02:59.2622425Z end_loss = loss_fct(end_logits, end_positions) 2025-09-07T07:02:59.2622429Z 2025-09-07T07:03:14.1076322Z Compilation time (from dynamo_timed): 28.59568108 2025-09-07T07:03:14.1078110Z pass 2025-09-07T07:03:14.1078461Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:14.1079453Z TIMING: _recursive_pre_grad_passes:0.01481 _recursive_joint_graph_passes:1.17996 _recursive_post_grad_passes:0.31576 async_compile.wait:0.51117 code_gen:13.62134 inductor_compile:16.75413 backend_compile:23.42953 gc:0.00076 entire_frame_compile:28.59568 total_wall_time:28.59568 2025-09-07T07:03:14.1083733Z STATS: call_* op count: 1087 | FakeTensorMode.__torch_dispatch__:30534 | FakeTensor.__torch_dispatch__:10573 | ProxyTorchDispatchMode.__torch_dispatch__:11524 2025-09-07T07:03:14.1084380Z Dynamo produced 1 graphs covering 1087 ops with 0 graph breaks (0 unique) 2025-09-07T07:03:17.4165805Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:03:17.4167325Z import pynvml # type: ignore[import] 2025-09-07T07:03:20.2151162Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:03:20.2152516Z from pkg_resources import resource_filename 2025-09-07T07:03:20.8667646Z 2025-09-07T07:03:21.6047257Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:03:21.6047762Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:03:21.6052016Z cpu eval DistilBertForMaskedLM 2025-09-07T07:03:21.7691139Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:21.8273471Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:21.8823644Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:26.8880505Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8886253Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8888751Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8889116Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8889386Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8889605Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8889862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8890288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8890715Z return mod(**inputs) 2025-09-07T07:03:26.8891213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8891701Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8892172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8892968Z return self.transformer( 2025-09-07T07:03:26.8893411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8893874Z layer_outputs = layer_module( 2025-09-07T07:03:26.8894263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8894704Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8895165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8895695Z sa_output = self.attention( 2025-09-07T07:03:26.8896144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:26.8896668Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:26.8896872Z 2025-09-07T07:03:26.8897000Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8897394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8897778Z return mod(**inputs) 2025-09-07T07:03:26.8898223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8898687Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8899138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8899592Z return self.transformer( 2025-09-07T07:03:26.8900043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8900501Z layer_outputs = layer_module( 2025-09-07T07:03:26.8900949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8901357Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8901871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8902316Z sa_output = self.attention( 2025-09-07T07:03:26.8902757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:26.8903259Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.8903455Z 2025-09-07T07:03:26.8903570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8903968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8904324Z return mod(**inputs) 2025-09-07T07:03:26.8904762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8905216Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8905905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8906374Z return self.transformer( 2025-09-07T07:03:26.8906819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8907285Z layer_outputs = layer_module( 2025-09-07T07:03:26.8907669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8908077Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8908527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8908985Z sa_output = self.attention( 2025-09-07T07:03:26.8909458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:26.8909956Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.8910150Z 2025-09-07T07:03:26.8910236Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8910496Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8910880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8911247Z return mod(**inputs) 2025-09-07T07:03:26.8911679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8912123Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8912558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8913000Z return self.transformer( 2025-09-07T07:03:26.8913406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8913826Z layer_outputs = layer_module( 2025-09-07T07:03:26.8914179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8914544Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8914970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8915396Z sa_output = self.attention( 2025-09-07T07:03:26.8915801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:26.8916277Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:26.8916473Z 2025-09-07T07:03:26.8916603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8917110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8917515Z return mod(**inputs) 2025-09-07T07:03:26.8917943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8918402Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8918841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8919289Z return self.transformer( 2025-09-07T07:03:26.8919969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8920434Z layer_outputs = layer_module( 2025-09-07T07:03:26.8920804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8921193Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8921699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8922145Z sa_output = self.attention( 2025-09-07T07:03:26.8922581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:26.8923046Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:26.8923209Z 2025-09-07T07:03:26.8923325Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8923720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8924095Z return mod(**inputs) 2025-09-07T07:03:26.8924530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8925023Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8925465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8925888Z return self.transformer( 2025-09-07T07:03:26.8926301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8926715Z layer_outputs = layer_module( 2025-09-07T07:03:26.8927077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8927476Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8927905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.8928372Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.8928831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.8929394Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.8929936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.8930351Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.8930778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:26.8931196Z x = self.lin1(input) 2025-09-07T07:03:26.8931312Z 2025-09-07T07:03:26.8931418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8931789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8932155Z return mod(**inputs) 2025-09-07T07:03:26.8932558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8933018Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8933465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8933888Z return self.transformer( 2025-09-07T07:03:26.8934297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8934718Z layer_outputs = layer_module( 2025-09-07T07:03:26.8935085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8935458Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8935892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.8936362Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.8936835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.8937419Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.8937988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.8938421Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.8938874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:26.8939320Z x = self.activation(x) 2025-09-07T07:03:26.8939683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:26.8940069Z return self.act(input) 2025-09-07T07:03:26.8940185Z 2025-09-07T07:03:26.8940306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8940690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8941043Z return mod(**inputs) 2025-09-07T07:03:26.8941472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8941924Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8942368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8942827Z return self.transformer( 2025-09-07T07:03:26.8943260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8943712Z layer_outputs = layer_module( 2025-09-07T07:03:26.8944097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8944491Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8944937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.8945422Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.8946029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.8946707Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.8947286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.8947716Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.8948171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:26.8948607Z x = self.lin2(x) 2025-09-07T07:03:26.8948708Z 2025-09-07T07:03:26.8948841Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8949205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8949538Z return mod(**inputs) 2025-09-07T07:03:26.8949936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8950364Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8950778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8951192Z return self.transformer( 2025-09-07T07:03:26.8951604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8952036Z layer_outputs = layer_module( 2025-09-07T07:03:26.8952394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8952767Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8953187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8953614Z sa_output = self.attention( 2025-09-07T07:03:26.8954025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:26.8954507Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:26.8954695Z 2025-09-07T07:03:26.8954809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8955180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8955541Z return mod(**inputs) 2025-09-07T07:03:26.8955947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8956379Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8956794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8957207Z return self.transformer( 2025-09-07T07:03:26.8957605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8958045Z layer_outputs = layer_module( 2025-09-07T07:03:26.8958394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8958750Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8959165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8959573Z sa_output = self.attention( 2025-09-07T07:03:26.8959980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:26.8960453Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.8960635Z 2025-09-07T07:03:26.8960738Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8961097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8961424Z return mod(**inputs) 2025-09-07T07:03:26.8961814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8962219Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8962664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8963089Z return self.transformer( 2025-09-07T07:03:26.8963501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8963913Z layer_outputs = layer_module( 2025-09-07T07:03:26.8964252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8964615Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8965039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8965455Z sa_output = self.attention( 2025-09-07T07:03:26.8965848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:26.8966317Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.8966504Z 2025-09-07T07:03:26.8966588Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.8966841Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8967208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8967530Z return mod(**inputs) 2025-09-07T07:03:26.8967926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8968360Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8968767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8969178Z return self.transformer( 2025-09-07T07:03:26.8969566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8970000Z layer_outputs = layer_module( 2025-09-07T07:03:26.8970351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8970722Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8971142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8971562Z sa_output = self.attention( 2025-09-07T07:03:26.8971974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:26.8972493Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:26.8972684Z 2025-09-07T07:03:26.8972796Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8973151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8973481Z return mod(**inputs) 2025-09-07T07:03:26.8973870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8974288Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8974712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8975133Z return self.transformer( 2025-09-07T07:03:26.8975539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8975965Z layer_outputs = layer_module( 2025-09-07T07:03:26.8976320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8976693Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8977159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.8977581Z sa_output = self.attention( 2025-09-07T07:03:26.8978003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:26.8978435Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:26.8978578Z 2025-09-07T07:03:26.8978683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8979052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8979385Z return mod(**inputs) 2025-09-07T07:03:26.8979781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8980206Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8980617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8981039Z return self.transformer( 2025-09-07T07:03:26.8981452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8981872Z layer_outputs = layer_module( 2025-09-07T07:03:26.8982223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8982604Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8983039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.8983505Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.8983966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.8984527Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.8985059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.8985465Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.8986008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:26.8986488Z x = self.lin1(input) 2025-09-07T07:03:26.8986606Z 2025-09-07T07:03:26.8986778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8987202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8987542Z return mod(**inputs) 2025-09-07T07:03:26.8987949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8988374Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8988792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8989214Z return self.transformer( 2025-09-07T07:03:26.8989620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8990044Z layer_outputs = layer_module( 2025-09-07T07:03:26.8990396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8990769Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8991194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.8991675Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.8992140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.8992701Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.8993217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.8993616Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.8994031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:26.8994449Z x = self.activation(x) 2025-09-07T07:03:26.8994771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:26.8995110Z return self.act(input) 2025-09-07T07:03:26.8995227Z 2025-09-07T07:03:26.8995330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.8995695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.8996020Z return mod(**inputs) 2025-09-07T07:03:26.8996403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.8996814Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.8997221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.8997631Z return self.transformer( 2025-09-07T07:03:26.8998020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.8998433Z layer_outputs = layer_module( 2025-09-07T07:03:26.8998792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.8999168Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.8999590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9000027Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9000472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9001008Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9002174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9002586Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9003015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:26.9003435Z x = self.lin2(x) 2025-09-07T07:03:26.9003545Z 2025-09-07T07:03:26.9003653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9004029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9004369Z return mod(**inputs) 2025-09-07T07:03:26.9004770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9005204Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9005628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9006062Z return self.transformer( 2025-09-07T07:03:26.9006456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9006891Z layer_outputs = layer_module( 2025-09-07T07:03:26.9007243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9007622Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9008048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9008466Z sa_output = self.attention( 2025-09-07T07:03:26.9008885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:26.9009376Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:26.9009555Z 2025-09-07T07:03:26.9009666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9010025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9010342Z return mod(**inputs) 2025-09-07T07:03:26.9010745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9011231Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9011648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9012065Z return self.transformer( 2025-09-07T07:03:26.9012473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9012894Z layer_outputs = layer_module( 2025-09-07T07:03:26.9013248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9013615Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9014041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9014488Z sa_output = self.attention( 2025-09-07T07:03:26.9014901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:26.9015377Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9015554Z 2025-09-07T07:03:26.9015668Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9016031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9016364Z return mod(**inputs) 2025-09-07T07:03:26.9016793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9017224Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9017635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9018057Z return self.transformer( 2025-09-07T07:03:26.9018462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9018884Z layer_outputs = layer_module( 2025-09-07T07:03:26.9019239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9019799Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9020240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9020671Z sa_output = self.attention( 2025-09-07T07:03:26.9021084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:26.9021625Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9021810Z 2025-09-07T07:03:26.9021895Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.9022146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9022543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9022902Z return mod(**inputs) 2025-09-07T07:03:26.9023322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9023766Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9024198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9024637Z return self.transformer( 2025-09-07T07:03:26.9025065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9025494Z layer_outputs = layer_module( 2025-09-07T07:03:26.9025908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9026293Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9026746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9027192Z sa_output = self.attention( 2025-09-07T07:03:26.9027623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:26.9028112Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:26.9028311Z 2025-09-07T07:03:26.9028418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9028800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9029146Z return mod(**inputs) 2025-09-07T07:03:26.9029607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9030068Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9030522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9030976Z return self.transformer( 2025-09-07T07:03:26.9031407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9031906Z layer_outputs = layer_module( 2025-09-07T07:03:26.9032281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9032672Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9033123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9033581Z sa_output = self.attention( 2025-09-07T07:03:26.9034027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:26.9034493Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:26.9034648Z 2025-09-07T07:03:26.9034773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9035174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9035536Z return mod(**inputs) 2025-09-07T07:03:26.9035975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9036427Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9036870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9037362Z return self.transformer( 2025-09-07T07:03:26.9037800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9038262Z layer_outputs = layer_module( 2025-09-07T07:03:26.9038639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9039030Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9039484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9039988Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9040471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9041058Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9041594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9041993Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9042407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:26.9042813Z x = self.lin1(input) 2025-09-07T07:03:26.9042917Z 2025-09-07T07:03:26.9043039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9043424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9043782Z return mod(**inputs) 2025-09-07T07:03:26.9044211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9044668Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9045127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9045561Z return self.transformer( 2025-09-07T07:03:26.9045968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9046377Z layer_outputs = layer_module( 2025-09-07T07:03:26.9046724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9047121Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9047605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9048096Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9048584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9049160Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9049677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9050064Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9050484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:26.9050917Z x = self.activation(x) 2025-09-07T07:03:26.9051253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:26.9051597Z return self.act(input) 2025-09-07T07:03:26.9051718Z 2025-09-07T07:03:26.9051828Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9052221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9052565Z return mod(**inputs) 2025-09-07T07:03:26.9052981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9053416Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9053859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9054310Z return self.transformer( 2025-09-07T07:03:26.9054751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9055202Z layer_outputs = layer_module( 2025-09-07T07:03:26.9055581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9055976Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9056413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9056876Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9057338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9057918Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9058474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9058904Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9059371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:26.9059838Z x = self.lin2(x) 2025-09-07T07:03:26.9059953Z 2025-09-07T07:03:26.9060070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9060486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9060861Z return mod(**inputs) 2025-09-07T07:03:26.9061287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9061733Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9062181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9062637Z return self.transformer( 2025-09-07T07:03:26.9063060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9063512Z layer_outputs = layer_module( 2025-09-07T07:03:26.9063881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9064270Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9064730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9065173Z sa_output = self.attention( 2025-09-07T07:03:26.9065683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:26.9066222Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:26.9066437Z 2025-09-07T07:03:26.9066559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9066970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9067322Z return mod(**inputs) 2025-09-07T07:03:26.9067760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9068213Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9068683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9069129Z return self.transformer( 2025-09-07T07:03:26.9069558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9069998Z layer_outputs = layer_module( 2025-09-07T07:03:26.9070375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9070766Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9071219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9071674Z sa_output = self.attention( 2025-09-07T07:03:26.9072078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:26.9072539Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9072718Z 2025-09-07T07:03:26.9072831Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9073185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9073499Z return mod(**inputs) 2025-09-07T07:03:26.9073882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9074293Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9074703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9075114Z return self.transformer( 2025-09-07T07:03:26.9075501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9075936Z layer_outputs = layer_module( 2025-09-07T07:03:26.9076283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9076649Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9077070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9077492Z sa_output = self.attention( 2025-09-07T07:03:26.9077922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:26.9078399Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9078582Z 2025-09-07T07:03:26.9078674Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.9078917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9079312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9079676Z return mod(**inputs) 2025-09-07T07:03:26.9080110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9080569Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9081011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9081452Z return self.transformer( 2025-09-07T07:03:26.9081856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9082278Z layer_outputs = layer_module( 2025-09-07T07:03:26.9082662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9083065Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9083516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9083934Z sa_output = self.attention( 2025-09-07T07:03:26.9084342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:26.9084817Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:26.9085013Z 2025-09-07T07:03:26.9085122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9085488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9085820Z return mod(**inputs) 2025-09-07T07:03:26.9086218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9086633Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9087050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9087472Z return self.transformer( 2025-09-07T07:03:26.9087874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9088295Z layer_outputs = layer_module( 2025-09-07T07:03:26.9088640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9089016Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9089442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9089860Z sa_output = self.attention( 2025-09-07T07:03:26.9090259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:26.9090723Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:26.9090871Z 2025-09-07T07:03:26.9090979Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9091346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9091674Z return mod(**inputs) 2025-09-07T07:03:26.9092059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9092497Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9092917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9093333Z return self.transformer( 2025-09-07T07:03:26.9093734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9094155Z layer_outputs = layer_module( 2025-09-07T07:03:26.9094512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9094882Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9095314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9095770Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9096235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9096793Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9097333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9097749Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9098186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:26.9098618Z x = self.lin1(input) 2025-09-07T07:03:26.9098741Z 2025-09-07T07:03:26.9098856Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9099251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9099606Z return mod(**inputs) 2025-09-07T07:03:26.9100010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9100435Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9100859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9101278Z return self.transformer( 2025-09-07T07:03:26.9101675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9102098Z layer_outputs = layer_module( 2025-09-07T07:03:26.9102455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9102827Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9103253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9103710Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9104169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9104719Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9105262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9105765Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9106222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:26.9106700Z x = self.activation(x) 2025-09-07T07:03:26.9107064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:26.9107471Z return self.act(input) 2025-09-07T07:03:26.9107594Z 2025-09-07T07:03:26.9107720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9108118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9108485Z return mod(**inputs) 2025-09-07T07:03:26.9108920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9109383Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9109831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9110290Z return self.transformer( 2025-09-07T07:03:26.9110728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9111189Z layer_outputs = layer_module( 2025-09-07T07:03:26.9111577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9111981Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9112445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9112982Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9113493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9114103Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9114675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9115120Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9115603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:26.9116148Z x = self.lin2(x) 2025-09-07T07:03:26.9116258Z 2025-09-07T07:03:26.9116384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9116781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9117144Z return mod(**inputs) 2025-09-07T07:03:26.9117580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9118056Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9118515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9118986Z return self.transformer( 2025-09-07T07:03:26.9119438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9120081Z layer_outputs = layer_module( 2025-09-07T07:03:26.9120487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9120884Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9121359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9121867Z sa_output = self.attention( 2025-09-07T07:03:26.9122299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:26.9122813Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:26.9123011Z 2025-09-07T07:03:26.9123122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9123512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9123923Z return mod(**inputs) 2025-09-07T07:03:26.9124341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9124805Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9125249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9125698Z return self.transformer( 2025-09-07T07:03:26.9126134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9126583Z layer_outputs = layer_module( 2025-09-07T07:03:26.9126954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9127346Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9127801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9128253Z sa_output = self.attention( 2025-09-07T07:03:26.9128692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:26.9129162Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9129347Z 2025-09-07T07:03:26.9129452Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9129841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9130176Z return mod(**inputs) 2025-09-07T07:03:26.9130568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9131000Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9131409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9131823Z return self.transformer( 2025-09-07T07:03:26.9132225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9132637Z layer_outputs = layer_module( 2025-09-07T07:03:26.9132997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9133372Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9133803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9134229Z sa_output = self.attention( 2025-09-07T07:03:26.9134636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:26.9135098Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9135283Z 2025-09-07T07:03:26.9135366Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.9135610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9135963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9136308Z return mod(**inputs) 2025-09-07T07:03:26.9136699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9137120Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9137539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9137953Z return self.transformer( 2025-09-07T07:03:26.9138362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9138801Z layer_outputs = layer_module( 2025-09-07T07:03:26.9139156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9139552Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9139998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9140440Z sa_output = self.attention( 2025-09-07T07:03:26.9140871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:26.9141385Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:26.9141585Z 2025-09-07T07:03:26.9141706Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9142091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9142443Z return mod(**inputs) 2025-09-07T07:03:26.9142964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9143425Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9143887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9144346Z return self.transformer( 2025-09-07T07:03:26.9144813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9145261Z layer_outputs = layer_module( 2025-09-07T07:03:26.9145700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9146110Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9146568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9147013Z sa_output = self.attention( 2025-09-07T07:03:26.9147446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:26.9147915Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:26.9148067Z 2025-09-07T07:03:26.9148180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9148578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9148934Z return mod(**inputs) 2025-09-07T07:03:26.9149356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9149798Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9150239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9150689Z return self.transformer( 2025-09-07T07:03:26.9151118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9151566Z layer_outputs = layer_module( 2025-09-07T07:03:26.9151965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9152363Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9152810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9153295Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9153778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9154374Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9154941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9155371Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9155840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:26.9156273Z x = self.lin1(input) 2025-09-07T07:03:26.9156380Z 2025-09-07T07:03:26.9156484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9156852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9157175Z return mod(**inputs) 2025-09-07T07:03:26.9157563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9157970Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9158379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9158788Z return self.transformer( 2025-09-07T07:03:26.9159199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9159612Z layer_outputs = layer_module( 2025-09-07T07:03:26.9159979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9160355Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9160782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9161249Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9161712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9162261Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9162789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9163196Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9163630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:26.9164094Z x = self.activation(x) 2025-09-07T07:03:26.9164416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:26.9164762Z return self.act(input) 2025-09-07T07:03:26.9164871Z 2025-09-07T07:03:26.9164982Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9165342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9165655Z return mod(**inputs) 2025-09-07T07:03:26.9166046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9166511Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9166931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9167349Z return self.transformer( 2025-09-07T07:03:26.9167749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9168162Z layer_outputs = layer_module( 2025-09-07T07:03:26.9168516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9168895Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9169306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9169773Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9170246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9170796Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9171316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9171707Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9172136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:26.9172549Z x = self.lin2(x) 2025-09-07T07:03:26.9172651Z 2025-09-07T07:03:26.9172766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9173133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9173456Z return mod(**inputs) 2025-09-07T07:03:26.9173871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9174287Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9174724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9175134Z return self.transformer( 2025-09-07T07:03:26.9175526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9175936Z layer_outputs = layer_module( 2025-09-07T07:03:26.9176297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9176649Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9177066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9177529Z sa_output = self.attention( 2025-09-07T07:03:26.9177955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:26.9178436Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:26.9178623Z 2025-09-07T07:03:26.9178737Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9179098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9179435Z return mod(**inputs) 2025-09-07T07:03:26.9179832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9180256Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9180681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9181149Z return self.transformer( 2025-09-07T07:03:26.9181576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9182030Z layer_outputs = layer_module( 2025-09-07T07:03:26.9182449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9182815Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9183239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9183682Z sa_output = self.attention( 2025-09-07T07:03:26.9184116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:26.9184625Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9184816Z 2025-09-07T07:03:26.9184930Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9185321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9185756Z return mod(**inputs) 2025-09-07T07:03:26.9186190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9186656Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9187114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9187566Z return self.transformer( 2025-09-07T07:03:26.9187988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9188409Z layer_outputs = layer_module( 2025-09-07T07:03:26.9188789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9189173Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9189650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9190069Z sa_output = self.attention( 2025-09-07T07:03:26.9190473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:26.9190936Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:26.9191127Z 2025-09-07T07:03:26.9191210Z cudagraph partition due to non gpu ops 2025-09-07T07:03:26.9191457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9191826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9192153Z return mod(**inputs) 2025-09-07T07:03:26.9192545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9192983Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9193397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9193820Z return self.transformer( 2025-09-07T07:03:26.9194218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9194644Z layer_outputs = layer_module( 2025-09-07T07:03:26.9195002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9195373Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9195802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9196277Z sa_output = self.attention( 2025-09-07T07:03:26.9196687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:26.9197166Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:26.9197353Z 2025-09-07T07:03:26.9197468Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9197842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9198163Z return mod(**inputs) 2025-09-07T07:03:26.9198585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9199013Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9199431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9199856Z return self.transformer( 2025-09-07T07:03:26.9200255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9200677Z layer_outputs = layer_module( 2025-09-07T07:03:26.9201031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9201413Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9201835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:26.9202259Z sa_output = self.attention( 2025-09-07T07:03:26.9202666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:26.9203106Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:26.9203247Z 2025-09-07T07:03:26.9203388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9203751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9204083Z return mod(**inputs) 2025-09-07T07:03:26.9204491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9204918Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9205328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9205754Z return self.transformer( 2025-09-07T07:03:26.9206159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9206581Z layer_outputs = layer_module( 2025-09-07T07:03:26.9206939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9207305Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9207736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9208202Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9208662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9209214Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9209741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9210144Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9210578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:26.9211019Z x = self.lin1(input) 2025-09-07T07:03:26.9211126Z 2025-09-07T07:03:26.9211238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9211602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9211937Z return mod(**inputs) 2025-09-07T07:03:26.9212336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9212761Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9213170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9213603Z return self.transformer( 2025-09-07T07:03:26.9214004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9214422Z layer_outputs = layer_module( 2025-09-07T07:03:26.9214784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9215148Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9215572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9216032Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9216486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9217032Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9217548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9217966Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9218394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:26.9218818Z x = self.activation(x) 2025-09-07T07:03:26.9219185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:26.9219682Z return self.act(input) 2025-09-07T07:03:26.9219817Z 2025-09-07T07:03:26.9219931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9220324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9220682Z return mod(**inputs) 2025-09-07T07:03:26.9221098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-09-07T07:03:26.9221571Z dlbrt_output = self.distilbert( 2025-09-07T07:03:26.9222027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:26.9222485Z return self.transformer( 2025-09-07T07:03:26.9222918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:26.9223368Z layer_outputs = layer_module( 2025-09-07T07:03:26.9223745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:26.9224139Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:26.9224591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:26.9225090Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:26.9225573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:26.9226309Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:26.9226891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:26.9227340Z return forward_fn(*input_tensors) 2025-09-07T07:03:26.9227798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:26.9228247Z x = self.lin2(x) 2025-09-07T07:03:26.9228361Z 2025-09-07T07:03:26.9228473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9228903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9229281Z return mod(**inputs) 2025-09-07T07:03:26.9229704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 836, in forward 2025-09-07T07:03:26.9230256Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-09-07T07:03:26.9230503Z 2025-09-07T07:03:26.9230617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9231017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9231374Z return mod(**inputs) 2025-09-07T07:03:26.9231795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 839, in forward 2025-09-07T07:03:26.9232376Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-09-07T07:03:26.9232663Z 2025-09-07T07:03:26.9232773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:26.9233148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:26.9233490Z return mod(**inputs) 2025-09-07T07:03:26.9233938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 843, in forward 2025-09-07T07:03:26.9234556Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-09-07T07:03:26.9234834Z 2025-09-07T07:03:36.7878334Z Compilation time (from dynamo_timed): 13.735121315 2025-09-07T07:03:36.7880090Z pass 2025-09-07T07:03:36.7885195Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:36.7890097Z TIMING: _recursive_pre_grad_passes:0.00538 _recursive_joint_graph_passes:0.25169 _recursive_post_grad_passes:0.04952 async_compile.wait:0.75662 code_gen:9.53463 inductor_compile:10.52153 backend_compile:12.30123 gc:0.00078 entire_frame_compile:13.73512 total_wall_time:13.73512 2025-09-07T07:03:36.7891340Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:6654 | FakeTensor.__torch_dispatch__:2344 | ProxyTorchDispatchMode.__torch_dispatch__:2359 2025-09-07T07:03:36.7891927Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-09-07T07:03:39.3379540Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:03:39.3380448Z import pynvml # type: ignore[import] 2025-09-07T07:03:42.1312289Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:03:42.1313259Z from pkg_resources import resource_filename 2025-09-07T07:03:42.7866384Z 2025-09-07T07:03:43.3739464Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:03:43.3739818Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:03:43.3744446Z cpu eval DistilBertForQuestionAnswering 2025-09-07T07:03:43.5128661Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:43.5693378Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:43.6222894Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:48.6714053Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6720642Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6722080Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6722354Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6722601Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6722831Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6723103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6723527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6723932Z return mod(**inputs) 2025-09-07T07:03:48.6724421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6724907Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6725395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6725857Z return self.transformer( 2025-09-07T07:03:48.6726308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6726887Z layer_outputs = layer_module( 2025-09-07T07:03:48.6727314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6727783Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6728258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6728782Z sa_output = self.attention( 2025-09-07T07:03:48.6729227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:48.6729758Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:48.6729972Z 2025-09-07T07:03:48.6730091Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6730494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6730848Z return mod(**inputs) 2025-09-07T07:03:48.6731289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6731766Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6732245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6732712Z return self.transformer( 2025-09-07T07:03:48.6733145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6733594Z layer_outputs = layer_module( 2025-09-07T07:03:48.6737117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6737535Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6737992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6738443Z sa_output = self.attention( 2025-09-07T07:03:48.6738884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:48.6739374Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6739557Z 2025-09-07T07:03:48.6739679Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6740068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6740434Z return mod(**inputs) 2025-09-07T07:03:48.6740865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6742040Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6742496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6742958Z return self.transformer( 2025-09-07T07:03:48.6743396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6743860Z layer_outputs = layer_module( 2025-09-07T07:03:48.6744249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6744648Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6745098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6745541Z sa_output = self.attention( 2025-09-07T07:03:48.6746265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:48.6746780Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6746982Z 2025-09-07T07:03:48.6747072Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6747371Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6747769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6748128Z return mod(**inputs) 2025-09-07T07:03:48.6748585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6749058Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6749520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6749980Z return self.transformer( 2025-09-07T07:03:48.6750412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6750866Z layer_outputs = layer_module( 2025-09-07T07:03:48.6751246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6751660Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6752125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6752575Z sa_output = self.attention( 2025-09-07T07:03:48.6753017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:48.6753508Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:48.6753761Z 2025-09-07T07:03:48.6753874Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6754251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6754572Z return mod(**inputs) 2025-09-07T07:03:48.6755008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6755437Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6755863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6756274Z return self.transformer( 2025-09-07T07:03:48.6756676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6757090Z layer_outputs = layer_module( 2025-09-07T07:03:48.6757444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6757836Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6758247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6758658Z sa_output = self.attention( 2025-09-07T07:03:48.6759060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:48.6759479Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:48.6759615Z 2025-09-07T07:03:48.6759729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6760090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6760418Z return mod(**inputs) 2025-09-07T07:03:48.6760813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6761234Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6761644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6762053Z return self.transformer( 2025-09-07T07:03:48.6762466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6762879Z layer_outputs = layer_module( 2025-09-07T07:03:48.6763245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6763605Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6764025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6764479Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6764926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6765473Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6765995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6766406Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6766844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:48.6767272Z x = self.lin1(input) 2025-09-07T07:03:48.6767375Z 2025-09-07T07:03:48.6767487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6767840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6768203Z return mod(**inputs) 2025-09-07T07:03:48.6768601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6769038Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6769460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6769881Z return self.transformer( 2025-09-07T07:03:48.6770292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6770703Z layer_outputs = layer_module( 2025-09-07T07:03:48.6771051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6771403Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6771822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6772317Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6772758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6773294Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6773801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6774200Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6774624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:48.6775053Z x = self.activation(x) 2025-09-07T07:03:48.6775389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:48.6775826Z return self.act(input) 2025-09-07T07:03:48.6775946Z 2025-09-07T07:03:48.6776050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6776414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6776741Z return mod(**inputs) 2025-09-07T07:03:48.6777181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6777622Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6778064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6778478Z return self.transformer( 2025-09-07T07:03:48.6778874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6779280Z layer_outputs = layer_module( 2025-09-07T07:03:48.6779630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6779990Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6780408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6780863Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6781302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6781908Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6782516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6783016Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6783689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:48.6784240Z x = self.lin2(x) 2025-09-07T07:03:48.6784376Z 2025-09-07T07:03:48.6784495Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6784996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6785427Z return mod(**inputs) 2025-09-07T07:03:48.6786003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6786593Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6787104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6787649Z return self.transformer( 2025-09-07T07:03:48.6788181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6788743Z layer_outputs = layer_module( 2025-09-07T07:03:48.6800260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6800689Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6801166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6801608Z sa_output = self.attention( 2025-09-07T07:03:48.6802035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:48.6802529Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:48.6802727Z 2025-09-07T07:03:48.6802845Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6803232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6803589Z return mod(**inputs) 2025-09-07T07:03:48.6804000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6804453Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6804984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6805443Z return self.transformer( 2025-09-07T07:03:48.6805909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6806368Z layer_outputs = layer_module( 2025-09-07T07:03:48.6806758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6807140Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6807572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6808004Z sa_output = self.attention( 2025-09-07T07:03:48.6808451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:48.6808933Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6809117Z 2025-09-07T07:03:48.6809230Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6809611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6809952Z return mod(**inputs) 2025-09-07T07:03:48.6810362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6810831Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6811260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6811684Z return self.transformer( 2025-09-07T07:03:48.6812094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6812519Z layer_outputs = layer_module( 2025-09-07T07:03:48.6812885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6813244Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6813659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6814070Z sa_output = self.attention( 2025-09-07T07:03:48.6814471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:48.6815017Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6815209Z 2025-09-07T07:03:48.6815296Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6815543Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6815915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6816250Z return mod(**inputs) 2025-09-07T07:03:48.6816652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6817072Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6817502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6817927Z return self.transformer( 2025-09-07T07:03:48.6818340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6818759Z layer_outputs = layer_module( 2025-09-07T07:03:48.6819114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6819488Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6820243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6820671Z sa_output = self.attention( 2025-09-07T07:03:48.6821143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:48.6821663Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:48.6821869Z 2025-09-07T07:03:48.6821997Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6822397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6822748Z return mod(**inputs) 2025-09-07T07:03:48.6823178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6823641Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6824096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6824548Z return self.transformer( 2025-09-07T07:03:48.6824970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6825415Z layer_outputs = layer_module( 2025-09-07T07:03:48.6825859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6826311Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6826762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6827208Z sa_output = self.attention( 2025-09-07T07:03:48.6827623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:48.6828057Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:48.6828198Z 2025-09-07T07:03:48.6828311Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6828676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6829013Z return mod(**inputs) 2025-09-07T07:03:48.6829419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6829857Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6830281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6830725Z return self.transformer( 2025-09-07T07:03:48.6831143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6831566Z layer_outputs = layer_module( 2025-09-07T07:03:48.6831924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6832287Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6832711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6833173Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6833620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6834166Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6834694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6835118Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6835545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:48.6835963Z x = self.lin1(input) 2025-09-07T07:03:48.6836096Z 2025-09-07T07:03:48.6836216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6836571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6836896Z return mod(**inputs) 2025-09-07T07:03:48.6837288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6837715Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6838144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6838563Z return self.transformer( 2025-09-07T07:03:48.6838974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6839403Z layer_outputs = layer_module( 2025-09-07T07:03:48.6839753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6840112Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6840526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6841001Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6841446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6841996Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6842519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6842933Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6843347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:48.6843761Z x = self.activation(x) 2025-09-07T07:03:48.6844087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:48.6844421Z return self.act(input) 2025-09-07T07:03:48.6844537Z 2025-09-07T07:03:48.6844642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6845041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6845381Z return mod(**inputs) 2025-09-07T07:03:48.6845803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6846256Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6846708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6847132Z return self.transformer( 2025-09-07T07:03:48.6847543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6847958Z layer_outputs = layer_module( 2025-09-07T07:03:48.6848315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6848690Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6849120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6849584Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6850063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6850673Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6851202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6851605Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6852028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:48.6852435Z x = self.lin2(x) 2025-09-07T07:03:48.6852548Z 2025-09-07T07:03:48.6852654Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6853028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6853370Z return mod(**inputs) 2025-09-07T07:03:48.6853766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6854191Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6854632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6855081Z return self.transformer( 2025-09-07T07:03:48.6855488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6855944Z layer_outputs = layer_module( 2025-09-07T07:03:48.6856316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6856704Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6857133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6857554Z sa_output = self.attention( 2025-09-07T07:03:48.6857961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:48.6858438Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:48.6858632Z 2025-09-07T07:03:48.6858741Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6859109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6859446Z return mod(**inputs) 2025-09-07T07:03:48.6859883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6860332Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6860757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6861176Z return self.transformer( 2025-09-07T07:03:48.6861595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6862041Z layer_outputs = layer_module( 2025-09-07T07:03:48.6862416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6862816Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6863266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6863705Z sa_output = self.attention( 2025-09-07T07:03:48.6864137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:48.6864660Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6864853Z 2025-09-07T07:03:48.6864973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6865381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6865840Z return mod(**inputs) 2025-09-07T07:03:48.6866294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6866779Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6867252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6867750Z return self.transformer( 2025-09-07T07:03:48.6868175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6868617Z layer_outputs = layer_module( 2025-09-07T07:03:48.6868995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6869397Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6869841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6870313Z sa_output = self.attention( 2025-09-07T07:03:48.6870746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:48.6871287Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6871485Z 2025-09-07T07:03:48.6871584Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6871840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6872231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6872596Z return mod(**inputs) 2025-09-07T07:03:48.6873020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6873479Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6873929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6874383Z return self.transformer( 2025-09-07T07:03:48.6874814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6875295Z layer_outputs = layer_module( 2025-09-07T07:03:48.6875671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6876075Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6876538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6876985Z sa_output = self.attention( 2025-09-07T07:03:48.6877425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:48.6877931Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:48.6878142Z 2025-09-07T07:03:48.6878254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6878650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6879002Z return mod(**inputs) 2025-09-07T07:03:48.6879434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6879893Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6880359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6880798Z return self.transformer( 2025-09-07T07:03:48.6881257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6881717Z layer_outputs = layer_module( 2025-09-07T07:03:48.6882111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6882523Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6882979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6883428Z sa_output = self.attention( 2025-09-07T07:03:48.6883873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:48.6884353Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:48.6884512Z 2025-09-07T07:03:48.6884629Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6885039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6885403Z return mod(**inputs) 2025-09-07T07:03:48.6885845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6886306Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6886777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6887225Z return self.transformer( 2025-09-07T07:03:48.6887651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6888094Z layer_outputs = layer_module( 2025-09-07T07:03:48.6888461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6888854Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6889307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6889793Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6890280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6890862Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6891433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6891857Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6892312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:48.6892754Z x = self.lin1(input) 2025-09-07T07:03:48.6892876Z 2025-09-07T07:03:48.6892984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6893356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6893689Z return mod(**inputs) 2025-09-07T07:03:48.6894092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6894530Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6894976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6895421Z return self.transformer( 2025-09-07T07:03:48.6895864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6896304Z layer_outputs = layer_module( 2025-09-07T07:03:48.6896686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6897057Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6897487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6897968Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6898458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6899042Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6899576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6899987Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6900439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:48.6900887Z x = self.activation(x) 2025-09-07T07:03:48.6901236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:48.6901606Z return self.act(input) 2025-09-07T07:03:48.6901751Z 2025-09-07T07:03:48.6901864Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6902261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6902619Z return mod(**inputs) 2025-09-07T07:03:48.6903022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6903460Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6903912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6904356Z return self.transformer( 2025-09-07T07:03:48.6904783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6905227Z layer_outputs = layer_module( 2025-09-07T07:03:48.6905670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6906086Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6906575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6907071Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6907577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6908164Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6908738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6909183Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6909638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:48.6910093Z x = self.lin2(x) 2025-09-07T07:03:48.6910214Z 2025-09-07T07:03:48.6910329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6910734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6911096Z return mod(**inputs) 2025-09-07T07:03:48.6911551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6912024Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6912508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6912965Z return self.transformer( 2025-09-07T07:03:48.6913397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6913856Z layer_outputs = layer_module( 2025-09-07T07:03:48.6914243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6914651Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6915115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6915567Z sa_output = self.attention( 2025-09-07T07:03:48.6916018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:48.6916536Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:48.6916736Z 2025-09-07T07:03:48.6916860Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6917258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6917635Z return mod(**inputs) 2025-09-07T07:03:48.6918080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6918550Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6919015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6919478Z return self.transformer( 2025-09-07T07:03:48.6920084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6920557Z layer_outputs = layer_module( 2025-09-07T07:03:48.6920952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6921357Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6921821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6922351Z sa_output = self.attention( 2025-09-07T07:03:48.6922786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:48.6923299Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6923488Z 2025-09-07T07:03:48.6923608Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6923996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6924370Z return mod(**inputs) 2025-09-07T07:03:48.6924799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6925263Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6925708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6926161Z return self.transformer( 2025-09-07T07:03:48.6926594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6927045Z layer_outputs = layer_module( 2025-09-07T07:03:48.6927450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6927839Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6928325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6928774Z sa_output = self.attention( 2025-09-07T07:03:48.6929209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:48.6929709Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6929904Z 2025-09-07T07:03:48.6929995Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6930257Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6930637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6930963Z return mod(**inputs) 2025-09-07T07:03:48.6931354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6931777Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6932195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6932620Z return self.transformer( 2025-09-07T07:03:48.6933027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6933476Z layer_outputs = layer_module( 2025-09-07T07:03:48.6933831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6934212Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6934625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6935035Z sa_output = self.attention( 2025-09-07T07:03:48.6935429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:48.6935918Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:48.6936112Z 2025-09-07T07:03:48.6936217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6936586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6936924Z return mod(**inputs) 2025-09-07T07:03:48.6937318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6937773Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6938199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6938621Z return self.transformer( 2025-09-07T07:03:48.6939029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6939440Z layer_outputs = layer_module( 2025-09-07T07:03:48.6939793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6940167Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6940592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6941031Z sa_output = self.attention( 2025-09-07T07:03:48.6941479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:48.6941953Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:48.6942108Z 2025-09-07T07:03:48.6942256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6942658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6943018Z return mod(**inputs) 2025-09-07T07:03:48.6943462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6943921Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6944370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6944815Z return self.transformer( 2025-09-07T07:03:48.6945260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6945786Z layer_outputs = layer_module( 2025-09-07T07:03:48.6946193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6946596Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6947057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6947560Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6948069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6948721Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6949291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6949736Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6950217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:48.6950673Z x = self.lin1(input) 2025-09-07T07:03:48.6950791Z 2025-09-07T07:03:48.6950911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6951319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6951690Z return mod(**inputs) 2025-09-07T07:03:48.6952134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6952609Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6953065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6953607Z return self.transformer( 2025-09-07T07:03:48.6954001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6954426Z layer_outputs = layer_module( 2025-09-07T07:03:48.6954784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6955151Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6955578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6956039Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6956501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6957053Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6957556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6957953Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6958391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:48.6958823Z x = self.activation(x) 2025-09-07T07:03:48.6959168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:48.6959521Z return self.act(input) 2025-09-07T07:03:48.6959638Z 2025-09-07T07:03:48.6959745Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6960119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6960463Z return mod(**inputs) 2025-09-07T07:03:48.6960849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6961271Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6961697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6962119Z return self.transformer( 2025-09-07T07:03:48.6962530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6962943Z layer_outputs = layer_module( 2025-09-07T07:03:48.6963303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6963695Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6964131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.6964604Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.6965069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.6965630Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.6966168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.6966580Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.6967005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:48.6967438Z x = self.lin2(x) 2025-09-07T07:03:48.6967550Z 2025-09-07T07:03:48.6967657Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6968043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6968372Z return mod(**inputs) 2025-09-07T07:03:48.6968756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6969184Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6969601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6970019Z return self.transformer( 2025-09-07T07:03:48.6970421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6970835Z layer_outputs = layer_module( 2025-09-07T07:03:48.6971197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6971569Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6972003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6972423Z sa_output = self.attention( 2025-09-07T07:03:48.6972861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:48.6973343Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:48.6973530Z 2025-09-07T07:03:48.6973662Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6974034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6974356Z return mod(**inputs) 2025-09-07T07:03:48.6974759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6975203Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6975621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6976043Z return self.transformer( 2025-09-07T07:03:48.6976461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6976910Z layer_outputs = layer_module( 2025-09-07T07:03:48.6977300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6977693Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6978135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6978609Z sa_output = self.attention( 2025-09-07T07:03:48.6979039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:48.6979522Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6979710Z 2025-09-07T07:03:48.6979832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6980213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6980563Z return mod(**inputs) 2025-09-07T07:03:48.6980997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6981454Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6981913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6982354Z return self.transformer( 2025-09-07T07:03:48.6982789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6983264Z layer_outputs = layer_module( 2025-09-07T07:03:48.6983638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6984036Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6984483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6984930Z sa_output = self.attention( 2025-09-07T07:03:48.6985361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:48.6985975Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.6986185Z 2025-09-07T07:03:48.6986280Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.6986561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6986979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6987325Z return mod(**inputs) 2025-09-07T07:03:48.6987777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6988203Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6988651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6989080Z return self.transformer( 2025-09-07T07:03:48.6989491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6989914Z layer_outputs = layer_module( 2025-09-07T07:03:48.6990265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6990635Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6991085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6991529Z sa_output = self.attention( 2025-09-07T07:03:48.6991933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:48.6992422Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:48.6992620Z 2025-09-07T07:03:48.6992725Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6993094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6993424Z return mod(**inputs) 2025-09-07T07:03:48.6993841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.6994277Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.6994732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.6995179Z return self.transformer( 2025-09-07T07:03:48.6995613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.6996060Z layer_outputs = layer_module( 2025-09-07T07:03:48.6996438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.6996817Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.6997275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.6997728Z sa_output = self.attention( 2025-09-07T07:03:48.6998182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:48.6998636Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:48.6998784Z 2025-09-07T07:03:48.6998903Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.6999288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.6999631Z return mod(**inputs) 2025-09-07T07:03:48.7000052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7000504Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7000957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7001402Z return self.transformer( 2025-09-07T07:03:48.7001824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7002269Z layer_outputs = layer_module( 2025-09-07T07:03:48.7002642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7003055Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7003502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.7004009Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.7004494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.7005079Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.7005641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.7006066Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.7006516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:48.7006958Z x = self.lin1(input) 2025-09-07T07:03:48.7007074Z 2025-09-07T07:03:48.7007195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7007587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7007935Z return mod(**inputs) 2025-09-07T07:03:48.7008369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7008802Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7009281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7009702Z return self.transformer( 2025-09-07T07:03:48.7010103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7010524Z layer_outputs = layer_module( 2025-09-07T07:03:48.7010882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7011251Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7011672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.7012135Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.7012603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.7013187Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.7013772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.7014202Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.7014660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:48.7015111Z x = self.activation(x) 2025-09-07T07:03:48.7015473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:48.7015849Z return self.act(input) 2025-09-07T07:03:48.7015971Z 2025-09-07T07:03:48.7016086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7016486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7016828Z return mod(**inputs) 2025-09-07T07:03:48.7017234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7017668Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7018117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7018534Z return self.transformer( 2025-09-07T07:03:48.7018933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7019371Z layer_outputs = layer_module( 2025-09-07T07:03:48.7019868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7020277Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7020743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.7021250Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.7021750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.7022348Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.7022932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.7023384Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.7023845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:48.7024299Z x = self.lin2(x) 2025-09-07T07:03:48.7024410Z 2025-09-07T07:03:48.7024574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7024986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7025359Z return mod(**inputs) 2025-09-07T07:03:48.7025876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7026358Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7026825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7027284Z return self.transformer( 2025-09-07T07:03:48.7027732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7028184Z layer_outputs = layer_module( 2025-09-07T07:03:48.7028561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7028962Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7029450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.7029892Z sa_output = self.attention( 2025-09-07T07:03:48.7030325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-09-07T07:03:48.7030816Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-09-07T07:03:48.7031019Z 2025-09-07T07:03:48.7031133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7031521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7031871Z return mod(**inputs) 2025-09-07T07:03:48.7032294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7032747Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7033212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7033670Z return self.transformer( 2025-09-07T07:03:48.7034133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7034558Z layer_outputs = layer_module( 2025-09-07T07:03:48.7034908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7035303Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7035729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.7036148Z sa_output = self.attention( 2025-09-07T07:03:48.7036552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-09-07T07:03:48.7037029Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.7037209Z 2025-09-07T07:03:48.7037312Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7037675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7038001Z return mod(**inputs) 2025-09-07T07:03:48.7038393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7038835Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7039256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7039670Z return self.transformer( 2025-09-07T07:03:48.7040078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7040493Z layer_outputs = layer_module( 2025-09-07T07:03:48.7040837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7041199Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7041615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.7042021Z sa_output = self.attention( 2025-09-07T07:03:48.7042420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-09-07T07:03:48.7042881Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-09-07T07:03:48.7043059Z 2025-09-07T07:03:48.7043148Z cudagraph partition due to non gpu ops 2025-09-07T07:03:48.7043387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7043737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7044095Z return mod(**inputs) 2025-09-07T07:03:48.7044483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7044910Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7045320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7045731Z return self.transformer( 2025-09-07T07:03:48.7046122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7046534Z layer_outputs = layer_module( 2025-09-07T07:03:48.7046879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7047232Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7047652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.7048070Z sa_output = self.attention( 2025-09-07T07:03:48.7048505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-09-07T07:03:48.7048978Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:03:48.7049163Z 2025-09-07T07:03:48.7049284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7049650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7049977Z return mod(**inputs) 2025-09-07T07:03:48.7050362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7050774Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7051205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7051630Z return self.transformer( 2025-09-07T07:03:48.7052024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7052427Z layer_outputs = layer_module( 2025-09-07T07:03:48.7052761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7053121Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7053532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-09-07T07:03:48.7053936Z sa_output = self.attention( 2025-09-07T07:03:48.7054385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-09-07T07:03:48.7054798Z attn_output = self.out_lin(attn_output) 2025-09-07T07:03:48.7054942Z 2025-09-07T07:03:48.7055045Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7055412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7055745Z return mod(**inputs) 2025-09-07T07:03:48.7056140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7056574Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7057004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7057413Z return self.transformer( 2025-09-07T07:03:48.7057808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7058238Z layer_outputs = layer_module( 2025-09-07T07:03:48.7058595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7058967Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7059392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.7059857Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.7060309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.7060861Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.7061391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.7061798Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.7062220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-09-07T07:03:48.7062629Z x = self.lin1(input) 2025-09-07T07:03:48.7062742Z 2025-09-07T07:03:48.7062866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7063241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7063582Z return mod(**inputs) 2025-09-07T07:03:48.7063997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7064431Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7064885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7065336Z return self.transformer( 2025-09-07T07:03:48.7065860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7066328Z layer_outputs = layer_module( 2025-09-07T07:03:48.7066721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7067132Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7067600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.7068105Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.7068558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.7069141Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.7069674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.7070078Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.7070499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-09-07T07:03:48.7070915Z x = self.activation(x) 2025-09-07T07:03:48.7071248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:03:48.7071598Z return self.act(input) 2025-09-07T07:03:48.7071709Z 2025-09-07T07:03:48.7071821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7072183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7072521Z return mod(**inputs) 2025-09-07T07:03:48.7072910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-09-07T07:03:48.7073350Z distilbert_output = self.distilbert( 2025-09-07T07:03:48.7073767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-09-07T07:03:48.7074169Z return self.transformer( 2025-09-07T07:03:48.7074559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-09-07T07:03:48.7074962Z layer_outputs = layer_module( 2025-09-07T07:03:48.7075317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:03:48.7075696Z return super().__call__(*args, **kwargs) 2025-09-07T07:03:48.7076103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-09-07T07:03:48.7076550Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-09-07T07:03:48.7077001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-09-07T07:03:48.7077521Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-09-07T07:03:48.7078033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:03:48.7078413Z return forward_fn(*input_tensors) 2025-09-07T07:03:48.7078837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-09-07T07:03:48.7079230Z x = self.lin2(x) 2025-09-07T07:03:48.7079326Z 2025-09-07T07:03:48.7079433Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7079777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7080093Z return mod(**inputs) 2025-09-07T07:03:48.7080471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1043, in forward 2025-09-07T07:03:48.7080922Z logits = self.qa_outputs(hidden_states) # (bs, max_query_len, 2) 2025-09-07T07:03:48.7081092Z 2025-09-07T07:03:48.7081252Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7081590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7081902Z return mod(**inputs) 2025-09-07T07:03:48.7082279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1061, in forward 2025-09-07T07:03:48.7082708Z start_loss = loss_fct(start_logits, start_positions) 2025-09-07T07:03:48.7082858Z 2025-09-07T07:03:48.7082984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:03:48.7083323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:03:48.7083639Z return mod(**inputs) 2025-09-07T07:03:48.7084015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1062, in forward 2025-09-07T07:03:48.7084437Z end_loss = loss_fct(end_logits, end_positions) 2025-09-07T07:03:48.7084585Z 2025-09-07T07:03:58.2442820Z Compilation time (from dynamo_timed): 13.493229137 2025-09-07T07:03:58.2447495Z pass 2025-09-07T07:03:58.2447984Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:03:58.2448813Z TIMING: _recursive_pre_grad_passes:0.00564 _recursive_joint_graph_passes:0.25257 _recursive_post_grad_passes:0.35765 async_compile.wait:0.72002 code_gen:9.23988 inductor_compile:10.24948 backend_compile:12.04882 gc:0.00085 entire_frame_compile:13.49323 total_wall_time:13.49323 2025-09-07T07:03:58.2449789Z STATS: call_* op count: 161 | FakeTensorMode.__torch_dispatch__:6699 | FakeTensor.__torch_dispatch__:2383 | ProxyTorchDispatchMode.__torch_dispatch__:2400 2025-09-07T07:03:58.2450524Z Dynamo produced 1 graphs covering 161 ops with 0 graph breaks (0 unique) 2025-09-07T07:04:00.9743824Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:04:00.9744735Z import pynvml # type: ignore[import] 2025-09-07T07:04:03.7304336Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:04:03.7305339Z from pkg_resources import resource_filename 2025-09-07T07:04:04.4037361Z 2025-09-07T07:04:06.4108760Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-09-07T07:04:06.4109543Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-09-07T07:04:06.4371600Z 2025-09-07T07:04:06.4372057Z loading model: 0it [00:02, ?it/s] 2025-09-07T07:04:06.4384641Z cpu eval DistillGPT2 2025-09-07T07:04:06.8291190Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:07.0164559Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:07.2018956Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:13.7784129Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7784460Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7784760Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7784998Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7785231Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7785470Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7785861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7786396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7786874Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7787344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7787761Z outputs = block( 2025-09-07T07:04:13.7788106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7788816Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7789274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7789709Z return func(*args, **kwargs) 2025-09-07T07:04:13.7790123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7790579Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7791029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7791461Z return func(*args, **kwargs) 2025-09-07T07:04:13.7791866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:04:13.7792413Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:04:13.7792944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7793523Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7793731Z 2025-09-07T07:04:13.7793835Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7794066Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7794302Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7794530Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7794795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7795322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7795779Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7796221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7796638Z outputs = block( 2025-09-07T07:04:13.7797007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7797407Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7797842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7798312Z return func(*args, **kwargs) 2025-09-07T07:04:13.7798734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7799177Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7799686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7800118Z return func(*args, **kwargs) 2025-09-07T07:04:13.7800531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7800998Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7801502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:04:13.7802164Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:04:13.7802381Z 2025-09-07T07:04:13.7802501Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7803010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7803467Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7803905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7804379Z outputs = block( 2025-09-07T07:04:13.7804791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7805239Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7805657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7806063Z return func(*args, **kwargs) 2025-09-07T07:04:13.7806467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7806902Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7807339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7807766Z return func(*args, **kwargs) 2025-09-07T07:04:13.7808170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7808621Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7809116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:04:13.7809660Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:04:13.7809862Z 2025-09-07T07:04:13.7809988Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7810484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7810939Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7811382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7811805Z outputs = block( 2025-09-07T07:04:13.7812165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7812585Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7813015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7813442Z return func(*args, **kwargs) 2025-09-07T07:04:13.7813854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7814302Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7814764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7815146Z return func(*args, **kwargs) 2025-09-07T07:04:13.7815542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:04:13.7815943Z attn_output = self.c_proj(attn_output) 2025-09-07T07:04:13.7816302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7816715Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7816902Z 2025-09-07T07:04:13.7817012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7817452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7817874Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7818293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7818691Z outputs = block( 2025-09-07T07:04:13.7819022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7819392Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7819976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7820409Z return func(*args, **kwargs) 2025-09-07T07:04:13.7820787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7821212Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7821634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:04:13.7822023Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:04:13.7822392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7822804Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7822980Z 2025-09-07T07:04:13.7823093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7823520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7823918Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7824314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7824735Z outputs = block( 2025-09-07T07:04:13.7825080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7825468Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7825941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7826347Z return func(*args, **kwargs) 2025-09-07T07:04:13.7826751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7827200Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7827635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:04:13.7828056Z hidden_states = self.act(hidden_states) 2025-09-07T07:04:13.7828437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:04:13.7828936Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:04:13.7829194Z 2025-09-07T07:04:13.7829338Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7829763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7830225Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7830624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7831006Z outputs = block( 2025-09-07T07:04:13.7831327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7831702Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7832087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7832472Z return func(*args, **kwargs) 2025-09-07T07:04:13.7832845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7833256Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7833676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:04:13.7834098Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:04:13.7834489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7834941Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7835127Z 2025-09-07T07:04:13.7835244Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7835695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7836118Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7836532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7836929Z outputs = block( 2025-09-07T07:04:13.7837279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7837673Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7838090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7838510Z return func(*args, **kwargs) 2025-09-07T07:04:13.7838900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7839381Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7839796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7840205Z return func(*args, **kwargs) 2025-09-07T07:04:13.7840603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:04:13.7841139Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:04:13.7841638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7842071Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7842263Z 2025-09-07T07:04:13.7842357Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7842590Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7842819Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7843037Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7843290Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7843737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7844195Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7844621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7845031Z outputs = block( 2025-09-07T07:04:13.7845392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7845766Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7846156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7846534Z return func(*args, **kwargs) 2025-09-07T07:04:13.7846968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7847375Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7847776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7848153Z return func(*args, **kwargs) 2025-09-07T07:04:13.7848523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7848938Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7849394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:04:13.7849914Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:04:13.7850103Z 2025-09-07T07:04:13.7850218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7850640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7851050Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7851452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7851834Z outputs = block( 2025-09-07T07:04:13.7852164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7852531Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7852917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7853300Z return func(*args, **kwargs) 2025-09-07T07:04:13.7853684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7854092Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7854483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7854864Z return func(*args, **kwargs) 2025-09-07T07:04:13.7855241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7855652Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7856102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:04:13.7856571Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:04:13.7856748Z 2025-09-07T07:04:13.7856855Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7857269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7857677Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7858067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7858445Z outputs = block( 2025-09-07T07:04:13.7858791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7859162Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7859560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7859940Z return func(*args, **kwargs) 2025-09-07T07:04:13.7860317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7860723Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7861129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7861502Z return func(*args, **kwargs) 2025-09-07T07:04:13.7861878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:04:13.7862280Z attn_output = self.c_proj(attn_output) 2025-09-07T07:04:13.7862655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7863065Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7863240Z 2025-09-07T07:04:13.7863347Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7863773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7864194Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7864592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7864965Z outputs = block( 2025-09-07T07:04:13.7865328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7865826Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7866267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7866692Z return func(*args, **kwargs) 2025-09-07T07:04:13.7867088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7867527Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7867945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:04:13.7868378Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:04:13.7868747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7869149Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7869333Z 2025-09-07T07:04:13.7869444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7869869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7870276Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7870666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7871050Z outputs = block( 2025-09-07T07:04:13.7871385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7871768Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7872155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7872531Z return func(*args, **kwargs) 2025-09-07T07:04:13.7872947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7873369Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7873810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:04:13.7874204Z hidden_states = self.act(hidden_states) 2025-09-07T07:04:13.7874558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:04:13.7875029Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:04:13.7875279Z 2025-09-07T07:04:13.7875387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7875813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7876223Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7876612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7876997Z outputs = block( 2025-09-07T07:04:13.7877327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7877692Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7878071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7878475Z return func(*args, **kwargs) 2025-09-07T07:04:13.7878853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7879273Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7879690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:04:13.7880094Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:04:13.7880469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7880880Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7881055Z 2025-09-07T07:04:13.7881168Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7881592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7881988Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7882385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7882787Z outputs = block( 2025-09-07T07:04:13.7883112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7883478Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7883865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7884248Z return func(*args, **kwargs) 2025-09-07T07:04:13.7884627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7885035Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7885430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7885818Z return func(*args, **kwargs) 2025-09-07T07:04:13.7886195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:04:13.7886701Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:04:13.7887212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7887634Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7887826Z 2025-09-07T07:04:13.7887947Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7888172Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7888388Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7888601Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7888857Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7889309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7889741Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7890159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7890550Z outputs = block( 2025-09-07T07:04:13.7890910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7891307Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7891720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7892124Z return func(*args, **kwargs) 2025-09-07T07:04:13.7892520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7892970Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7893387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7893796Z return func(*args, **kwargs) 2025-09-07T07:04:13.7894183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7894623Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7895106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:04:13.7895630Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:04:13.7895822Z 2025-09-07T07:04:13.7895942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7896386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7896812Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7897253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7897710Z outputs = block( 2025-09-07T07:04:13.7898029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7898403Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7898792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7899174Z return func(*args, **kwargs) 2025-09-07T07:04:13.7899547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7899944Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7900350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7900753Z return func(*args, **kwargs) 2025-09-07T07:04:13.7901148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7901581Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7902076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:04:13.7902582Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:04:13.7902765Z 2025-09-07T07:04:13.7902900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7903352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7903795Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7904212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7904628Z outputs = block( 2025-09-07T07:04:13.7904994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7905413Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7905921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7906346Z return func(*args, **kwargs) 2025-09-07T07:04:13.7906763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7907193Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7907589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7907959Z return func(*args, **kwargs) 2025-09-07T07:04:13.7908367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:04:13.7908769Z attn_output = self.c_proj(attn_output) 2025-09-07T07:04:13.7909142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7909545Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7909734Z 2025-09-07T07:04:13.7909842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7910266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7910668Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7911061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7911455Z outputs = block( 2025-09-07T07:04:13.7911789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7912183Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7912570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7912951Z return func(*args, **kwargs) 2025-09-07T07:04:13.7913320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7913745Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7914163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:04:13.7914559Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:04:13.7914916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7915323Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7915505Z 2025-09-07T07:04:13.7915613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7916037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7916440Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7916853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7917233Z outputs = block( 2025-09-07T07:04:13.7917577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7917950Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7918334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7918708Z return func(*args, **kwargs) 2025-09-07T07:04:13.7919074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7919487Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7920092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:04:13.7920494Z hidden_states = self.act(hidden_states) 2025-09-07T07:04:13.7920853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:04:13.7921316Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:04:13.7921562Z 2025-09-07T07:04:13.7921672Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7922095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7922558Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7922955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7923349Z outputs = block( 2025-09-07T07:04:13.7923681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7924071Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7924448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7924855Z return func(*args, **kwargs) 2025-09-07T07:04:13.7925234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7925655Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7926073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:04:13.7926508Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:04:13.7926878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7927282Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7927455Z 2025-09-07T07:04:13.7927572Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7927989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7928395Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7928785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7929161Z outputs = block( 2025-09-07T07:04:13.7929486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7929851Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7930237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7930616Z return func(*args, **kwargs) 2025-09-07T07:04:13.7931017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-09-07T07:04:13.7931446Z hidden_states = residual + feed_forward_hidden_states 2025-09-07T07:04:13.7931608Z 2025-09-07T07:04:13.7931740Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7932168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7932569Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7932969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7933345Z outputs = block( 2025-09-07T07:04:13.7933657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7934018Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7934395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7934765Z return func(*args, **kwargs) 2025-09-07T07:04:13.7935125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7935523Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7935923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7936308Z return func(*args, **kwargs) 2025-09-07T07:04:13.7936720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:04:13.7937207Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:04:13.7937675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7938082Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7938255Z 2025-09-07T07:04:13.7938347Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7938567Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7938785Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7938991Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7939222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7939631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7940071Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7940525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7940929Z outputs = block( 2025-09-07T07:04:13.7941278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7941666Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7942074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7942480Z return func(*args, **kwargs) 2025-09-07T07:04:13.7942882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7943317Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7943731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7944135Z return func(*args, **kwargs) 2025-09-07T07:04:13.7944546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7944996Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7945524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:04:13.7946097Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:04:13.7946306Z 2025-09-07T07:04:13.7946453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7946916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7947315Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7947707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7948075Z outputs = block( 2025-09-07T07:04:13.7948399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7948769Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7949141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7949498Z return func(*args, **kwargs) 2025-09-07T07:04:13.7949859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7950245Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7950633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7951001Z return func(*args, **kwargs) 2025-09-07T07:04:13.7951387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7951792Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7952233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:04:13.7952690Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:04:13.7952859Z 2025-09-07T07:04:13.7952967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7953358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7953739Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7954113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7954474Z outputs = block( 2025-09-07T07:04:13.7954785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7955179Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7955565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7955944Z return func(*args, **kwargs) 2025-09-07T07:04:13.7956328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7956712Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7957106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7957467Z return func(*args, **kwargs) 2025-09-07T07:04:13.7957825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:04:13.7958198Z attn_output = self.c_proj(attn_output) 2025-09-07T07:04:13.7958560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7958964Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7959141Z 2025-09-07T07:04:13.7959257Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7959706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7960115Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7960522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7960892Z outputs = block( 2025-09-07T07:04:13.7961216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7961579Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7961957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7962342Z return func(*args, **kwargs) 2025-09-07T07:04:13.7962715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7963127Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7963536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:04:13.7963920Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:04:13.7964274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7964670Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7964862Z 2025-09-07T07:04:13.7964976Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7965397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7965807Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7966206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7966595Z outputs = block( 2025-09-07T07:04:13.7966928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7967283Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7967661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7968038Z return func(*args, **kwargs) 2025-09-07T07:04:13.7968417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7968842Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7969274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:04:13.7969667Z hidden_states = self.act(hidden_states) 2025-09-07T07:04:13.7970029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:04:13.7970473Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:04:13.7970702Z 2025-09-07T07:04:13.7970814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7971219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7971616Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7972011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7972393Z outputs = block( 2025-09-07T07:04:13.7972706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7973064Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7973456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7973833Z return func(*args, **kwargs) 2025-09-07T07:04:13.7974215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.7974627Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.7975071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:04:13.7975499Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:04:13.7975892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7976324Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7976520Z 2025-09-07T07:04:13.7976628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7977058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7977474Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7977878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7978256Z outputs = block( 2025-09-07T07:04:13.7978591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7978969Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7979410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7979822Z return func(*args, **kwargs) 2025-09-07T07:04:13.7980216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7980647Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7981068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7981471Z return func(*args, **kwargs) 2025-09-07T07:04:13.7981862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:04:13.7982411Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:04:13.7982915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.7983368Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.7983552Z 2025-09-07T07:04:13.7983649Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7983877Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7984103Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7984326Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.7984583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7985033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7985467Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7985981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7986401Z outputs = block( 2025-09-07T07:04:13.7986771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7987171Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7987596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7988019Z return func(*args, **kwargs) 2025-09-07T07:04:13.7988453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7988885Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7989320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7989730Z return func(*args, **kwargs) 2025-09-07T07:04:13.7990128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7990571Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7991044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:04:13.7991563Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:04:13.7991762Z 2025-09-07T07:04:13.7991872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7992316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7992740Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7993158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.7993533Z outputs = block( 2025-09-07T07:04:13.7993864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.7994272Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.7994653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7995028Z return func(*args, **kwargs) 2025-09-07T07:04:13.7995401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.7995807Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.7996204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.7996582Z return func(*args, **kwargs) 2025-09-07T07:04:13.7996955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.7997363Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.7997821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:04:13.7998324Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:04:13.7998494Z 2025-09-07T07:04:13.7998599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.7999023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.7999430Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.7999830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8000211Z outputs = block( 2025-09-07T07:04:13.8000532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8000904Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8001290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8001671Z return func(*args, **kwargs) 2025-09-07T07:04:13.8002044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.8002440Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.8002864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8003242Z return func(*args, **kwargs) 2025-09-07T07:04:13.8003620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:04:13.8004031Z attn_output = self.c_proj(attn_output) 2025-09-07T07:04:13.8004400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.8004821Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.8005011Z 2025-09-07T07:04:13.8005133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8005589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8006031Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8006432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8006819Z outputs = block( 2025-09-07T07:04:13.8007168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8007559Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8007939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8008322Z return func(*args, **kwargs) 2025-09-07T07:04:13.8008720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.8009145Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.8009550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:04:13.8009945Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:04:13.8010307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.8010708Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.8010882Z 2025-09-07T07:04:13.8010998Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8011415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8011816Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8012213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8012619Z outputs = block( 2025-09-07T07:04:13.8012952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8013316Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8013704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8014086Z return func(*args, **kwargs) 2025-09-07T07:04:13.8014463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.8014877Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.8015294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:04:13.8015691Z hidden_states = self.act(hidden_states) 2025-09-07T07:04:13.8016050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:04:13.8016510Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:04:13.8016749Z 2025-09-07T07:04:13.8016858Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8017307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8017716Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8018148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8018532Z outputs = block( 2025-09-07T07:04:13.8018854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8019226Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8019801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8020201Z return func(*args, **kwargs) 2025-09-07T07:04:13.8020580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.8021018Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.8021462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:04:13.8021892Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:04:13.8022291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.8022734Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.8022994Z 2025-09-07T07:04:13.8023107Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8023557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8023993Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8024410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8024805Z outputs = block( 2025-09-07T07:04:13.8025153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8025552Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8026016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8026434Z return func(*args, **kwargs) 2025-09-07T07:04:13.8026825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-09-07T07:04:13.8027278Z hidden_states = residual + feed_forward_hidden_states 2025-09-07T07:04:13.8027496Z 2025-09-07T07:04:13.8027610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8028059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8028490Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8028902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8029302Z outputs = block( 2025-09-07T07:04:13.8029651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8030043Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8030447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8030852Z return func(*args, **kwargs) 2025-09-07T07:04:13.8031253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.8031688Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.8032107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8033534Z return func(*args, **kwargs) 2025-09-07T07:04:13.8033948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:04:13.8034518Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:04:13.8035025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.8035457Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.8035644Z 2025-09-07T07:04:13.8035734Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.8035977Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.8036205Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.8036428Z cudagraph partition due to non gpu ops 2025-09-07T07:04:13.8036678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8037135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8037561Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8037962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8038356Z outputs = block( 2025-09-07T07:04:13.8038704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8039119Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8039526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8039930Z return func(*args, **kwargs) 2025-09-07T07:04:13.8040300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.8040730Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.8041148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8041555Z return func(*args, **kwargs) 2025-09-07T07:04:13.8041955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.8042383Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.8042872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:04:13.8043394Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:04:13.8043581Z 2025-09-07T07:04:13.8043696Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8044114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8044522Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8044927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8045326Z outputs = block( 2025-09-07T07:04:13.8045672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8046066Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8046473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8046873Z return func(*args, **kwargs) 2025-09-07T07:04:13.8047335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.8047734Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.8048148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8048522Z return func(*args, **kwargs) 2025-09-07T07:04:13.8048936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:04:13.8049349Z attn_output, attn_weights = attention_interface( 2025-09-07T07:04:13.8049802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:04:13.8050268Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:04:13.8050433Z 2025-09-07T07:04:13.8050539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8050963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8051372Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8051772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8052153Z outputs = block( 2025-09-07T07:04:13.8052476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8052848Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8053241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8053621Z return func(*args, **kwargs) 2025-09-07T07:04:13.8054008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:04:13.8054415Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:04:13.8054814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8055252Z return func(*args, **kwargs) 2025-09-07T07:04:13.8055651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:04:13.8056070Z attn_output = self.c_proj(attn_output) 2025-09-07T07:04:13.8056438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.8056845Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.8057018Z 2025-09-07T07:04:13.8057133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8057556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8057974Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8058370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8058745Z outputs = block( 2025-09-07T07:04:13.8059079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8059443Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8059829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8060209Z return func(*args, **kwargs) 2025-09-07T07:04:13.8060592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.8061014Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.8061418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:04:13.8061812Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:04:13.8062182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.8062608Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.8062783Z 2025-09-07T07:04:13.8062894Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8063324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8063729Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8064122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8064509Z outputs = block( 2025-09-07T07:04:13.8064849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8065237Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8065729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8066157Z return func(*args, **kwargs) 2025-09-07T07:04:13.8066568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.8067031Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.8067487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:04:13.8067892Z hidden_states = self.act(hidden_states) 2025-09-07T07:04:13.8068241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:04:13.8068721Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:04:13.8068954Z 2025-09-07T07:04:13.8069059Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8069481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-09-07T07:04:13.8069888Z transformer_outputs = self.transformer( 2025-09-07T07:04:13.8070284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:04:13.8070663Z outputs = block( 2025-09-07T07:04:13.8070985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:13.8071354Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:13.8071739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:13.8072128Z return func(*args, **kwargs) 2025-09-07T07:04:13.8072482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:04:13.8072884Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:04:13.8073289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:04:13.8073683Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:04:13.8074045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:04:13.8074430Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:04:13.8074605Z 2025-09-07T07:04:13.8074709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:13.8075128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1207, in forward 2025-09-07T07:04:13.8075562Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-09-07T07:04:13.8075733Z 2025-09-07T07:04:24.2315799Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:24.2316416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-09-07T07:04:24.2317322Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-09-07T07:04:24.2317908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-09-07T07:04:24.2318455Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-09-07T07:04:24.2318719Z 2025-09-07T07:04:25.4137568Z Compilation time (from dynamo_timed): 16.807192624 2025-09-07T07:04:25.4295696Z pass 2025-09-07T07:04:25.4300441Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:25.4303315Z TIMING: gc:0.00468 entire_frame_compile:16.80719 _recursive_pre_grad_passes:0.00784 _recursive_joint_graph_passes:0.22929 _recursive_post_grad_passes:0.05537 async_compile.wait:1.45192 code_gen:11.33633 inductor_compile:12.05865 backend_compile:13.78015 total_wall_time:16.80719 2025-09-07T07:04:25.4304384Z STATS: call_* op count: 299 | FakeTensorMode.__torch_dispatch__:7239 | FakeTensor.__torch_dispatch__:2276 | ProxyTorchDispatchMode.__torch_dispatch__:2190 2025-09-07T07:04:25.4304953Z Dynamo produced 2 graphs covering 299 ops with 2 graph breaks (1 unique) 2025-09-07T07:04:28.4440148Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:04:28.4442887Z import pynvml # type: ignore[import] 2025-09-07T07:04:31.2406846Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:04:31.2409921Z from pkg_resources import resource_filename 2025-09-07T07:04:31.9258050Z 2025-09-07T07:04:31.9269006Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-09-07T07:04:31.9269835Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-09-07T07:04:32.1581433Z 2025-09-07T07:04:32.1582265Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:04:32.1596888Z cpu eval ElectraForCausalLM 2025-09-07T07:04:32.3573909Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:32.4437300Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:32.5291763Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:40.9010462Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9011084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9016772Z return mod(**inputs) 2025-09-07T07:04:40.9021999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9022576Z outputs = self.electra( 2025-09-07T07:04:40.9023014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-09-07T07:04:40.9023516Z hidden_states = self.embeddings_project(hidden_states) 2025-09-07T07:04:40.9023713Z 2025-09-07T07:04:40.9023845Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9024253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9024616Z return mod(**inputs) 2025-09-07T07:04:40.9025351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9026159Z outputs = self.electra( 2025-09-07T07:04:40.9026679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9027159Z hidden_states = self.encoder( 2025-09-07T07:04:40.9027654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9028053Z layer_outputs = layer_module( 2025-09-07T07:04:40.9028426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9028807Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9029256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9029707Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9030144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9030564Z return func(*args, **kwargs) 2025-09-07T07:04:40.9030994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9031410Z self_outputs = self.self( 2025-09-07T07:04:40.9031786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9032230Z return func(*args, **kwargs) 2025-09-07T07:04:40.9032627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9033044Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9033187Z 2025-09-07T07:04:40.9033313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9033676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9034010Z return mod(**inputs) 2025-09-07T07:04:40.9034390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9034788Z outputs = self.electra( 2025-09-07T07:04:40.9035162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9035551Z hidden_states = self.encoder( 2025-09-07T07:04:40.9035939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9036396Z layer_outputs = layer_module( 2025-09-07T07:04:40.9036752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9037118Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9037530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9037947Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9038343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9038722Z return func(*args, **kwargs) 2025-09-07T07:04:40.9039106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9039513Z self_outputs = self.self( 2025-09-07T07:04:40.9039883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9040260Z return func(*args, **kwargs) 2025-09-07T07:04:40.9040643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9041072Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9041215Z 2025-09-07T07:04:40.9041324Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9041711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9042043Z return mod(**inputs) 2025-09-07T07:04:40.9042422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9042839Z outputs = self.electra( 2025-09-07T07:04:40.9043215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9043608Z hidden_states = self.encoder( 2025-09-07T07:04:40.9044021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9044408Z layer_outputs = layer_module( 2025-09-07T07:04:40.9044759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9045180Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9045579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9045982Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9046400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9046792Z return func(*args, **kwargs) 2025-09-07T07:04:40.9047185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9047588Z self_outputs = self.self( 2025-09-07T07:04:40.9047962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9048351Z return func(*args, **kwargs) 2025-09-07T07:04:40.9048721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9049143Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9049287Z 2025-09-07T07:04:40.9049374Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9049598Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9049847Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9050248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9050603Z return mod(**inputs) 2025-09-07T07:04:40.9050988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9051389Z outputs = self.electra( 2025-09-07T07:04:40.9051769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9052168Z hidden_states = self.encoder( 2025-09-07T07:04:40.9052564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9052968Z layer_outputs = layer_module( 2025-09-07T07:04:40.9053320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9053686Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9054092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9054514Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9054923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9055305Z return func(*args, **kwargs) 2025-09-07T07:04:40.9055754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9056218Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9056692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9057109Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9057251Z 2025-09-07T07:04:40.9057359Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9057743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9058113Z return mod(**inputs) 2025-09-07T07:04:40.9058543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9058971Z outputs = self.electra( 2025-09-07T07:04:40.9059375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9059802Z hidden_states = self.encoder( 2025-09-07T07:04:40.9060226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9060649Z layer_outputs = layer_module( 2025-09-07T07:04:40.9061016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9061428Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9061856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9062299Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9062735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9063168Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9063650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9064195Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9064676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9065115Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9065266Z 2025-09-07T07:04:40.9065378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9065881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9066240Z return mod(**inputs) 2025-09-07T07:04:40.9066654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9067074Z outputs = self.electra( 2025-09-07T07:04:40.9067480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9067914Z hidden_states = self.encoder( 2025-09-07T07:04:40.9068335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9068759Z layer_outputs = layer_module( 2025-09-07T07:04:40.9069127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9069526Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9069958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9070391Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9070847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9071270Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9071749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9072263Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9072740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9073211Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9073718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9074100Z return self.act(input) 2025-09-07T07:04:40.9074220Z 2025-09-07T07:04:40.9074341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9074745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9075092Z return mod(**inputs) 2025-09-07T07:04:40.9075509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9075943Z outputs = self.electra( 2025-09-07T07:04:40.9076325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9076719Z hidden_states = self.encoder( 2025-09-07T07:04:40.9077118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9077514Z layer_outputs = layer_module( 2025-09-07T07:04:40.9077862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9078231Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9078631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9079050Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9079462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9079864Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9080299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9080787Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9081267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9081687Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9081829Z 2025-09-07T07:04:40.9081946Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9082323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9082651Z return mod(**inputs) 2025-09-07T07:04:40.9083037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9083439Z outputs = self.electra( 2025-09-07T07:04:40.9083863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9084259Z hidden_states = self.encoder( 2025-09-07T07:04:40.9084637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9085033Z layer_outputs = layer_module( 2025-09-07T07:04:40.9085377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9085777Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9086178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9086606Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9086999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9087379Z return func(*args, **kwargs) 2025-09-07T07:04:40.9087768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9088171Z self_outputs = self.self( 2025-09-07T07:04:40.9088531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9088912Z return func(*args, **kwargs) 2025-09-07T07:04:40.9089303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9089706Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9089854Z 2025-09-07T07:04:40.9089961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9090338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9090669Z return mod(**inputs) 2025-09-07T07:04:40.9091052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9091486Z outputs = self.electra( 2025-09-07T07:04:40.9091893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9092555Z hidden_states = self.encoder( 2025-09-07T07:04:40.9092969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9093390Z layer_outputs = layer_module( 2025-09-07T07:04:40.9093756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9094151Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9094580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9095018Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9095428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9095848Z return func(*args, **kwargs) 2025-09-07T07:04:40.9096261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9096684Z self_outputs = self.self( 2025-09-07T07:04:40.9097071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9097463Z return func(*args, **kwargs) 2025-09-07T07:04:40.9097874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9098305Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9098448Z 2025-09-07T07:04:40.9098566Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9099003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9099353Z return mod(**inputs) 2025-09-07T07:04:40.9099762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9100198Z outputs = self.electra( 2025-09-07T07:04:40.9100603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9101040Z hidden_states = self.encoder( 2025-09-07T07:04:40.9101468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9101920Z layer_outputs = layer_module( 2025-09-07T07:04:40.9102305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9102712Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9103136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9103577Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9103992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9104393Z return func(*args, **kwargs) 2025-09-07T07:04:40.9104806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9105221Z self_outputs = self.self( 2025-09-07T07:04:40.9105744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9106171Z return func(*args, **kwargs) 2025-09-07T07:04:40.9106600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9107045Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9107272Z 2025-09-07T07:04:40.9107360Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9107596Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9107854Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9108246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9108593Z return mod(**inputs) 2025-09-07T07:04:40.9109001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9109428Z outputs = self.electra( 2025-09-07T07:04:40.9109836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9110256Z hidden_states = self.encoder( 2025-09-07T07:04:40.9110677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9111081Z layer_outputs = layer_module( 2025-09-07T07:04:40.9111468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9111868Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9112281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9112707Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9113116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9113529Z return func(*args, **kwargs) 2025-09-07T07:04:40.9113931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9114399Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9114871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9115317Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9115463Z 2025-09-07T07:04:40.9115579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9115958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9116322Z return mod(**inputs) 2025-09-07T07:04:40.9116720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9117153Z outputs = self.electra( 2025-09-07T07:04:40.9117545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9117962Z hidden_states = self.encoder( 2025-09-07T07:04:40.9118368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9118783Z layer_outputs = layer_module( 2025-09-07T07:04:40.9119148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9119509Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9120107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9120533Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9120950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9121356Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9121788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9122338Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9122788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9123207Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9123347Z 2025-09-07T07:04:40.9123463Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9123823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9124153Z return mod(**inputs) 2025-09-07T07:04:40.9124545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9124954Z outputs = self.electra( 2025-09-07T07:04:40.9125338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9125737Z hidden_states = self.encoder( 2025-09-07T07:04:40.9126139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9126569Z layer_outputs = layer_module( 2025-09-07T07:04:40.9126924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9127291Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9127706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9128126Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9128543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9128950Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9129381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9129878Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9130345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9130823Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9131272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9131634Z return self.act(input) 2025-09-07T07:04:40.9131764Z 2025-09-07T07:04:40.9131877Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9132298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9132652Z return mod(**inputs) 2025-09-07T07:04:40.9133053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9133481Z outputs = self.electra( 2025-09-07T07:04:40.9133894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9134302Z hidden_states = self.encoder( 2025-09-07T07:04:40.9134696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9135094Z layer_outputs = layer_module( 2025-09-07T07:04:40.9135453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9135827Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9136239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9136655Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9137063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9137496Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9137951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9138483Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9138990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9139426Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9139586Z 2025-09-07T07:04:40.9139702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9140101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9140456Z return mod(**inputs) 2025-09-07T07:04:40.9140860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9141310Z outputs = self.electra( 2025-09-07T07:04:40.9141718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9142145Z hidden_states = self.encoder( 2025-09-07T07:04:40.9142560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9142975Z layer_outputs = layer_module( 2025-09-07T07:04:40.9143351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9143744Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9144178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9144614Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9145025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9145434Z return func(*args, **kwargs) 2025-09-07T07:04:40.9145922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9146349Z self_outputs = self.self( 2025-09-07T07:04:40.9146769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9147179Z return func(*args, **kwargs) 2025-09-07T07:04:40.9147593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9148016Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9148160Z 2025-09-07T07:04:40.9148278Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9148652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9148990Z return mod(**inputs) 2025-09-07T07:04:40.9149399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9149827Z outputs = self.electra( 2025-09-07T07:04:40.9150226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9150651Z hidden_states = self.encoder( 2025-09-07T07:04:40.9151076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9151503Z layer_outputs = layer_module( 2025-09-07T07:04:40.9151874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9152263Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9152723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9153163Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9153573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9153976Z return func(*args, **kwargs) 2025-09-07T07:04:40.9154378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9154802Z self_outputs = self.self( 2025-09-07T07:04:40.9155197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9155598Z return func(*args, **kwargs) 2025-09-07T07:04:40.9156005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9156439Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9156610Z 2025-09-07T07:04:40.9156723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9157124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9157470Z return mod(**inputs) 2025-09-07T07:04:40.9157872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9158306Z outputs = self.electra( 2025-09-07T07:04:40.9158714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9159136Z hidden_states = self.encoder( 2025-09-07T07:04:40.9159550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9159972Z layer_outputs = layer_module( 2025-09-07T07:04:40.9160352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9160743Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9161176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9161606Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9162062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9162446Z return func(*args, **kwargs) 2025-09-07T07:04:40.9162863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9163294Z self_outputs = self.self( 2025-09-07T07:04:40.9163685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9164094Z return func(*args, **kwargs) 2025-09-07T07:04:40.9164513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9164950Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9165101Z 2025-09-07T07:04:40.9165195Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9165428Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9165685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9166076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9166431Z return mod(**inputs) 2025-09-07T07:04:40.9166831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9167259Z outputs = self.electra( 2025-09-07T07:04:40.9167711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9168148Z hidden_states = self.encoder( 2025-09-07T07:04:40.9168561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9168999Z layer_outputs = layer_module( 2025-09-07T07:04:40.9169376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9169766Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9170192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9170604Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9171000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9171390Z return func(*args, **kwargs) 2025-09-07T07:04:40.9171790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9172306Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9172787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9173229Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9173387Z 2025-09-07T07:04:40.9173500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9173894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9174248Z return mod(**inputs) 2025-09-07T07:04:40.9174647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9175083Z outputs = self.electra( 2025-09-07T07:04:40.9175490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9175898Z hidden_states = self.encoder( 2025-09-07T07:04:40.9176287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9176687Z layer_outputs = layer_module( 2025-09-07T07:04:40.9177067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9177439Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9177866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9178280Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9178716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9179149Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9179620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9180135Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9180611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9181051Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9181206Z 2025-09-07T07:04:40.9181319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9181717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9182090Z return mod(**inputs) 2025-09-07T07:04:40.9182507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9182973Z outputs = self.electra( 2025-09-07T07:04:40.9183393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9183831Z hidden_states = self.encoder( 2025-09-07T07:04:40.9184251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9184691Z layer_outputs = layer_module( 2025-09-07T07:04:40.9185079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9185495Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9186047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9186519Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9186972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9187427Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9187892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9188408Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9188900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9189377Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9189798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9190182Z return self.act(input) 2025-09-07T07:04:40.9190309Z 2025-09-07T07:04:40.9190422Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9190820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9191177Z return mod(**inputs) 2025-09-07T07:04:40.9191585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9192011Z outputs = self.electra( 2025-09-07T07:04:40.9192434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9192871Z hidden_states = self.encoder( 2025-09-07T07:04:40.9193285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9193734Z layer_outputs = layer_module( 2025-09-07T07:04:40.9194100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9194503Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9194940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9195388Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9195826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9196248Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9196714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9197247Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9197732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9198150Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9198294Z 2025-09-07T07:04:40.9198424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9198794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9199126Z return mod(**inputs) 2025-09-07T07:04:40.9199511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9199918Z outputs = self.electra( 2025-09-07T07:04:40.9200325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9200749Z hidden_states = self.encoder( 2025-09-07T07:04:40.9201177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9201578Z layer_outputs = layer_module( 2025-09-07T07:04:40.9201925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9202296Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9202763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9203189Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9203583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9203962Z return func(*args, **kwargs) 2025-09-07T07:04:40.9204357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9204759Z self_outputs = self.self( 2025-09-07T07:04:40.9205132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9205513Z return func(*args, **kwargs) 2025-09-07T07:04:40.9205923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9206373Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9206527Z 2025-09-07T07:04:40.9206652Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9207044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9207388Z return mod(**inputs) 2025-09-07T07:04:40.9207793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9208196Z outputs = self.electra( 2025-09-07T07:04:40.9208603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9209012Z hidden_states = self.encoder( 2025-09-07T07:04:40.9209421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9209849Z layer_outputs = layer_module( 2025-09-07T07:04:40.9210222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9210613Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9211042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9211483Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9211890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9212298Z return func(*args, **kwargs) 2025-09-07T07:04:40.9212710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9213128Z self_outputs = self.self( 2025-09-07T07:04:40.9213549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9213968Z return func(*args, **kwargs) 2025-09-07T07:04:40.9214402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9214846Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9215009Z 2025-09-07T07:04:40.9215124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9215522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9215878Z return mod(**inputs) 2025-09-07T07:04:40.9216284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9216713Z outputs = self.electra( 2025-09-07T07:04:40.9217122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9217550Z hidden_states = self.encoder( 2025-09-07T07:04:40.9217987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9218409Z layer_outputs = layer_module( 2025-09-07T07:04:40.9218777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9219167Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9219716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9220175Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9220583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9220989Z return func(*args, **kwargs) 2025-09-07T07:04:40.9221410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9221838Z self_outputs = self.self( 2025-09-07T07:04:40.9222233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9222635Z return func(*args, **kwargs) 2025-09-07T07:04:40.9223148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9223591Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9223736Z 2025-09-07T07:04:40.9223884Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9224124Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9224376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9224771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9225136Z return mod(**inputs) 2025-09-07T07:04:40.9225604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9226043Z outputs = self.electra( 2025-09-07T07:04:40.9226453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9226892Z hidden_states = self.encoder( 2025-09-07T07:04:40.9227325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9227757Z layer_outputs = layer_module( 2025-09-07T07:04:40.9228128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9228591Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9229074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9229538Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9229947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9230351Z return func(*args, **kwargs) 2025-09-07T07:04:40.9230768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9231257Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9231782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9232216Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9232376Z 2025-09-07T07:04:40.9232492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9232881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9233229Z return mod(**inputs) 2025-09-07T07:04:40.9233675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9234100Z outputs = self.electra( 2025-09-07T07:04:40.9234510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9234932Z hidden_states = self.encoder( 2025-09-07T07:04:40.9235350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9235767Z layer_outputs = layer_module( 2025-09-07T07:04:40.9236139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9236528Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9236958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9237394Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9237821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9238242Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9238726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9239249Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9239752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9240189Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9240348Z 2025-09-07T07:04:40.9240458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9240856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9241212Z return mod(**inputs) 2025-09-07T07:04:40.9241636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9242082Z outputs = self.electra( 2025-09-07T07:04:40.9242491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9242917Z hidden_states = self.encoder( 2025-09-07T07:04:40.9243335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9243756Z layer_outputs = layer_module( 2025-09-07T07:04:40.9244130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9244547Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9244986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9245518Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9245951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9246377Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9246836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9247351Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9247842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9248312Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9248728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9249130Z return self.act(input) 2025-09-07T07:04:40.9249254Z 2025-09-07T07:04:40.9249372Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9249757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9250117Z return mod(**inputs) 2025-09-07T07:04:40.9250540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9250978Z outputs = self.electra( 2025-09-07T07:04:40.9251405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9251836Z hidden_states = self.encoder( 2025-09-07T07:04:40.9252259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9252698Z layer_outputs = layer_module( 2025-09-07T07:04:40.9253087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9253480Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9253927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9254405Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9254853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9255316Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9255786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9256334Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9256854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9257304Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9257452Z 2025-09-07T07:04:40.9257574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9257970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9258334Z return mod(**inputs) 2025-09-07T07:04:40.9258751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9259189Z outputs = self.electra( 2025-09-07T07:04:40.9259599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9260037Z hidden_states = self.encoder( 2025-09-07T07:04:40.9260496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9260934Z layer_outputs = layer_module( 2025-09-07T07:04:40.9261318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9261716Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9262178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9262621Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9263049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9263463Z return func(*args, **kwargs) 2025-09-07T07:04:40.9263876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9264313Z self_outputs = self.self( 2025-09-07T07:04:40.9264720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9265144Z return func(*args, **kwargs) 2025-09-07T07:04:40.9265621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9266094Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9266255Z 2025-09-07T07:04:40.9266369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9266770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9267122Z return mod(**inputs) 2025-09-07T07:04:40.9267520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9267946Z outputs = self.electra( 2025-09-07T07:04:40.9268359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9268786Z hidden_states = self.encoder( 2025-09-07T07:04:40.9269203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9269616Z layer_outputs = layer_module( 2025-09-07T07:04:40.9270012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9270400Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9270847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9271276Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9271691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9272092Z return func(*args, **kwargs) 2025-09-07T07:04:40.9272502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9272926Z self_outputs = self.self( 2025-09-07T07:04:40.9273306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9273708Z return func(*args, **kwargs) 2025-09-07T07:04:40.9274117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9274550Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9274695Z 2025-09-07T07:04:40.9274813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9275196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9275552Z return mod(**inputs) 2025-09-07T07:04:40.9275991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9276428Z outputs = self.electra( 2025-09-07T07:04:40.9276823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9277245Z hidden_states = self.encoder( 2025-09-07T07:04:40.9277661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9278089Z layer_outputs = layer_module( 2025-09-07T07:04:40.9278464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9278848Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9279285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9279735Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9280193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9280594Z return func(*args, **kwargs) 2025-09-07T07:04:40.9281009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9281441Z self_outputs = self.self( 2025-09-07T07:04:40.9281834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9282240Z return func(*args, **kwargs) 2025-09-07T07:04:40.9282652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9283100Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9283263Z 2025-09-07T07:04:40.9283366Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9283601Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9283863Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9284250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9284602Z return mod(**inputs) 2025-09-07T07:04:40.9285038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9285463Z outputs = self.electra( 2025-09-07T07:04:40.9285879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9286298Z hidden_states = self.encoder( 2025-09-07T07:04:40.9286721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9287164Z layer_outputs = layer_module( 2025-09-07T07:04:40.9287536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9287923Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9288355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9288791Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9289203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9289593Z return func(*args, **kwargs) 2025-09-07T07:04:40.9290013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9290521Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9291005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9291457Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9291607Z 2025-09-07T07:04:40.9291721Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9292122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9292484Z return mod(**inputs) 2025-09-07T07:04:40.9292901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9293338Z outputs = self.electra( 2025-09-07T07:04:40.9293762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9294185Z hidden_states = self.encoder( 2025-09-07T07:04:40.9294598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9295022Z layer_outputs = layer_module( 2025-09-07T07:04:40.9295385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9295792Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9296230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9296693Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9297140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9297571Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9298048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9298579Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9299073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9299532Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9299687Z 2025-09-07T07:04:40.9299802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9300208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9300593Z return mod(**inputs) 2025-09-07T07:04:40.9301012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9301452Z outputs = self.electra( 2025-09-07T07:04:40.9301880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9302320Z hidden_states = self.encoder( 2025-09-07T07:04:40.9302748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9303181Z layer_outputs = layer_module( 2025-09-07T07:04:40.9303562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9303976Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9304435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9304889Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9305343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9305839Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9306317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9306850Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9307371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9307864Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9308287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9308676Z return self.act(input) 2025-09-07T07:04:40.9308814Z 2025-09-07T07:04:40.9308929Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9309340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9309711Z return mod(**inputs) 2025-09-07T07:04:40.9310128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9310572Z outputs = self.electra( 2025-09-07T07:04:40.9310996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9311476Z hidden_states = self.encoder( 2025-09-07T07:04:40.9311902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9312340Z layer_outputs = layer_module( 2025-09-07T07:04:40.9312726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9313139Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9313590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9314057Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9314501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9314956Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9315437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9315986Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9316516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9316964Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9317113Z 2025-09-07T07:04:40.9317233Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9317649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9317996Z return mod(**inputs) 2025-09-07T07:04:40.9318405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9318834Z outputs = self.electra( 2025-09-07T07:04:40.9319252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9319852Z hidden_states = self.encoder( 2025-09-07T07:04:40.9320282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9320728Z layer_outputs = layer_module( 2025-09-07T07:04:40.9321118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9321510Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9321942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9322390Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9322814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9323296Z return func(*args, **kwargs) 2025-09-07T07:04:40.9323729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9324166Z self_outputs = self.self( 2025-09-07T07:04:40.9324584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9325015Z return func(*args, **kwargs) 2025-09-07T07:04:40.9325444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9325889Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9326040Z 2025-09-07T07:04:40.9326157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9326562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9326928Z return mod(**inputs) 2025-09-07T07:04:40.9327344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9327797Z outputs = self.electra( 2025-09-07T07:04:40.9328207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9328657Z hidden_states = self.encoder( 2025-09-07T07:04:40.9329081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9329511Z layer_outputs = layer_module( 2025-09-07T07:04:40.9329872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9330241Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9330649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9331065Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9331449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9331830Z return func(*args, **kwargs) 2025-09-07T07:04:40.9332242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9332645Z self_outputs = self.self( 2025-09-07T07:04:40.9333052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9333450Z return func(*args, **kwargs) 2025-09-07T07:04:40.9333863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9334298Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9334444Z 2025-09-07T07:04:40.9334565Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9334957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9335286Z return mod(**inputs) 2025-09-07T07:04:40.9335690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9336128Z outputs = self.electra( 2025-09-07T07:04:40.9336513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9336910Z hidden_states = self.encoder( 2025-09-07T07:04:40.9337307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9337707Z layer_outputs = layer_module( 2025-09-07T07:04:40.9338061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9338451Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9338855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9339270Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9339661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9340041Z return func(*args, **kwargs) 2025-09-07T07:04:40.9340435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9340849Z self_outputs = self.self( 2025-09-07T07:04:40.9341237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9341647Z return func(*args, **kwargs) 2025-09-07T07:04:40.9342060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9342518Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9342674Z 2025-09-07T07:04:40.9342763Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9342997Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9343257Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9343649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9343998Z return mod(**inputs) 2025-09-07T07:04:40.9344407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9344833Z outputs = self.electra( 2025-09-07T07:04:40.9345244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9345748Z hidden_states = self.encoder( 2025-09-07T07:04:40.9346182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9346611Z layer_outputs = layer_module( 2025-09-07T07:04:40.9346995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9347416Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9347841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9348299Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9348718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9349121Z return func(*args, **kwargs) 2025-09-07T07:04:40.9349526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9350014Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9350495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9350931Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9351082Z 2025-09-07T07:04:40.9351202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9351582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9351931Z return mod(**inputs) 2025-09-07T07:04:40.9352338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9352768Z outputs = self.electra( 2025-09-07T07:04:40.9353177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9353646Z hidden_states = self.encoder( 2025-09-07T07:04:40.9354059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9354476Z layer_outputs = layer_module( 2025-09-07T07:04:40.9354854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9355244Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9355661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9356100Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9356536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9371977Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9372694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9373328Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9373842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9374305Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9374484Z 2025-09-07T07:04:40.9374612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9375045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9375423Z return mod(**inputs) 2025-09-07T07:04:40.9375855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9376297Z outputs = self.electra( 2025-09-07T07:04:40.9376730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9377179Z hidden_states = self.encoder( 2025-09-07T07:04:40.9377616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9378056Z layer_outputs = layer_module( 2025-09-07T07:04:40.9378475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9378883Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9379353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9379804Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9380239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9380674Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9381142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9381662Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9382137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9382608Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9383027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9383403Z return self.act(input) 2025-09-07T07:04:40.9383524Z 2025-09-07T07:04:40.9383646Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9384039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9384429Z return mod(**inputs) 2025-09-07T07:04:40.9384841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9385270Z outputs = self.electra( 2025-09-07T07:04:40.9385778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9386236Z hidden_states = self.encoder( 2025-09-07T07:04:40.9386680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9387138Z layer_outputs = layer_module( 2025-09-07T07:04:40.9387520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9387910Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9388348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9388790Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9389254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9389673Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9390137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9390663Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9391148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9391584Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9391734Z 2025-09-07T07:04:40.9391849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9392251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9392613Z return mod(**inputs) 2025-09-07T07:04:40.9393024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9393453Z outputs = self.electra( 2025-09-07T07:04:40.9393872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9394294Z hidden_states = self.encoder( 2025-09-07T07:04:40.9394704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9395107Z layer_outputs = layer_module( 2025-09-07T07:04:40.9395456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9395833Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9396249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9396672Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9397068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9397446Z return func(*args, **kwargs) 2025-09-07T07:04:40.9397839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9398239Z self_outputs = self.self( 2025-09-07T07:04:40.9398613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9399017Z return func(*args, **kwargs) 2025-09-07T07:04:40.9399435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9399888Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9400046Z 2025-09-07T07:04:40.9400160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9400557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9400880Z return mod(**inputs) 2025-09-07T07:04:40.9401281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9401706Z outputs = self.electra( 2025-09-07T07:04:40.9402110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9402533Z hidden_states = self.encoder( 2025-09-07T07:04:40.9402948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9403343Z layer_outputs = layer_module( 2025-09-07T07:04:40.9403768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9404183Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9404615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9405042Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9405459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9405862Z return func(*args, **kwargs) 2025-09-07T07:04:40.9406254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9406650Z self_outputs = self.self( 2025-09-07T07:04:40.9407016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9407397Z return func(*args, **kwargs) 2025-09-07T07:04:40.9407788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9408205Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9408352Z 2025-09-07T07:04:40.9408464Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9408870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9409219Z return mod(**inputs) 2025-09-07T07:04:40.9409656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9410083Z outputs = self.electra( 2025-09-07T07:04:40.9410490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9410918Z hidden_states = self.encoder( 2025-09-07T07:04:40.9411340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9411766Z layer_outputs = layer_module( 2025-09-07T07:04:40.9412138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9412540Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9412986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9413425Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9413841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9414236Z return func(*args, **kwargs) 2025-09-07T07:04:40.9414646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9415086Z self_outputs = self.self( 2025-09-07T07:04:40.9415479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9415874Z return func(*args, **kwargs) 2025-09-07T07:04:40.9416291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9416728Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9416877Z 2025-09-07T07:04:40.9416973Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9417210Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9417466Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9417866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9418218Z return mod(**inputs) 2025-09-07T07:04:40.9418631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9419076Z outputs = self.electra( 2025-09-07T07:04:40.9419488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9420058Z hidden_states = self.encoder( 2025-09-07T07:04:40.9420480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9420908Z layer_outputs = layer_module( 2025-09-07T07:04:40.9421289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9421696Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9422158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9422611Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9423038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9423442Z return func(*args, **kwargs) 2025-09-07T07:04:40.9423868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9424452Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9424961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9425443Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9425681Z 2025-09-07T07:04:40.9425804Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9426214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9426578Z return mod(**inputs) 2025-09-07T07:04:40.9427017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9427441Z outputs = self.electra( 2025-09-07T07:04:40.9427864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9428304Z hidden_states = self.encoder( 2025-09-07T07:04:40.9428736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9429175Z layer_outputs = layer_module( 2025-09-07T07:04:40.9429559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9429969Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9430436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9430926Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9431371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9431811Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9432289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9432825Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9433316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9433759Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9433921Z 2025-09-07T07:04:40.9434039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9434442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9434814Z return mod(**inputs) 2025-09-07T07:04:40.9435258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9435702Z outputs = self.electra( 2025-09-07T07:04:40.9436121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9436562Z hidden_states = self.encoder( 2025-09-07T07:04:40.9436978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9437358Z layer_outputs = layer_module( 2025-09-07T07:04:40.9437703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9438064Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9438501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9438947Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9439374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9439797Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9440253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9440743Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9441202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9441638Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9442031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9442392Z return self.act(input) 2025-09-07T07:04:40.9442512Z 2025-09-07T07:04:40.9442634Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9443031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9443369Z return mod(**inputs) 2025-09-07T07:04:40.9443750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9444150Z outputs = self.electra( 2025-09-07T07:04:40.9444537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9444944Z hidden_states = self.encoder( 2025-09-07T07:04:40.9445321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9445716Z layer_outputs = layer_module( 2025-09-07T07:04:40.9446076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9446440Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9446840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9447235Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9447634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9448024Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9448451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9448932Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9449370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9449798Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9449941Z 2025-09-07T07:04:40.9450044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9450406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9450729Z return mod(**inputs) 2025-09-07T07:04:40.9451099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9451499Z outputs = self.electra( 2025-09-07T07:04:40.9451887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9452287Z hidden_states = self.encoder( 2025-09-07T07:04:40.9452672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9453068Z layer_outputs = layer_module( 2025-09-07T07:04:40.9453420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9453789Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9454195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9454621Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9455033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9455456Z return func(*args, **kwargs) 2025-09-07T07:04:40.9455876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9456293Z self_outputs = self.self( 2025-09-07T07:04:40.9456564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9456643Z return func(*args, **kwargs) 2025-09-07T07:04:40.9456938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9457021Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9457025Z 2025-09-07T07:04:40.9457141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9457348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9457417Z return mod(**inputs) 2025-09-07T07:04:40.9457699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9457770Z outputs = self.electra( 2025-09-07T07:04:40.9458042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9458131Z hidden_states = self.encoder( 2025-09-07T07:04:40.9458402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9458480Z layer_outputs = layer_module( 2025-09-07T07:04:40.9458708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9458799Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9459069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9459160Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9459411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9459482Z return func(*args, **kwargs) 2025-09-07T07:04:40.9459761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9459850Z self_outputs = self.self( 2025-09-07T07:04:40.9460102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9460175Z return func(*args, **kwargs) 2025-09-07T07:04:40.9460441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9460531Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9460534Z 2025-09-07T07:04:40.9460641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9460853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9460920Z return mod(**inputs) 2025-09-07T07:04:40.9461191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9461269Z outputs = self.electra( 2025-09-07T07:04:40.9461536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9461613Z hidden_states = self.encoder( 2025-09-07T07:04:40.9461897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9462005Z layer_outputs = layer_module( 2025-09-07T07:04:40.9462248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9462348Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9462639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9462727Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9462998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9463075Z return func(*args, **kwargs) 2025-09-07T07:04:40.9463354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9463436Z self_outputs = self.self( 2025-09-07T07:04:40.9463695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9463776Z return func(*args, **kwargs) 2025-09-07T07:04:40.9464059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9464152Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9464162Z 2025-09-07T07:04:40.9464245Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9464328Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9464486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9464691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9464766Z return mod(**inputs) 2025-09-07T07:04:40.9465036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9465105Z outputs = self.electra( 2025-09-07T07:04:40.9465382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9465458Z hidden_states = self.encoder( 2025-09-07T07:04:40.9465845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9465928Z layer_outputs = layer_module( 2025-09-07T07:04:40.9466172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9466271Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9466599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9466697Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9466957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9467031Z return func(*args, **kwargs) 2025-09-07T07:04:40.9467316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9467456Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9467742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9467833Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9467839Z 2025-09-07T07:04:40.9467959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9468176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9468243Z return mod(**inputs) 2025-09-07T07:04:40.9468518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9468603Z outputs = self.electra( 2025-09-07T07:04:40.9468877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9468964Z hidden_states = self.encoder( 2025-09-07T07:04:40.9469230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9469311Z layer_outputs = layer_module( 2025-09-07T07:04:40.9469533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9469622Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9469888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9469982Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9470244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9470324Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9470631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9470754Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9471035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9471133Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9471138Z 2025-09-07T07:04:40.9471240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9471443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9471508Z return mod(**inputs) 2025-09-07T07:04:40.9471797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9471870Z outputs = self.electra( 2025-09-07T07:04:40.9472161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9472237Z hidden_states = self.encoder( 2025-09-07T07:04:40.9472522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9472609Z layer_outputs = layer_module( 2025-09-07T07:04:40.9472854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9472968Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9473255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9473346Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9473639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9473723Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9474058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9474191Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9474487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9474611Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9474826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9474907Z return self.act(input) 2025-09-07T07:04:40.9474910Z 2025-09-07T07:04:40.9475035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9475247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9475314Z return mod(**inputs) 2025-09-07T07:04:40.9475602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9475686Z outputs = self.electra( 2025-09-07T07:04:40.9475970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9476056Z hidden_states = self.encoder( 2025-09-07T07:04:40.9476341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9476415Z layer_outputs = layer_module( 2025-09-07T07:04:40.9476663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9476748Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9477034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9477120Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9477396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9477471Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9477780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9477923Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9478185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9478272Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9478277Z 2025-09-07T07:04:40.9478379Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9478578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9478654Z return mod(**inputs) 2025-09-07T07:04:40.9478928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9479006Z outputs = self.electra( 2025-09-07T07:04:40.9479271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9479366Z hidden_states = self.encoder( 2025-09-07T07:04:40.9479633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9479704Z layer_outputs = layer_module( 2025-09-07T07:04:40.9479950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9480030Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9480300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9480383Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9480632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9480715Z return func(*args, **kwargs) 2025-09-07T07:04:40.9480982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9481064Z self_outputs = self.self( 2025-09-07T07:04:40.9481309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9481380Z return func(*args, **kwargs) 2025-09-07T07:04:40.9481671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9481757Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9481781Z 2025-09-07T07:04:40.9481895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9482093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9482165Z return mod(**inputs) 2025-09-07T07:04:40.9482435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9482505Z outputs = self.electra( 2025-09-07T07:04:40.9482778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9482849Z hidden_states = self.encoder( 2025-09-07T07:04:40.9483118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9483190Z layer_outputs = layer_module( 2025-09-07T07:04:40.9483415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9483508Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9483763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9483871Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9484109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9484180Z return func(*args, **kwargs) 2025-09-07T07:04:40.9484442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9484510Z self_outputs = self.self( 2025-09-07T07:04:40.9484755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9484824Z return func(*args, **kwargs) 2025-09-07T07:04:40.9485089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9485166Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9485169Z 2025-09-07T07:04:40.9485271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9485479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9485574Z return mod(**inputs) 2025-09-07T07:04:40.9485842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9485911Z outputs = self.electra( 2025-09-07T07:04:40.9486169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9486248Z hidden_states = self.encoder( 2025-09-07T07:04:40.9486505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9486582Z layer_outputs = layer_module( 2025-09-07T07:04:40.9486798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9486885Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9487146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9487230Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9487480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9487565Z return func(*args, **kwargs) 2025-09-07T07:04:40.9487839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9487911Z self_outputs = self.self( 2025-09-07T07:04:40.9488176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9488255Z return func(*args, **kwargs) 2025-09-07T07:04:40.9488525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9488615Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9488621Z 2025-09-07T07:04:40.9488704Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9488785Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9488897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9489101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9489174Z return mod(**inputs) 2025-09-07T07:04:40.9489441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9489520Z outputs = self.electra( 2025-09-07T07:04:40.9489785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9489856Z hidden_states = self.encoder( 2025-09-07T07:04:40.9490154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9490227Z layer_outputs = layer_module( 2025-09-07T07:04:40.9490456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9490535Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9490803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9490893Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9491137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9491213Z return func(*args, **kwargs) 2025-09-07T07:04:40.9491477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9491611Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9491896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9491980Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9491984Z 2025-09-07T07:04:40.9492093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9492296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9492371Z return mod(**inputs) 2025-09-07T07:04:40.9492643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9492713Z outputs = self.electra( 2025-09-07T07:04:40.9492989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9493063Z hidden_states = self.encoder( 2025-09-07T07:04:40.9493333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9493405Z layer_outputs = layer_module( 2025-09-07T07:04:40.9493628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9493715Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9493995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9494090Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9494375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9494463Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9494772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9494897Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9495173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9495255Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9495258Z 2025-09-07T07:04:40.9495370Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9495572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9495638Z return mod(**inputs) 2025-09-07T07:04:40.9495914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9495984Z outputs = self.electra( 2025-09-07T07:04:40.9496251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9496343Z hidden_states = self.encoder( 2025-09-07T07:04:40.9496621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9496692Z layer_outputs = layer_module( 2025-09-07T07:04:40.9496922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9497011Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9497283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9497376Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9497644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9497721Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9498037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9498175Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9498452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9498569Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9498789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9498872Z return self.act(input) 2025-09-07T07:04:40.9498876Z 2025-09-07T07:04:40.9498987Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9499206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9499288Z return mod(**inputs) 2025-09-07T07:04:40.9499581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9499662Z outputs = self.electra( 2025-09-07T07:04:40.9499942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9500025Z hidden_states = self.encoder( 2025-09-07T07:04:40.9500322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9500398Z layer_outputs = layer_module( 2025-09-07T07:04:40.9501816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9501918Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9502215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9502313Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9502600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9502700Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9503039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9503200Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9503503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9503605Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9503611Z 2025-09-07T07:04:40.9503728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9503958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9504059Z return mod(**inputs) 2025-09-07T07:04:40.9504340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9504420Z outputs = self.electra( 2025-09-07T07:04:40.9504700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9504774Z hidden_states = self.encoder( 2025-09-07T07:04:40.9505062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9505136Z layer_outputs = layer_module( 2025-09-07T07:04:40.9505380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9505463Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9505841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9505942Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9506238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9506325Z return func(*args, **kwargs) 2025-09-07T07:04:40.9506628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9506713Z self_outputs = self.self( 2025-09-07T07:04:40.9506975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9507051Z return func(*args, **kwargs) 2025-09-07T07:04:40.9507343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9507430Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9507436Z 2025-09-07T07:04:40.9507555Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9507773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9507851Z return mod(**inputs) 2025-09-07T07:04:40.9508139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9508232Z outputs = self.electra( 2025-09-07T07:04:40.9508521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9508598Z hidden_states = self.encoder( 2025-09-07T07:04:40.9508905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9508980Z layer_outputs = layer_module( 2025-09-07T07:04:40.9509218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9509312Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9509592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9509685Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9509947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9510021Z return func(*args, **kwargs) 2025-09-07T07:04:40.9510313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9510387Z self_outputs = self.self( 2025-09-07T07:04:40.9510652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9510725Z return func(*args, **kwargs) 2025-09-07T07:04:40.9511036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9511125Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9511128Z 2025-09-07T07:04:40.9511239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9511459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9511532Z return mod(**inputs) 2025-09-07T07:04:40.9511820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9511892Z outputs = self.electra( 2025-09-07T07:04:40.9512170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9512253Z hidden_states = self.encoder( 2025-09-07T07:04:40.9512529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9512614Z layer_outputs = layer_module( 2025-09-07T07:04:40.9512871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9512955Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9513245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9513330Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9513596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9513670Z return func(*args, **kwargs) 2025-09-07T07:04:40.9513958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9514033Z self_outputs = self.self( 2025-09-07T07:04:40.9514292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9514375Z return func(*args, **kwargs) 2025-09-07T07:04:40.9514657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9514751Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9514755Z 2025-09-07T07:04:40.9514859Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9514947Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9515067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9515303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9515385Z return mod(**inputs) 2025-09-07T07:04:40.9515670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9515745Z outputs = self.electra( 2025-09-07T07:04:40.9516033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9516109Z hidden_states = self.encoder( 2025-09-07T07:04:40.9516397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9516472Z layer_outputs = layer_module( 2025-09-07T07:04:40.9516718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9516801Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9517087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9517182Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9517438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9517536Z return func(*args, **kwargs) 2025-09-07T07:04:40.9517826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9517965Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9518260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9518349Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9518353Z 2025-09-07T07:04:40.9518470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9518687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9518767Z return mod(**inputs) 2025-09-07T07:04:40.9519057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9519130Z outputs = self.electra( 2025-09-07T07:04:40.9519434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9519509Z hidden_states = self.encoder( 2025-09-07T07:04:40.9519912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9519994Z layer_outputs = layer_module( 2025-09-07T07:04:40.9520233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9520330Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9520612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9520710Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9520988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9521074Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9521397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9521529Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9521867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9521955Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9521985Z 2025-09-07T07:04:40.9522104Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9522324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9522395Z return mod(**inputs) 2025-09-07T07:04:40.9522695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9522770Z outputs = self.electra( 2025-09-07T07:04:40.9523063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9523139Z hidden_states = self.encoder( 2025-09-07T07:04:40.9523428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9523511Z layer_outputs = layer_module( 2025-09-07T07:04:40.9523754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9523848Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9524140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9524284Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9524547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9524625Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9524931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9525052Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9525327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9525444Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9525663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9525740Z return self.act(input) 2025-09-07T07:04:40.9525745Z 2025-09-07T07:04:40.9525851Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9526104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9526176Z return mod(**inputs) 2025-09-07T07:04:40.9526474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9526548Z outputs = self.electra( 2025-09-07T07:04:40.9526840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9526925Z hidden_states = self.encoder( 2025-09-07T07:04:40.9527214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9527300Z layer_outputs = layer_module( 2025-09-07T07:04:40.9527548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9527635Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9527925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9528023Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9528304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9528383Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9528696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9528841Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9529109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9529203Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9529207Z 2025-09-07T07:04:40.9529316Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9529544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9529615Z return mod(**inputs) 2025-09-07T07:04:40.9529911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9529994Z outputs = self.electra( 2025-09-07T07:04:40.9530291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9530379Z hidden_states = self.encoder( 2025-09-07T07:04:40.9530681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9530758Z layer_outputs = layer_module( 2025-09-07T07:04:40.9531032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9531120Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9531416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9531505Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9531786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9531863Z return func(*args, **kwargs) 2025-09-07T07:04:40.9532145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9532223Z self_outputs = self.self( 2025-09-07T07:04:40.9532467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9532548Z return func(*args, **kwargs) 2025-09-07T07:04:40.9532812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9532911Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9532914Z 2025-09-07T07:04:40.9533028Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9533231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9533305Z return mod(**inputs) 2025-09-07T07:04:40.9533573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9533642Z outputs = self.electra( 2025-09-07T07:04:40.9533912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9533980Z hidden_states = self.encoder( 2025-09-07T07:04:40.9534252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9534323Z layer_outputs = layer_module( 2025-09-07T07:04:40.9534551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9534629Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9534915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9535007Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9535277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9535358Z return func(*args, **kwargs) 2025-09-07T07:04:40.9535628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9535700Z self_outputs = self.self( 2025-09-07T07:04:40.9535956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9536027Z return func(*args, **kwargs) 2025-09-07T07:04:40.9536303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9536384Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9536387Z 2025-09-07T07:04:40.9536500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9536703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9536769Z return mod(**inputs) 2025-09-07T07:04:40.9537045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9537114Z outputs = self.electra( 2025-09-07T07:04:40.9537403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9537475Z hidden_states = self.encoder( 2025-09-07T07:04:40.9537741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9537821Z layer_outputs = layer_module( 2025-09-07T07:04:40.9538043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9538129Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9538394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9538475Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9538726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9538798Z return func(*args, **kwargs) 2025-09-07T07:04:40.9539085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9539155Z self_outputs = self.self( 2025-09-07T07:04:40.9539406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9539477Z return func(*args, **kwargs) 2025-09-07T07:04:40.9539745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9539834Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9539838Z 2025-09-07T07:04:40.9539920Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9540010Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9540116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9540324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9540400Z return mod(**inputs) 2025-09-07T07:04:40.9540671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9540744Z outputs = self.electra( 2025-09-07T07:04:40.9541028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9541100Z hidden_states = self.encoder( 2025-09-07T07:04:40.9541394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9541464Z layer_outputs = layer_module( 2025-09-07T07:04:40.9541696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9541775Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9542047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9542130Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9542378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9542457Z return func(*args, **kwargs) 2025-09-07T07:04:40.9542774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9542925Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9543215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9543306Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9543310Z 2025-09-07T07:04:40.9543430Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9543669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9543749Z return mod(**inputs) 2025-09-07T07:04:40.9544042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9544117Z outputs = self.electra( 2025-09-07T07:04:40.9544416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9544494Z hidden_states = self.encoder( 2025-09-07T07:04:40.9544794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9544881Z layer_outputs = layer_module( 2025-09-07T07:04:40.9545126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9545212Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9545498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9545697Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9545986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9546085Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9546412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9546548Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9546855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9546945Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9546951Z 2025-09-07T07:04:40.9547073Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9547302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9547384Z return mod(**inputs) 2025-09-07T07:04:40.9547677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9547778Z outputs = self.electra( 2025-09-07T07:04:40.9548077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9548176Z hidden_states = self.encoder( 2025-09-07T07:04:40.9548474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9548550Z layer_outputs = layer_module( 2025-09-07T07:04:40.9548791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9548886Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9549177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9549275Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9549564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9549656Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9549983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9550114Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9550412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9550552Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9550795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9550875Z return self.act(input) 2025-09-07T07:04:40.9550879Z 2025-09-07T07:04:40.9550991Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9551223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9551295Z return mod(**inputs) 2025-09-07T07:04:40.9551599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9551675Z outputs = self.electra( 2025-09-07T07:04:40.9551978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9552057Z hidden_states = self.encoder( 2025-09-07T07:04:40.9552351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9552457Z layer_outputs = layer_module( 2025-09-07T07:04:40.9552699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9552794Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9553087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9553179Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9553475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9553559Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9553891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9554051Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9554308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9554394Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9554397Z 2025-09-07T07:04:40.9554498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9554722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9554788Z return mod(**inputs) 2025-09-07T07:04:40.9555069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9555137Z outputs = self.electra( 2025-09-07T07:04:40.9555395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9555471Z hidden_states = self.encoder( 2025-09-07T07:04:40.9555728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9555803Z layer_outputs = layer_module( 2025-09-07T07:04:40.9556021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9556099Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9556363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9556443Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9556687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9556756Z return func(*args, **kwargs) 2025-09-07T07:04:40.9557019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9557109Z self_outputs = self.self( 2025-09-07T07:04:40.9557348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9557422Z return func(*args, **kwargs) 2025-09-07T07:04:40.9557679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:04:40.9557768Z query_layer = self.query(hidden_states) 2025-09-07T07:04:40.9557772Z 2025-09-07T07:04:40.9557885Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9558107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9558185Z return mod(**inputs) 2025-09-07T07:04:40.9558483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9558562Z outputs = self.electra( 2025-09-07T07:04:40.9558856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9558930Z hidden_states = self.encoder( 2025-09-07T07:04:40.9559215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9559291Z layer_outputs = layer_module( 2025-09-07T07:04:40.9559533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9559616Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9559906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9559988Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9560235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9560318Z return func(*args, **kwargs) 2025-09-07T07:04:40.9560588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9560664Z self_outputs = self.self( 2025-09-07T07:04:40.9560933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9561004Z return func(*args, **kwargs) 2025-09-07T07:04:40.9561295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:04:40.9561377Z key_layer = self.key(current_states) 2025-09-07T07:04:40.9561380Z 2025-09-07T07:04:40.9561490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9561701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9561775Z return mod(**inputs) 2025-09-07T07:04:40.9562037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9562104Z outputs = self.electra( 2025-09-07T07:04:40.9562370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9562440Z hidden_states = self.encoder( 2025-09-07T07:04:40.9562705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9562775Z layer_outputs = layer_module( 2025-09-07T07:04:40.9562990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9563073Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9563331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9563441Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9563682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9563753Z return func(*args, **kwargs) 2025-09-07T07:04:40.9564022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:04:40.9564102Z self_outputs = self.self( 2025-09-07T07:04:40.9564351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9564422Z return func(*args, **kwargs) 2025-09-07T07:04:40.9564694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:04:40.9564778Z value_layer = self.value(current_states) 2025-09-07T07:04:40.9564782Z 2025-09-07T07:04:40.9564866Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9564969Z cudagraph partition due to non gpu ops 2025-09-07T07:04:40.9565072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9565273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9565337Z return mod(**inputs) 2025-09-07T07:04:40.9565602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9565679Z outputs = self.electra( 2025-09-07T07:04:40.9565945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9566023Z hidden_states = self.encoder( 2025-09-07T07:04:40.9566297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9566370Z layer_outputs = layer_module( 2025-09-07T07:04:40.9566596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9566673Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9566935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:04:40.9567031Z self_attention_outputs = self.attention( 2025-09-07T07:04:40.9567281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:04:40.9567412Z return func(*args, **kwargs) 2025-09-07T07:04:40.9567679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:04:40.9567819Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:04:40.9568092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:04:40.9568193Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9568197Z 2025-09-07T07:04:40.9568308Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9568522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9568603Z return mod(**inputs) 2025-09-07T07:04:40.9568891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9568973Z outputs = self.electra( 2025-09-07T07:04:40.9569256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9569339Z hidden_states = self.encoder( 2025-09-07T07:04:40.9569609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9569699Z layer_outputs = layer_module( 2025-09-07T07:04:40.9569931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9570011Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9570285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9570369Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9570632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9570719Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9571020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9571152Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9571420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:04:40.9571519Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9571529Z 2025-09-07T07:04:40.9571631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9571833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9571906Z return mod(**inputs) 2025-09-07T07:04:40.9572174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9572249Z outputs = self.electra( 2025-09-07T07:04:40.9572512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9572582Z hidden_states = self.encoder( 2025-09-07T07:04:40.9572854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9572925Z layer_outputs = layer_module( 2025-09-07T07:04:40.9573157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9573235Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9573517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9573611Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9573910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9574003Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9574331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:04:40.9574470Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:04:40.9574761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:04:40.9574887Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:04:40.9575142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:04:40.9575214Z return self.act(input) 2025-09-07T07:04:40.9575217Z 2025-09-07T07:04:40.9575329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9575535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9575602Z return mod(**inputs) 2025-09-07T07:04:40.9575878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-09-07T07:04:40.9575964Z outputs = self.electra( 2025-09-07T07:04:40.9576237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:04:40.9576310Z hidden_states = self.encoder( 2025-09-07T07:04:40.9576575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:04:40.9576655Z layer_outputs = layer_module( 2025-09-07T07:04:40.9576879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:04:40.9576968Z return super().__call__(*args, **kwargs) 2025-09-07T07:04:40.9577238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:04:40.9577330Z layer_output = apply_chunking_to_forward( 2025-09-07T07:04:40.9577596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:04:40.9577697Z return forward_fn(*input_tensors) 2025-09-07T07:04:40.9578019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:04:40.9578163Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:04:40.9578458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:04:40.9578545Z hidden_states = self.dense(hidden_states) 2025-09-07T07:04:40.9578549Z 2025-09-07T07:04:40.9578667Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9578887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9578954Z return mod(**inputs) 2025-09-07T07:04:40.9579231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-09-07T07:04:40.9579419Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-09-07T07:04:40.9579694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 640, in forward 2025-09-07T07:04:40.9579800Z hidden_states = self.dense(generator_hidden_states) 2025-09-07T07:04:40.9579818Z 2025-09-07T07:04:40.9579921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9580131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9580214Z return mod(**inputs) 2025-09-07T07:04:40.9580493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-09-07T07:04:40.9580673Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-09-07T07:04:40.9580678Z 2025-09-07T07:04:40.9580788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:04:40.9580989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:04:40.9581054Z return mod(**inputs) 2025-09-07T07:04:40.9581335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1564, in forward 2025-09-07T07:04:40.9581408Z lm_loss = self.loss_function( 2025-09-07T07:04:40.9581669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-09-07T07:04:40.9581852Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-09-07T07:04:40.9582116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-09-07T07:04:40.9582328Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-09-07T07:04:40.9582346Z 2025-09-07T07:04:51.8040884Z Compilation time (from dynamo_timed): 18.077729082 2025-09-07T07:04:51.8133310Z pass 2025-09-07T07:04:51.8133753Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:51.8134723Z TIMING: _recursive_pre_grad_passes:0.00783 _recursive_joint_graph_passes:0.48184 _recursive_post_grad_passes:0.07737 async_compile.wait:0.78598 code_gen:10.223 inductor_compile:11.50161 backend_compile:15.17113 gc:0.00179 entire_frame_compile:18.07773 total_wall_time:18.07773 2025-09-07T07:04:51.8135952Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:15035 | FakeTensor.__torch_dispatch__:4346 | ProxyTorchDispatchMode.__torch_dispatch__:5671 2025-09-07T07:04:51.8136514Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-09-07T07:04:54.3993473Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:04:54.3995401Z import pynvml # type: ignore[import] 2025-09-07T07:04:57.2350644Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:04:57.2351660Z from pkg_resources import resource_filename 2025-09-07T07:04:57.9008809Z 2025-09-07T07:04:58.0899654Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:04:58.0900249Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:04:58.0908111Z cpu eval ElectraForQuestionAnswering 2025-09-07T07:04:58.2284879Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:58.3078172Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:04:58.3657998Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:06.6283666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6286830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6287489Z return mod(**inputs) 2025-09-07T07:05:06.6288865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6289456Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6289971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-09-07T07:05:06.6290467Z hidden_states = self.embeddings_project(hidden_states) 2025-09-07T07:05:06.6290701Z 2025-09-07T07:05:06.6290833Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6291260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6291606Z return mod(**inputs) 2025-09-07T07:05:06.6292029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6292489Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6292909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6293321Z hidden_states = self.encoder( 2025-09-07T07:05:06.6293730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6294167Z layer_outputs = layer_module( 2025-09-07T07:05:06.6294596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6295003Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6295464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6295884Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6296289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6296676Z return func(*args, **kwargs) 2025-09-07T07:05:06.6297106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6297551Z self_outputs = self.self( 2025-09-07T07:05:06.6297953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6298366Z return func(*args, **kwargs) 2025-09-07T07:05:06.6298780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6299308Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6299475Z 2025-09-07T07:05:06.6299598Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6300000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6300363Z return mod(**inputs) 2025-09-07T07:05:06.6300773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6301243Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6301703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6302142Z hidden_states = self.encoder( 2025-09-07T07:05:06.6302572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6303020Z layer_outputs = layer_module( 2025-09-07T07:05:06.6303434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6303837Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6304332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6304790Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6305258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6305898Z return func(*args, **kwargs) 2025-09-07T07:05:06.6306346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6306787Z self_outputs = self.self( 2025-09-07T07:05:06.6307201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6307610Z return func(*args, **kwargs) 2025-09-07T07:05:06.6308036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6308515Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6308666Z 2025-09-07T07:05:06.6308792Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6309198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6309566Z return mod(**inputs) 2025-09-07T07:05:06.6310272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6310764Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6311229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6311674Z hidden_states = self.encoder( 2025-09-07T07:05:06.6312095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6312529Z layer_outputs = layer_module( 2025-09-07T07:05:06.6312918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6313327Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6313766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6314212Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6314635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6315108Z return func(*args, **kwargs) 2025-09-07T07:05:06.6315570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6316006Z self_outputs = self.self( 2025-09-07T07:05:06.6316412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6316823Z return func(*args, **kwargs) 2025-09-07T07:05:06.6317251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6317683Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6317822Z 2025-09-07T07:05:06.6317908Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6318128Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6318383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6318774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6319098Z return mod(**inputs) 2025-09-07T07:05:06.6319483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6320279Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6320810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6321215Z hidden_states = self.encoder( 2025-09-07T07:05:06.6321634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6322041Z layer_outputs = layer_module( 2025-09-07T07:05:06.6322397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6322776Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6323176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6323596Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6323989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6324375Z return func(*args, **kwargs) 2025-09-07T07:05:06.6324768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6325228Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6325689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6326107Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6326279Z 2025-09-07T07:05:06.6326395Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6326772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6327103Z return mod(**inputs) 2025-09-07T07:05:06.6327494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6327924Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6328345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6328747Z hidden_states = self.encoder( 2025-09-07T07:05:06.6329148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6329548Z layer_outputs = layer_module( 2025-09-07T07:05:06.6329902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6330277Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6330709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6331126Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6331546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6331950Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6332391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6332876Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6333338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6333740Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6333876Z 2025-09-07T07:05:06.6333988Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6334349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6334667Z return mod(**inputs) 2025-09-07T07:05:06.6335063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6335485Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6335927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6336391Z hidden_states = self.encoder( 2025-09-07T07:05:06.6336778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6337167Z layer_outputs = layer_module( 2025-09-07T07:05:06.6337517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6337898Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6338282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6338686Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6339083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6339472Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6339908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6340389Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6340836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6341298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6341690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6342031Z return self.act(input) 2025-09-07T07:05:06.6342154Z 2025-09-07T07:05:06.6342264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6342635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6342978Z return mod(**inputs) 2025-09-07T07:05:06.6343367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6343817Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6344277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6344707Z hidden_states = self.encoder( 2025-09-07T07:05:06.6345148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6345583Z layer_outputs = layer_module( 2025-09-07T07:05:06.6346038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6346443Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6346887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6347316Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6347712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6348106Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6348539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6349091Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6349542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6349937Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6350123Z 2025-09-07T07:05:06.6350228Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6350606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6350934Z return mod(**inputs) 2025-09-07T07:05:06.6351308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6351711Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6352121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6352511Z hidden_states = self.encoder( 2025-09-07T07:05:06.6352893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6353285Z layer_outputs = layer_module( 2025-09-07T07:05:06.6353615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6353975Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6354383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6354771Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6355134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6355523Z return func(*args, **kwargs) 2025-09-07T07:05:06.6355908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6356305Z self_outputs = self.self( 2025-09-07T07:05:06.6356667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6357040Z return func(*args, **kwargs) 2025-09-07T07:05:06.6357426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6357835Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6357982Z 2025-09-07T07:05:06.6358090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6358440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6358762Z return mod(**inputs) 2025-09-07T07:05:06.6359139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6359573Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6359987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6360379Z hidden_states = self.encoder( 2025-09-07T07:05:06.6360761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6361149Z layer_outputs = layer_module( 2025-09-07T07:05:06.6361497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6361861Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6362250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6362654Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6363038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6363402Z return func(*args, **kwargs) 2025-09-07T07:05:06.6363767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6364162Z self_outputs = self.self( 2025-09-07T07:05:06.6364522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6364948Z return func(*args, **kwargs) 2025-09-07T07:05:06.6365320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6365707Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6365847Z 2025-09-07T07:05:06.6365949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6366309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6366635Z return mod(**inputs) 2025-09-07T07:05:06.6367007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6367416Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6367829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6368223Z hidden_states = self.encoder( 2025-09-07T07:05:06.6368602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6368980Z layer_outputs = layer_module( 2025-09-07T07:05:06.6369319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6369690Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6370076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6370469Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6370835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6371205Z return func(*args, **kwargs) 2025-09-07T07:05:06.6371585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6371980Z self_outputs = self.self( 2025-09-07T07:05:06.6372336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6372710Z return func(*args, **kwargs) 2025-09-07T07:05:06.6373081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6373503Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6373634Z 2025-09-07T07:05:06.6373722Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6373937Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6374177Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6374548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6374872Z return mod(**inputs) 2025-09-07T07:05:06.6375248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6375650Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6376059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6376452Z hidden_states = self.encoder( 2025-09-07T07:05:06.6376835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6377224Z layer_outputs = layer_module( 2025-09-07T07:05:06.6377574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6377964Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6378366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6378782Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6379155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6379538Z return func(*args, **kwargs) 2025-09-07T07:05:06.6379915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6380363Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6380808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6381204Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6381349Z 2025-09-07T07:05:06.6381454Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6381827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6382161Z return mod(**inputs) 2025-09-07T07:05:06.6382542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6382974Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6383422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6383858Z hidden_states = self.encoder( 2025-09-07T07:05:06.6384252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6384649Z layer_outputs = layer_module( 2025-09-07T07:05:06.6385004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6385378Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6385914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6386381Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6386821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6387260Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6387732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6388281Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6388728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6389149Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6389299Z 2025-09-07T07:05:06.6389404Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6389776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6390109Z return mod(**inputs) 2025-09-07T07:05:06.6390483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6390903Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6391336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6391774Z hidden_states = self.encoder( 2025-09-07T07:05:06.6392172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6392570Z layer_outputs = layer_module( 2025-09-07T07:05:06.6392945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6393314Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6393734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6394156Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6394557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6394964Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6395406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6395893Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6396343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6396794Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6397183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6397530Z return self.act(input) 2025-09-07T07:05:06.6397643Z 2025-09-07T07:05:06.6397756Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6398118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6398468Z return mod(**inputs) 2025-09-07T07:05:06.6398854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6399275Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6399690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6400086Z hidden_states = self.encoder( 2025-09-07T07:05:06.6400482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6400886Z layer_outputs = layer_module( 2025-09-07T07:05:06.6401238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6401605Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6402015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6402465Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6402881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6403287Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6403719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6404218Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6404682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6405117Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6405253Z 2025-09-07T07:05:06.6405367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6405737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6406073Z return mod(**inputs) 2025-09-07T07:05:06.6406458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6406883Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6407319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6407721Z hidden_states = self.encoder( 2025-09-07T07:05:06.6408131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6408536Z layer_outputs = layer_module( 2025-09-07T07:05:06.6408893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6409261Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6409679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6410093Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6410489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6410883Z return func(*args, **kwargs) 2025-09-07T07:05:06.6411263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6411670Z self_outputs = self.self( 2025-09-07T07:05:06.6412041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6412517Z return func(*args, **kwargs) 2025-09-07T07:05:06.6412950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6413402Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6413560Z 2025-09-07T07:05:06.6413666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6414031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6414361Z return mod(**inputs) 2025-09-07T07:05:06.6414733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6415148Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6415561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6415956Z hidden_states = self.encoder( 2025-09-07T07:05:06.6416348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6416763Z layer_outputs = layer_module( 2025-09-07T07:05:06.6417141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6417516Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6417914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6418312Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6418696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6419069Z return func(*args, **kwargs) 2025-09-07T07:05:06.6419449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6419966Z self_outputs = self.self( 2025-09-07T07:05:06.6420330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6420709Z return func(*args, **kwargs) 2025-09-07T07:05:06.6421100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6421514Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6421650Z 2025-09-07T07:05:06.6421809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6422182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6422525Z return mod(**inputs) 2025-09-07T07:05:06.6422941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6423368Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6423780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6424188Z hidden_states = self.encoder( 2025-09-07T07:05:06.6424586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6424995Z layer_outputs = layer_module( 2025-09-07T07:05:06.6425371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6425812Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6426258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6426704Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6427172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6427565Z return func(*args, **kwargs) 2025-09-07T07:05:06.6427977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6428383Z self_outputs = self.self( 2025-09-07T07:05:06.6428752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6429134Z return func(*args, **kwargs) 2025-09-07T07:05:06.6429516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6429925Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6430073Z 2025-09-07T07:05:06.6430159Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6430381Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6430625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6430991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6431328Z return mod(**inputs) 2025-09-07T07:05:06.6431715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6432172Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6432588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6432992Z hidden_states = self.encoder( 2025-09-07T07:05:06.6433384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6433788Z layer_outputs = layer_module( 2025-09-07T07:05:06.6434151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6434510Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6434919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6435334Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6435718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6436098Z return func(*args, **kwargs) 2025-09-07T07:05:06.6436500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6436961Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6437434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6437850Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6438001Z 2025-09-07T07:05:06.6438106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6438468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6438794Z return mod(**inputs) 2025-09-07T07:05:06.6439168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6439577Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6439977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6440369Z hidden_states = self.encoder( 2025-09-07T07:05:06.6440753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6441145Z layer_outputs = layer_module( 2025-09-07T07:05:06.6441487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6441839Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6442251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6442652Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6443046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6443427Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6443855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6444338Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6444784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6445210Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6445350Z 2025-09-07T07:05:06.6445456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6445816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6446161Z return mod(**inputs) 2025-09-07T07:05:06.6446535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6446946Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6447345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6447732Z hidden_states = self.encoder( 2025-09-07T07:05:06.6448121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6448511Z layer_outputs = layer_module( 2025-09-07T07:05:06.6448847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6449211Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6449614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6450020Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6450436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6450821Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6451305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6451802Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6452253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6452708Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6453106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6453448Z return self.act(input) 2025-09-07T07:05:06.6453565Z 2025-09-07T07:05:06.6453669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6454032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6454356Z return mod(**inputs) 2025-09-07T07:05:06.6454720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6455137Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6455559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6455951Z hidden_states = self.encoder( 2025-09-07T07:05:06.6456365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6456771Z layer_outputs = layer_module( 2025-09-07T07:05:06.6457122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6457490Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6457892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6458299Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6458702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6459101Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6459528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6460024Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6460487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6460895Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6461039Z 2025-09-07T07:05:06.6461144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6461513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6461855Z return mod(**inputs) 2025-09-07T07:05:06.6462262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6462713Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6463154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6463594Z hidden_states = self.encoder( 2025-09-07T07:05:06.6463983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6464404Z layer_outputs = layer_module( 2025-09-07T07:05:06.6464781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6465195Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6465699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6466178Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6466599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6467007Z return func(*args, **kwargs) 2025-09-07T07:05:06.6467422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6467843Z self_outputs = self.self( 2025-09-07T07:05:06.6468246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6468657Z return func(*args, **kwargs) 2025-09-07T07:05:06.6469076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6469512Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6469663Z 2025-09-07T07:05:06.6469778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6470173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6470528Z return mod(**inputs) 2025-09-07T07:05:06.6470942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6471425Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6471926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6472350Z hidden_states = self.encoder( 2025-09-07T07:05:06.6472768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6473199Z layer_outputs = layer_module( 2025-09-07T07:05:06.6473572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6473970Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6474405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6474847Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6475282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6475714Z return func(*args, **kwargs) 2025-09-07T07:05:06.6476128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6476554Z self_outputs = self.self( 2025-09-07T07:05:06.6476950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6477357Z return func(*args, **kwargs) 2025-09-07T07:05:06.6477760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6478188Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6478335Z 2025-09-07T07:05:06.6478446Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6478836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6479176Z return mod(**inputs) 2025-09-07T07:05:06.6479582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6480001Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6480440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6480841Z hidden_states = self.encoder( 2025-09-07T07:05:06.6481243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6481649Z layer_outputs = layer_module( 2025-09-07T07:05:06.6482001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6482376Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6482805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6483243Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6483637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6484016Z return func(*args, **kwargs) 2025-09-07T07:05:06.6484407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6484805Z self_outputs = self.self( 2025-09-07T07:05:06.6485179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6485561Z return func(*args, **kwargs) 2025-09-07T07:05:06.6485952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6486395Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6486538Z 2025-09-07T07:05:06.6486625Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6486853Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6487102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6487482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6487821Z return mod(**inputs) 2025-09-07T07:05:06.6488215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6488652Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6489082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6489497Z hidden_states = self.encoder( 2025-09-07T07:05:06.6489891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6490321Z layer_outputs = layer_module( 2025-09-07T07:05:06.6490679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6491050Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6491454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6491866Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6492258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6492646Z return func(*args, **kwargs) 2025-09-07T07:05:06.6493035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6493492Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6493950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6494371Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6494514Z 2025-09-07T07:05:06.6494629Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6495022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6495352Z return mod(**inputs) 2025-09-07T07:05:06.6495771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6496195Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6496606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6497009Z hidden_states = self.encoder( 2025-09-07T07:05:06.6497386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6497775Z layer_outputs = layer_module( 2025-09-07T07:05:06.6498119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6498479Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6498869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6499278Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6499678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6500075Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6500500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6500985Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6501429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6501841Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6501983Z 2025-09-07T07:05:06.6502100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6502491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6502837Z return mod(**inputs) 2025-09-07T07:05:06.6503252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6503697Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6504138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6504561Z hidden_states = self.encoder( 2025-09-07T07:05:06.6504995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6505415Z layer_outputs = layer_module( 2025-09-07T07:05:06.6505877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6506291Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6506727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6507151Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6507565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6507963Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6508388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6508854Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6509295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6509754Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6510144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6510500Z return self.act(input) 2025-09-07T07:05:06.6510639Z 2025-09-07T07:05:06.6510744Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6511108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6511432Z return mod(**inputs) 2025-09-07T07:05:06.6511816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6512232Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6512650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6513051Z hidden_states = self.encoder( 2025-09-07T07:05:06.6513451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6513848Z layer_outputs = layer_module( 2025-09-07T07:05:06.6514199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6514560Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6514955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6515386Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6515790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6516192Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6516622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6517119Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6517578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6517983Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6518130Z 2025-09-07T07:05:06.6518236Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6518601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6518937Z return mod(**inputs) 2025-09-07T07:05:06.6519320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6519929Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6520361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6520782Z hidden_states = self.encoder( 2025-09-07T07:05:06.6521189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6521596Z layer_outputs = layer_module( 2025-09-07T07:05:06.6521946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6522322Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6522736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6523155Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6523546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6523932Z return func(*args, **kwargs) 2025-09-07T07:05:06.6524370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6524784Z self_outputs = self.self( 2025-09-07T07:05:06.6525179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6525557Z return func(*args, **kwargs) 2025-09-07T07:05:06.6525948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6526363Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6526506Z 2025-09-07T07:05:06.6526622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6526995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6527319Z return mod(**inputs) 2025-09-07T07:05:06.6527706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6528126Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6528544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6528942Z hidden_states = self.encoder( 2025-09-07T07:05:06.6529336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6529735Z layer_outputs = layer_module( 2025-09-07T07:05:06.6530127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6530512Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6530920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6531347Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6531784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6532211Z return func(*args, **kwargs) 2025-09-07T07:05:06.6532653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6533095Z self_outputs = self.self( 2025-09-07T07:05:06.6533504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6533913Z return func(*args, **kwargs) 2025-09-07T07:05:06.6534333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6534796Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6534935Z 2025-09-07T07:05:06.6535038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6535397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6535718Z return mod(**inputs) 2025-09-07T07:05:06.6536093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6536496Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6536918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6537319Z hidden_states = self.encoder( 2025-09-07T07:05:06.6537713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6538114Z layer_outputs = layer_module( 2025-09-07T07:05:06.6538466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6538828Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6539241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6539654Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6540051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6540421Z return func(*args, **kwargs) 2025-09-07T07:05:06.6540831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6541271Z self_outputs = self.self( 2025-09-07T07:05:06.6541678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6542084Z return func(*args, **kwargs) 2025-09-07T07:05:06.6542506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6542955Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6543106Z 2025-09-07T07:05:06.6543205Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6543436Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6543704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6544108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6544471Z return mod(**inputs) 2025-09-07T07:05:06.6544892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6545377Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6545906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6546352Z hidden_states = self.encoder( 2025-09-07T07:05:06.6546783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6547188Z layer_outputs = layer_module( 2025-09-07T07:05:06.6547581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6547992Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6548438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6548904Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6549326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6549771Z return func(*args, **kwargs) 2025-09-07T07:05:06.6550195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6550698Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6551207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6551665Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6551828Z 2025-09-07T07:05:06.6551944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6552347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6552733Z return mod(**inputs) 2025-09-07T07:05:06.6553156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6553615Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6554052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6554441Z hidden_states = self.encoder( 2025-09-07T07:05:06.6554843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6555230Z layer_outputs = layer_module( 2025-09-07T07:05:06.6555594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6555961Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6556357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6556770Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6557165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6557564Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6557998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6558474Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6558921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6559323Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6559477Z 2025-09-07T07:05:06.6559577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6559932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6560271Z return mod(**inputs) 2025-09-07T07:05:06.6560657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6561075Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6561503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6561918Z hidden_states = self.encoder( 2025-09-07T07:05:06.6562325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6562730Z layer_outputs = layer_module( 2025-09-07T07:05:06.6563093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6563474Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6563898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6564332Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6564731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6565133Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6565573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6566054Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6566499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6566940Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6567328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6567678Z return self.act(input) 2025-09-07T07:05:06.6567793Z 2025-09-07T07:05:06.6567904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6568270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6568598Z return mod(**inputs) 2025-09-07T07:05:06.6569009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6569436Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6569867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6570268Z hidden_states = self.encoder( 2025-09-07T07:05:06.6570665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6571076Z layer_outputs = layer_module( 2025-09-07T07:05:06.6571447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6571834Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6572281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6572731Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6573169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6573576Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6574007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6574548Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6575078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6575529Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6575669Z 2025-09-07T07:05:06.6575785Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6576146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6576487Z return mod(**inputs) 2025-09-07T07:05:06.6576871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6577332Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6577774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6578193Z hidden_states = self.encoder( 2025-09-07T07:05:06.6578609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6579060Z layer_outputs = layer_module( 2025-09-07T07:05:06.6579413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6579773Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6580185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6580604Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6581015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6581429Z return func(*args, **kwargs) 2025-09-07T07:05:06.6581836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6582264Z self_outputs = self.self( 2025-09-07T07:05:06.6582658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6583063Z return func(*args, **kwargs) 2025-09-07T07:05:06.6583479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6583911Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6584086Z 2025-09-07T07:05:06.6584200Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6584593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6584983Z return mod(**inputs) 2025-09-07T07:05:06.6585382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6585900Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6586369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6586810Z hidden_states = self.encoder( 2025-09-07T07:05:06.6587226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6587646Z layer_outputs = layer_module( 2025-09-07T07:05:06.6588021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6588409Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6588841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6589282Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6589685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6590108Z return func(*args, **kwargs) 2025-09-07T07:05:06.6590518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6590943Z self_outputs = self.self( 2025-09-07T07:05:06.6591325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6591727Z return func(*args, **kwargs) 2025-09-07T07:05:06.6592141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6592570Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6592716Z 2025-09-07T07:05:06.6592837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6593220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6593570Z return mod(**inputs) 2025-09-07T07:05:06.6593977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6594443Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6594878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6595307Z hidden_states = self.encoder( 2025-09-07T07:05:06.6595720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6596147Z layer_outputs = layer_module( 2025-09-07T07:05:06.6596523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6596906Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6597351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6597787Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6598201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6598609Z return func(*args, **kwargs) 2025-09-07T07:05:06.6599023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6599469Z self_outputs = self.self( 2025-09-07T07:05:06.6599837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6600228Z return func(*args, **kwargs) 2025-09-07T07:05:06.6600614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6601023Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6601168Z 2025-09-07T07:05:06.6601255Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6601491Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6601752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6602135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6602490Z return mod(**inputs) 2025-09-07T07:05:06.6602902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6603350Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6603786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6604208Z hidden_states = self.encoder( 2025-09-07T07:05:06.6604624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6605072Z layer_outputs = layer_module( 2025-09-07T07:05:06.6605445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6605830Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6606260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6606699Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6607117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6607530Z return func(*args, **kwargs) 2025-09-07T07:05:06.6607913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6608374Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6608832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6609269Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6609406Z 2025-09-07T07:05:06.6609519Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6609881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6610209Z return mod(**inputs) 2025-09-07T07:05:06.6610589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6611011Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6611416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6611814Z hidden_states = self.encoder( 2025-09-07T07:05:06.6612205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6612605Z layer_outputs = layer_module( 2025-09-07T07:05:06.6612957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6613315Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6613739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6614165Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6614600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6615050Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6615508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6616027Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6616486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6616927Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6617076Z 2025-09-07T07:05:06.6617187Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6617582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6617932Z return mod(**inputs) 2025-09-07T07:05:06.6618336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6618787Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6619225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6619812Z hidden_states = self.encoder( 2025-09-07T07:05:06.6621295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6621844Z layer_outputs = layer_module( 2025-09-07T07:05:06.6622242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6622654Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6623121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6623584Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6624030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6624468Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6624955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6625481Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6626298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6626793Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6627220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6627607Z return self.act(input) 2025-09-07T07:05:06.6627737Z 2025-09-07T07:05:06.6627855Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6628272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6628651Z return mod(**inputs) 2025-09-07T07:05:06.6629055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6629540Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6630004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6630473Z hidden_states = self.encoder( 2025-09-07T07:05:06.6630911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6631348Z layer_outputs = layer_module( 2025-09-07T07:05:06.6631712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6632176Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6632580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6632984Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6633395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6633796Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6634303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6634787Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6635225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6635635Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6635818Z 2025-09-07T07:05:06.6635933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6636308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6636646Z return mod(**inputs) 2025-09-07T07:05:06.6637031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6637468Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6637882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6638266Z hidden_states = self.encoder( 2025-09-07T07:05:06.6638633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6639022Z layer_outputs = layer_module( 2025-09-07T07:05:06.6639426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6639781Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6640170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6640562Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6640953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6641362Z return func(*args, **kwargs) 2025-09-07T07:05:06.6641746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6642138Z self_outputs = self.self( 2025-09-07T07:05:06.6642497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6642871Z return func(*args, **kwargs) 2025-09-07T07:05:06.6643251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6643655Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6643791Z 2025-09-07T07:05:06.6643897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6644263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6644597Z return mod(**inputs) 2025-09-07T07:05:06.6644960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6645356Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6645767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6646154Z hidden_states = self.encoder( 2025-09-07T07:05:06.6646552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6646944Z layer_outputs = layer_module( 2025-09-07T07:05:06.6647282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6647642Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6648036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6648440Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6648819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6649188Z return func(*args, **kwargs) 2025-09-07T07:05:06.6649569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6649962Z self_outputs = self.self( 2025-09-07T07:05:06.6650324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6650691Z return func(*args, **kwargs) 2025-09-07T07:05:06.6651063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6651500Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6651645Z 2025-09-07T07:05:06.6651755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6652133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6652459Z return mod(**inputs) 2025-09-07T07:05:06.6652854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6653279Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6653704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6654106Z hidden_states = self.encoder( 2025-09-07T07:05:06.6654496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6654900Z layer_outputs = layer_module( 2025-09-07T07:05:06.6655276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6655651Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6656073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6656500Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6656902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6657303Z return func(*args, **kwargs) 2025-09-07T07:05:06.6657710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6658124Z self_outputs = self.self( 2025-09-07T07:05:06.6658511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6658896Z return func(*args, **kwargs) 2025-09-07T07:05:06.6659294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6659721Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6659865Z 2025-09-07T07:05:06.6659977Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6660207Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6660454Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6660842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6661179Z return mod(**inputs) 2025-09-07T07:05:06.6661574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6662013Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6662439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6662850Z hidden_states = self.encoder( 2025-09-07T07:05:06.6663242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6663648Z layer_outputs = layer_module( 2025-09-07T07:05:06.6664010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6664391Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6664801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6665237Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6665776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6666241Z return func(*args, **kwargs) 2025-09-07T07:05:06.6666671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6667164Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6667663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6668090Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6668237Z 2025-09-07T07:05:06.6668359Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6668740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6669075Z return mod(**inputs) 2025-09-07T07:05:06.6669469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6669906Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6670349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6670760Z hidden_states = self.encoder( 2025-09-07T07:05:06.6671156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6671563Z layer_outputs = layer_module( 2025-09-07T07:05:06.6671932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6672310Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6672718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6673138Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6673579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6673994Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6674439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6674927Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6675408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6675824Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6675988Z 2025-09-07T07:05:06.6676105Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6676484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6676803Z return mod(**inputs) 2025-09-07T07:05:06.6677199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6677604Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6678003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6678391Z hidden_states = self.encoder( 2025-09-07T07:05:06.6678767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6679160Z layer_outputs = layer_module( 2025-09-07T07:05:06.6679509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6679873Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6680275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6680695Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6681085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6681471Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6681895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6682367Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6682794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6683216Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6683589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6683924Z return self.act(input) 2025-09-07T07:05:06.6684032Z 2025-09-07T07:05:06.6684133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6684503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6684831Z return mod(**inputs) 2025-09-07T07:05:06.6685214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6685632Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6686049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6686450Z hidden_states = self.encoder( 2025-09-07T07:05:06.6686856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6687247Z layer_outputs = layer_module( 2025-09-07T07:05:06.6687585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6687947Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6688354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6688769Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6689198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6689591Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6690038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6690536Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6690997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6691420Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6691564Z 2025-09-07T07:05:06.6691673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6692041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6692371Z return mod(**inputs) 2025-09-07T07:05:06.6692761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6693177Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6693594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6693999Z hidden_states = self.encoder( 2025-09-07T07:05:06.6694394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6694804Z layer_outputs = layer_module( 2025-09-07T07:05:06.6695145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6695510Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6695910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6696315Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6696688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6697063Z return func(*args, **kwargs) 2025-09-07T07:05:06.6697444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6697867Z self_outputs = self.self( 2025-09-07T07:05:06.6698127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6698213Z return func(*args, **kwargs) 2025-09-07T07:05:06.6698522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6698616Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6698620Z 2025-09-07T07:05:06.6698732Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6698950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6699030Z return mod(**inputs) 2025-09-07T07:05:06.6699318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6699417Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6699695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6699773Z hidden_states = self.encoder( 2025-09-07T07:05:06.6700064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6700141Z layer_outputs = layer_module( 2025-09-07T07:05:06.6700389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6700493Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6700788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6700901Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6701173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6701256Z return func(*args, **kwargs) 2025-09-07T07:05:06.6701540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6701626Z self_outputs = self.self( 2025-09-07T07:05:06.6701889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6701965Z return func(*args, **kwargs) 2025-09-07T07:05:06.6702255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6702343Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6702347Z 2025-09-07T07:05:06.6702466Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6702685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6702757Z return mod(**inputs) 2025-09-07T07:05:06.6703055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6703171Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6703462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6703538Z hidden_states = self.encoder( 2025-09-07T07:05:06.6703829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6703907Z layer_outputs = layer_module( 2025-09-07T07:05:06.6704148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6704242Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6704526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6704621Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6704880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6705019Z return func(*args, **kwargs) 2025-09-07T07:05:06.6705317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6705396Z self_outputs = self.self( 2025-09-07T07:05:06.6706130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6706261Z return func(*args, **kwargs) 2025-09-07T07:05:06.6706640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6706757Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6706763Z 2025-09-07T07:05:06.6706853Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6706945Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6707058Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6707272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6707344Z return mod(**inputs) 2025-09-07T07:05:06.6707615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6707711Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6708011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6708094Z hidden_states = self.encoder( 2025-09-07T07:05:06.6708381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6708458Z layer_outputs = layer_module( 2025-09-07T07:05:06.6708693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6708776Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6709051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6709133Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6709381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6709460Z return func(*args, **kwargs) 2025-09-07T07:05:06.6709730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6709870Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6710138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6710254Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6710257Z 2025-09-07T07:05:06.6710368Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6710578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6710659Z return mod(**inputs) 2025-09-07T07:05:06.6710937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6711041Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6711329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6711411Z hidden_states = self.encoder( 2025-09-07T07:05:06.6711711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6711787Z layer_outputs = layer_module( 2025-09-07T07:05:06.6712028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6712132Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6712404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6712491Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6712759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6712846Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6713152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6713281Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6713550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6713637Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6713642Z 2025-09-07T07:05:06.6713755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6713960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6714034Z return mod(**inputs) 2025-09-07T07:05:06.6714325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6714422Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6714716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6714793Z hidden_states = self.encoder( 2025-09-07T07:05:06.6715078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6715154Z layer_outputs = layer_module( 2025-09-07T07:05:06.6715399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6715481Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6715779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6715878Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6716158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6716248Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6716566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6716694Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6716999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6717134Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6717359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6717457Z return self.act(input) 2025-09-07T07:05:06.6717462Z 2025-09-07T07:05:06.6717593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6717796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6717863Z return mod(**inputs) 2025-09-07T07:05:06.6718147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6718234Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6718513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6718609Z hidden_states = self.encoder( 2025-09-07T07:05:06.6718878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6718956Z layer_outputs = layer_module( 2025-09-07T07:05:06.6719187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6719275Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6719546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6720063Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6720435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6720524Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6720838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6720979Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6721255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6721395Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6721400Z 2025-09-07T07:05:06.6721510Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6721759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6721829Z return mod(**inputs) 2025-09-07T07:05:06.6722104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6722195Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6722467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6722540Z hidden_states = self.encoder( 2025-09-07T07:05:06.6722805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6722885Z layer_outputs = layer_module( 2025-09-07T07:05:06.6723112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6723202Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6723476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6723561Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6723817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6723918Z return func(*args, **kwargs) 2025-09-07T07:05:06.6724196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6724270Z self_outputs = self.self( 2025-09-07T07:05:06.6724531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6724602Z return func(*args, **kwargs) 2025-09-07T07:05:06.6724874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6724970Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6724974Z 2025-09-07T07:05:06.6725078Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6725292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6725359Z return mod(**inputs) 2025-09-07T07:05:06.6725640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6725771Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6726054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6726142Z hidden_states = self.encoder( 2025-09-07T07:05:06.6726425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6726499Z layer_outputs = layer_module( 2025-09-07T07:05:06.6726737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6726817Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6727090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6727177Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6727430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6727505Z return func(*args, **kwargs) 2025-09-07T07:05:06.6727805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6727894Z self_outputs = self.self( 2025-09-07T07:05:06.6728172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6728254Z return func(*args, **kwargs) 2025-09-07T07:05:06.6728537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6728620Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6728625Z 2025-09-07T07:05:06.6728744Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6728965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6729042Z return mod(**inputs) 2025-09-07T07:05:06.6729334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6729430Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6729717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6729794Z hidden_states = self.encoder( 2025-09-07T07:05:06.6730083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6730157Z layer_outputs = layer_module( 2025-09-07T07:05:06.6730426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6730512Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6730791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6730886Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6731146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6731227Z return func(*args, **kwargs) 2025-09-07T07:05:06.6731513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6731587Z self_outputs = self.self( 2025-09-07T07:05:06.6731853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6731929Z return func(*args, **kwargs) 2025-09-07T07:05:06.6732215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6732321Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6732325Z 2025-09-07T07:05:06.6732417Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6732503Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6732615Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6732835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6732905Z return mod(**inputs) 2025-09-07T07:05:06.6733203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6733296Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6733573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6733662Z hidden_states = self.encoder( 2025-09-07T07:05:06.6733942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6734025Z layer_outputs = layer_module( 2025-09-07T07:05:06.6734288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6734367Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6734660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6734740Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6734986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6735055Z return func(*args, **kwargs) 2025-09-07T07:05:06.6735331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6735465Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6735729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6735825Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6735830Z 2025-09-07T07:05:06.6735939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6736161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6736233Z return mod(**inputs) 2025-09-07T07:05:06.6736506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6736605Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6736895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6736975Z hidden_states = self.encoder( 2025-09-07T07:05:06.6737240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6737320Z layer_outputs = layer_module( 2025-09-07T07:05:06.6737546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6737625Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6737896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6737983Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6738255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6738337Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6738655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6738786Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6739056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6739148Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6739152Z 2025-09-07T07:05:06.6739258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6739472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6739539Z return mod(**inputs) 2025-09-07T07:05:06.6739819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6739918Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6740184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6740266Z hidden_states = self.encoder( 2025-09-07T07:05:06.6740545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6740645Z layer_outputs = layer_module( 2025-09-07T07:05:06.6740877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6740984Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6741279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6741363Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6741624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6741713Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6742011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6742142Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6742432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6742555Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6742775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6742847Z return self.act(input) 2025-09-07T07:05:06.6742851Z 2025-09-07T07:05:06.6742963Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6743206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6743281Z return mod(**inputs) 2025-09-07T07:05:06.6743562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6743650Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6743939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6744013Z hidden_states = self.encoder( 2025-09-07T07:05:06.6744298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6744368Z layer_outputs = layer_module( 2025-09-07T07:05:06.6744603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6744683Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6744973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6745083Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6745344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6745427Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6745816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6745963Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6746253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6746337Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6746343Z 2025-09-07T07:05:06.6746456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6746665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6746740Z return mod(**inputs) 2025-09-07T07:05:06.6747010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6747119Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6747409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6747485Z hidden_states = self.encoder( 2025-09-07T07:05:06.6747776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6747848Z layer_outputs = layer_module( 2025-09-07T07:05:06.6748077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6748166Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6748434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6748525Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6748775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6748850Z return func(*args, **kwargs) 2025-09-07T07:05:06.6749123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6749192Z self_outputs = self.self( 2025-09-07T07:05:06.6749436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6749504Z return func(*args, **kwargs) 2025-09-07T07:05:06.6749787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6749869Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6749873Z 2025-09-07T07:05:06.6749973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6750178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6750242Z return mod(**inputs) 2025-09-07T07:05:06.6750517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6750604Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6750862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6750939Z hidden_states = self.encoder( 2025-09-07T07:05:06.6751195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6751292Z layer_outputs = layer_module( 2025-09-07T07:05:06.6751509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6751592Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6751852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6751933Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6752182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6752253Z return func(*args, **kwargs) 2025-09-07T07:05:06.6752525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6752598Z self_outputs = self.self( 2025-09-07T07:05:06.6752853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6752931Z return func(*args, **kwargs) 2025-09-07T07:05:06.6753189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6753276Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6753298Z 2025-09-07T07:05:06.6753404Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6753609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6753695Z return mod(**inputs) 2025-09-07T07:05:06.6753961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6754054Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6754315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6754398Z hidden_states = self.encoder( 2025-09-07T07:05:06.6754662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6754732Z layer_outputs = layer_module( 2025-09-07T07:05:06.6754959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6755037Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6755303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6755383Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6755619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6755715Z return func(*args, **kwargs) 2025-09-07T07:05:06.6755981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6756058Z self_outputs = self.self( 2025-09-07T07:05:06.6756299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6756373Z return func(*args, **kwargs) 2025-09-07T07:05:06.6756636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6756715Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6756720Z 2025-09-07T07:05:06.6756808Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6756885Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6756995Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6757193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6757259Z return mod(**inputs) 2025-09-07T07:05:06.6757577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6757662Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6757929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6757998Z hidden_states = self.encoder( 2025-09-07T07:05:06.6758256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6758333Z layer_outputs = layer_module( 2025-09-07T07:05:06.6758549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6758634Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6758895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6758989Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6759231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6759301Z return func(*args, **kwargs) 2025-09-07T07:05:06.6759591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6759724Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6760020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6760116Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6760119Z 2025-09-07T07:05:06.6760224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6760429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6760494Z return mod(**inputs) 2025-09-07T07:05:06.6760764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6760848Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6761118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6761187Z hidden_states = self.encoder( 2025-09-07T07:05:06.6761445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6761521Z layer_outputs = layer_module( 2025-09-07T07:05:06.6761742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6761846Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6762104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6762187Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6762450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6762528Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6762826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6762945Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6763200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6763291Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6763296Z 2025-09-07T07:05:06.6763395Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6763637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6763703Z return mod(**inputs) 2025-09-07T07:05:06.6763970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6764056Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6764318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6764399Z hidden_states = self.encoder( 2025-09-07T07:05:06.6764659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6764734Z layer_outputs = layer_module( 2025-09-07T07:05:06.6764956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6765040Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6765317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6765400Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6765688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6765769Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6766094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6766219Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6766486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6766610Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6766829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6766908Z return self.act(input) 2025-09-07T07:05:06.6766912Z 2025-09-07T07:05:06.6767016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6767229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6767303Z return mod(**inputs) 2025-09-07T07:05:06.6767596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6767696Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6768002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6768144Z hidden_states = self.encoder( 2025-09-07T07:05:06.6768440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6768519Z layer_outputs = layer_module( 2025-09-07T07:05:06.6768775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6768857Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6769136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6769223Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6769492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6769578Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6769880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6770045Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6770311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6770401Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6770404Z 2025-09-07T07:05:06.6770510Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6770717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6770795Z return mod(**inputs) 2025-09-07T07:05:06.6771066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6771162Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6771429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6771504Z hidden_states = self.encoder( 2025-09-07T07:05:06.6771781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6771852Z layer_outputs = layer_module( 2025-09-07T07:05:06.6772102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6772182Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6772468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6772564Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6772810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6772888Z return func(*args, **kwargs) 2025-09-07T07:05:06.6773164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6773245Z self_outputs = self.self( 2025-09-07T07:05:06.6773492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6773566Z return func(*args, **kwargs) 2025-09-07T07:05:06.6773856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6773943Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6773946Z 2025-09-07T07:05:06.6774066Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6774287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6774355Z return mod(**inputs) 2025-09-07T07:05:06.6774639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6774748Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6775028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6775098Z hidden_states = self.encoder( 2025-09-07T07:05:06.6775369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6775441Z layer_outputs = layer_module( 2025-09-07T07:05:06.6775666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6775752Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6776016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6776108Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6776354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6776442Z return func(*args, **kwargs) 2025-09-07T07:05:06.6776716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6776788Z self_outputs = self.self( 2025-09-07T07:05:06.6777046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6777114Z return func(*args, **kwargs) 2025-09-07T07:05:06.6777384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6777471Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6777475Z 2025-09-07T07:05:06.6777578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6777790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6777860Z return mod(**inputs) 2025-09-07T07:05:06.6778136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6778224Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6778517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6778597Z hidden_states = self.encoder( 2025-09-07T07:05:06.6778886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6778965Z layer_outputs = layer_module( 2025-09-07T07:05:06.6779190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6779272Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6779552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6779639Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6779889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6779961Z return func(*args, **kwargs) 2025-09-07T07:05:06.6780233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6780305Z self_outputs = self.self( 2025-09-07T07:05:06.6780552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6780630Z return func(*args, **kwargs) 2025-09-07T07:05:06.6780933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6781047Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6781052Z 2025-09-07T07:05:06.6781138Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6781222Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6781344Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6781563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6781639Z return mod(**inputs) 2025-09-07T07:05:06.6781927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6782020Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6782312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6782388Z hidden_states = self.encoder( 2025-09-07T07:05:06.6782681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6782778Z layer_outputs = layer_module( 2025-09-07T07:05:06.6783024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6783109Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6783405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6783502Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6783773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6783857Z return func(*args, **kwargs) 2025-09-07T07:05:06.6784157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6784299Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6784594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6784682Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6784685Z 2025-09-07T07:05:06.6784803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6785036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6785108Z return mod(**inputs) 2025-09-07T07:05:06.6785421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6785519Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6785915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6786003Z hidden_states = self.encoder( 2025-09-07T07:05:06.6786307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6786386Z layer_outputs = layer_module( 2025-09-07T07:05:06.6786635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6786734Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6787043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6787144Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6787429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6787512Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6787837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6788002Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6788299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6788385Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6788389Z 2025-09-07T07:05:06.6788508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6788722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6788794Z return mod(**inputs) 2025-09-07T07:05:06.6789086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6789178Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6789462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6789572Z hidden_states = self.encoder( 2025-09-07T07:05:06.6789853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6789936Z layer_outputs = layer_module( 2025-09-07T07:05:06.6790183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6790274Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6790568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6790664Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6790946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6791031Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6791362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6791491Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6791777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6791930Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6792161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6792268Z return self.act(input) 2025-09-07T07:05:06.6792273Z 2025-09-07T07:05:06.6792383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6792610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6792683Z return mod(**inputs) 2025-09-07T07:05:06.6792980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6793074Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6793359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6793444Z hidden_states = self.encoder( 2025-09-07T07:05:06.6793728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6793812Z layer_outputs = layer_module( 2025-09-07T07:05:06.6794052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6794135Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6794424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6794540Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6794828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6794910Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6795229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6795378Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6795663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6795756Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6795759Z 2025-09-07T07:05:06.6795869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6796089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6796161Z return mod(**inputs) 2025-09-07T07:05:06.6796481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6796584Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6796885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6796968Z hidden_states = self.encoder( 2025-09-07T07:05:06.6797250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6797324Z layer_outputs = layer_module( 2025-09-07T07:05:06.6797569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6797653Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6797945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6798034Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6798305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6798382Z return func(*args, **kwargs) 2025-09-07T07:05:06.6798685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6798773Z self_outputs = self.self( 2025-09-07T07:05:06.6799048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6799133Z return func(*args, **kwargs) 2025-09-07T07:05:06.6799424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-09-07T07:05:06.6799512Z query_layer = self.query(hidden_states) 2025-09-07T07:05:06.6799518Z 2025-09-07T07:05:06.6799636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6799850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6799928Z return mod(**inputs) 2025-09-07T07:05:06.6800213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6800314Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6800599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6800674Z hidden_states = self.encoder( 2025-09-07T07:05:06.6800962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6801055Z layer_outputs = layer_module( 2025-09-07T07:05:06.6801301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6801386Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6801671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6801767Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6802031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6802114Z return func(*args, **kwargs) 2025-09-07T07:05:06.6802397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6802473Z self_outputs = self.self( 2025-09-07T07:05:06.6802741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6802817Z return func(*args, **kwargs) 2025-09-07T07:05:06.6803137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-09-07T07:05:06.6803221Z key_layer = self.key(current_states) 2025-09-07T07:05:06.6803224Z 2025-09-07T07:05:06.6803345Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6803552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6803618Z return mod(**inputs) 2025-09-07T07:05:06.6803898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6803985Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6804286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6804363Z hidden_states = self.encoder( 2025-09-07T07:05:06.6804643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6804725Z layer_outputs = layer_module( 2025-09-07T07:05:06.6804961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6805075Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6805355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6805469Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6805739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6805815Z return func(*args, **kwargs) 2025-09-07T07:05:06.6806104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-09-07T07:05:06.6806182Z self_outputs = self.self( 2025-09-07T07:05:06.6806456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6806532Z return func(*args, **kwargs) 2025-09-07T07:05:06.6806814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-09-07T07:05:06.6806910Z value_layer = self.value(current_states) 2025-09-07T07:05:06.6806913Z 2025-09-07T07:05:06.6807001Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6807095Z cudagraph partition due to non gpu ops 2025-09-07T07:05:06.6807206Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6807423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6807501Z return mod(**inputs) 2025-09-07T07:05:06.6807808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6807919Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6808186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6808265Z hidden_states = self.encoder( 2025-09-07T07:05:06.6808535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6808605Z layer_outputs = layer_module( 2025-09-07T07:05:06.6808838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6808918Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6809190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-09-07T07:05:06.6809274Z self_attention_outputs = self.attention( 2025-09-07T07:05:06.6809535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:06.6809614Z return func(*args, **kwargs) 2025-09-07T07:05:06.6809877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-09-07T07:05:06.6810015Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:05:06.6810286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-09-07T07:05:06.6810371Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6810383Z 2025-09-07T07:05:06.6810487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6810686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6810762Z return mod(**inputs) 2025-09-07T07:05:06.6811029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6811127Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6811394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6811483Z hidden_states = self.encoder( 2025-09-07T07:05:06.6811756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6811849Z layer_outputs = layer_module( 2025-09-07T07:05:06.6812082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6812166Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6812447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6812546Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6812827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6812912Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6813214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6813344Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6813627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-09-07T07:05:06.6813714Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6813718Z 2025-09-07T07:05:06.6813837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6814083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6814160Z return mod(**inputs) 2025-09-07T07:05:06.6814432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6814520Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6814797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6814870Z hidden_states = self.encoder( 2025-09-07T07:05:06.6815147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6815218Z layer_outputs = layer_module( 2025-09-07T07:05:06.6815450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6815531Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6815802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6815915Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6816180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6816265Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6816569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-09-07T07:05:06.6816693Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:05:06.6816970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-09-07T07:05:06.6817084Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:05:06.6817319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:05:06.6817394Z return self.act(input) 2025-09-07T07:05:06.6817397Z 2025-09-07T07:05:06.6817509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6817713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6817781Z return mod(**inputs) 2025-09-07T07:05:06.6818087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-09-07T07:05:06.6818175Z discriminator_hidden_states = self.electra( 2025-09-07T07:05:06.6818475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-09-07T07:05:06.6818550Z hidden_states = self.encoder( 2025-09-07T07:05:06.6818814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-09-07T07:05:06.6818895Z layer_outputs = layer_module( 2025-09-07T07:05:06.6819121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:06.6819209Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:06.6819476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-09-07T07:05:06.6820027Z layer_output = apply_chunking_to_forward( 2025-09-07T07:05:06.6821203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:05:06.6821333Z return forward_fn(*input_tensors) 2025-09-07T07:05:06.6821671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-09-07T07:05:06.6821822Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:05:06.6822276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-09-07T07:05:06.6822372Z hidden_states = self.dense(hidden_states) 2025-09-07T07:05:06.6822378Z 2025-09-07T07:05:06.6822496Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6822730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6822803Z return mod(**inputs) 2025-09-07T07:05:06.6823111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1330, in forward 2025-09-07T07:05:06.6823205Z logits = self.qa_outputs(sequence_output) 2025-09-07T07:05:06.6823208Z 2025-09-07T07:05:06.6823329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6823549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6823623Z return mod(**inputs) 2025-09-07T07:05:06.6823964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1348, in forward 2025-09-07T07:05:06.6824078Z start_loss = loss_fct(start_logits, start_positions) 2025-09-07T07:05:06.6824082Z 2025-09-07T07:05:06.6824199Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:06.6824415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:06.6824486Z return mod(**inputs) 2025-09-07T07:05:06.6824788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1349, in forward 2025-09-07T07:05:06.6824888Z end_loss = loss_fct(end_logits, end_positions) 2025-09-07T07:05:06.6824892Z 2025-09-07T07:05:16.5403719Z Compilation time (from dynamo_timed): 17.032677505 2025-09-07T07:05:16.5404022Z pass 2025-09-07T07:05:16.5404661Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:16.5405601Z TIMING: _recursive_pre_grad_passes:0.00749 _recursive_joint_graph_passes:0.45678 _recursive_post_grad_passes:0.08082 async_compile.wait:0.00232 code_gen:9.27285 inductor_compile:10.53141 backend_compile:14.13266 gc:0.00175 entire_frame_compile:17.03268 total_wall_time:17.03268 2025-09-07T07:05:16.5406820Z STATS: call_* op count: 378 | FakeTensorMode.__torch_dispatch__:15000 | FakeTensor.__torch_dispatch__:4378 | ProxyTorchDispatchMode.__torch_dispatch__:5698 2025-09-07T07:05:16.5412817Z Dynamo produced 1 graphs covering 378 ops with 0 graph breaks (0 unique) 2025-09-07T07:05:19.1853744Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:05:19.1854727Z import pynvml # type: ignore[import] 2025-09-07T07:05:21.9778624Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:05:21.9780571Z from pkg_resources import resource_filename 2025-09-07T07:05:22.6539068Z 2025-09-07T07:05:24.0215166Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:05:24.0215472Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:05:24.0222133Z cpu eval GPT2ForSequenceClassification 2025-09-07T07:05:24.7554653Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:25.0787905Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:25.4044861Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:32.5012341Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5012699Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5012931Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5013160Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5013383Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5013634Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5013850Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5014076Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5014298Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5014528Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5014778Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5015002Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5015267Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5015689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5016443Z return mod(**inputs) 2025-09-07T07:05:32.5016890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1509, in forward 2025-09-07T07:05:32.5017395Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-09-07T07:05:32.5017589Z 2025-09-07T07:05:32.5017724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5018131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5018482Z return mod(**inputs) 2025-09-07T07:05:32.5018893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5019382Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5020022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5020443Z outputs = block( 2025-09-07T07:05:32.5020802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5021257Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5021740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5022270Z return func(*args, **kwargs) 2025-09-07T07:05:32.5022709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5023212Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5023671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5024084Z return func(*args, **kwargs) 2025-09-07T07:05:32.5024489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5025036Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5025548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5026049Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5026242Z 2025-09-07T07:05:32.5026343Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5026571Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5026787Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5027008Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5027264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5027654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5028003Z return mod(**inputs) 2025-09-07T07:05:32.5028451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5028886Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5029309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5029708Z outputs = block( 2025-09-07T07:05:32.5030066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5030469Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5030888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5031325Z return func(*args, **kwargs) 2025-09-07T07:05:32.5031722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5032154Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5032573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5033006Z return func(*args, **kwargs) 2025-09-07T07:05:32.5033399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5033844Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5034330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5034853Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5035042Z 2025-09-07T07:05:32.5035158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5035526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5035864Z return mod(**inputs) 2025-09-07T07:05:32.5036237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5036646Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5037044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5037415Z outputs = block( 2025-09-07T07:05:32.5037768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5038144Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5038549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5039153Z return func(*args, **kwargs) 2025-09-07T07:05:32.5039554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5040024Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5040442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5040846Z return func(*args, **kwargs) 2025-09-07T07:05:32.5041254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5041693Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5042178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5042678Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5042852Z 2025-09-07T07:05:32.5042973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5043356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5043769Z return mod(**inputs) 2025-09-07T07:05:32.5044166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5044617Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5045018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5045398Z outputs = block( 2025-09-07T07:05:32.5045727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5046106Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5046495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5046872Z return func(*args, **kwargs) 2025-09-07T07:05:32.5047263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5047697Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5048130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5048534Z return func(*args, **kwargs) 2025-09-07T07:05:32.5048929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5049354Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5049748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5050182Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5050370Z 2025-09-07T07:05:32.5050494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5050880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5051244Z return mod(**inputs) 2025-09-07T07:05:32.5051635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5052065Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5052485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5052908Z outputs = block( 2025-09-07T07:05:32.5053256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5053666Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5054075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5054468Z return func(*args, **kwargs) 2025-09-07T07:05:32.5054865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5055313Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5055755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5056173Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5056555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5056985Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5057175Z 2025-09-07T07:05:32.5057289Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5057681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5058027Z return mod(**inputs) 2025-09-07T07:05:32.5058416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5058871Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5059314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5059700Z outputs = block( 2025-09-07T07:05:32.5060036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5060416Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5060810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5061202Z return func(*args, **kwargs) 2025-09-07T07:05:32.5061578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5062018Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5062469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5062908Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5063282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5063770Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5064032Z 2025-09-07T07:05:32.5064146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5064534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5064890Z return mod(**inputs) 2025-09-07T07:05:32.5065281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5065888Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5066319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5066723Z outputs = block( 2025-09-07T07:05:32.5067068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5067458Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5067891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5068306Z return func(*args, **kwargs) 2025-09-07T07:05:32.5068737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5069183Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5069615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5070024Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5070395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5070808Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5070984Z 2025-09-07T07:05:32.5071097Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5071463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5071796Z return mod(**inputs) 2025-09-07T07:05:32.5072160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5072565Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5072952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5073332Z outputs = block( 2025-09-07T07:05:32.5073680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5074056Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5074445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5074826Z return func(*args, **kwargs) 2025-09-07T07:05:32.5075210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5075622Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5076029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5076409Z return func(*args, **kwargs) 2025-09-07T07:05:32.5076783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5077296Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5077797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5078207Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5078384Z 2025-09-07T07:05:32.5078477Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5078697Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5078916Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5079131Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5079376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5079743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5080077Z return mod(**inputs) 2025-09-07T07:05:32.5080447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5080854Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5081244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5081623Z outputs = block( 2025-09-07T07:05:32.5081948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5082341Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5082729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5083124Z return func(*args, **kwargs) 2025-09-07T07:05:32.5083502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5083907Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5084309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5084682Z return func(*args, **kwargs) 2025-09-07T07:05:32.5085058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5085471Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5085935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5086426Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5086611Z 2025-09-07T07:05:32.5086719Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5087092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5087422Z return mod(**inputs) 2025-09-07T07:05:32.5087792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5088220Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5088614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5089000Z outputs = block( 2025-09-07T07:05:32.5089335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5089727Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5090138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5090549Z return func(*args, **kwargs) 2025-09-07T07:05:32.5090951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5091385Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5091813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5092222Z return func(*args, **kwargs) 2025-09-07T07:05:32.5092620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5093056Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5093538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5094035Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5094212Z 2025-09-07T07:05:32.5094323Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5094714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5095066Z return mod(**inputs) 2025-09-07T07:05:32.5095457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5095876Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5096300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5096701Z outputs = block( 2025-09-07T07:05:32.5097073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5097465Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5097906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5098313Z return func(*args, **kwargs) 2025-09-07T07:05:32.5098711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5099141Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5099568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5099969Z return func(*args, **kwargs) 2025-09-07T07:05:32.5100398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5100823Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5101215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5101646Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5101842Z 2025-09-07T07:05:32.5101955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5102349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5102705Z return mod(**inputs) 2025-09-07T07:05:32.5103116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5103537Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5103957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5104355Z outputs = block( 2025-09-07T07:05:32.5104700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5105093Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5105498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5105988Z return func(*args, **kwargs) 2025-09-07T07:05:32.5106400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5106869Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5107307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5107749Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5108135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5108572Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5108756Z 2025-09-07T07:05:32.5108877Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5109260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5109610Z return mod(**inputs) 2025-09-07T07:05:32.5109996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5110410Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5110798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5111175Z outputs = block( 2025-09-07T07:05:32.5111499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5111868Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5112271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5112650Z return func(*args, **kwargs) 2025-09-07T07:05:32.5113048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5113483Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5113923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5114348Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5114721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5115214Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5115478Z 2025-09-07T07:05:32.5115593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5115981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5116313Z return mod(**inputs) 2025-09-07T07:05:32.5116674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5117089Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5117510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5117936Z outputs = block( 2025-09-07T07:05:32.5118275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5118665Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5119098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5119483Z return func(*args, **kwargs) 2025-09-07T07:05:32.5120052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5120470Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5120890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5121294Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5121675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5122153Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5122341Z 2025-09-07T07:05:32.5122450Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5139028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5139400Z return mod(**inputs) 2025-09-07T07:05:32.5139834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5140277Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5140728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5141141Z outputs = block( 2025-09-07T07:05:32.5141509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5141924Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5142353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5142766Z return func(*args, **kwargs) 2025-09-07T07:05:32.5143179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5143759Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5144201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5144662Z return func(*args, **kwargs) 2025-09-07T07:05:32.5145076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5145734Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5146267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5146715Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5146913Z 2025-09-07T07:05:32.5147014Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5147251Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5147484Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5147715Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5147971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5148376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5148738Z return mod(**inputs) 2025-09-07T07:05:32.5149138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5149567Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5150040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5150448Z outputs = block( 2025-09-07T07:05:32.5150802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5151203Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5151609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5152015Z return func(*args, **kwargs) 2025-09-07T07:05:32.5152421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5152853Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5153270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5153738Z return func(*args, **kwargs) 2025-09-07T07:05:32.5154170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5154613Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5155088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5155582Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5155772Z 2025-09-07T07:05:32.5155878Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5156238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5156566Z return mod(**inputs) 2025-09-07T07:05:32.5156924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5157313Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5157698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5158072Z outputs = block( 2025-09-07T07:05:32.5158397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5158788Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5159191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5159570Z return func(*args, **kwargs) 2025-09-07T07:05:32.5159951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5160354Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5160745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5161145Z return func(*args, **kwargs) 2025-09-07T07:05:32.5161547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5161986Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5162452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5162931Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5163105Z 2025-09-07T07:05:32.5163216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5163588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5163926Z return mod(**inputs) 2025-09-07T07:05:32.5164311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5164768Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5165202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5165587Z outputs = block( 2025-09-07T07:05:32.5165914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5166281Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5166689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5167091Z return func(*args, **kwargs) 2025-09-07T07:05:32.5167491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5167913Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5168332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5168757Z return func(*args, **kwargs) 2025-09-07T07:05:32.5169155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5169580Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5169967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5170404Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5170597Z 2025-09-07T07:05:32.5170712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5171104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5171453Z return mod(**inputs) 2025-09-07T07:05:32.5171832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5172263Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5172685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5173087Z outputs = block( 2025-09-07T07:05:32.5173426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5173840Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5174252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5174670Z return func(*args, **kwargs) 2025-09-07T07:05:32.5175067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5175506Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5175948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5176369Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5176755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5177186Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5177369Z 2025-09-07T07:05:32.5177482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5177872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5178226Z return mod(**inputs) 2025-09-07T07:05:32.5178612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5179034Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5179449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5179911Z outputs = block( 2025-09-07T07:05:32.5180264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5180657Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5181063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5181468Z return func(*args, **kwargs) 2025-09-07T07:05:32.5181864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5182309Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5182739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5183163Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5183550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5184073Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5184329Z 2025-09-07T07:05:32.5184449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5184834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5185185Z return mod(**inputs) 2025-09-07T07:05:32.5185573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5186114Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5186537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5186978Z outputs = block( 2025-09-07T07:05:32.5187329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5187724Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5188134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5188537Z return func(*args, **kwargs) 2025-09-07T07:05:32.5188946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5189393Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5189858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5190283Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5190663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5191088Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5191283Z 2025-09-07T07:05:32.5191396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5191786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5192131Z return mod(**inputs) 2025-09-07T07:05:32.5192512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5192935Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5193352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5193748Z outputs = block( 2025-09-07T07:05:32.5194087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5194470Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5194912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5195315Z return func(*args, **kwargs) 2025-09-07T07:05:32.5195710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-09-07T07:05:32.5196150Z hidden_states = residual + feed_forward_hidden_states 2025-09-07T07:05:32.5196321Z 2025-09-07T07:05:32.5196428Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5196797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5197134Z return mod(**inputs) 2025-09-07T07:05:32.5197499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5197897Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5198300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5198704Z outputs = block( 2025-09-07T07:05:32.5199034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5199401Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5199788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5200191Z return func(*args, **kwargs) 2025-09-07T07:05:32.5200589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5201016Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5201430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5201829Z return func(*args, **kwargs) 2025-09-07T07:05:32.5202226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5202764Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5203262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5203703Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5203899Z 2025-09-07T07:05:32.5203989Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5204223Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5204473Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5204691Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5204948Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5205342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5205710Z return mod(**inputs) 2025-09-07T07:05:32.5206072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5206479Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5206877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5207260Z outputs = block( 2025-09-07T07:05:32.5207588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5207956Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5208350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5208732Z return func(*args, **kwargs) 2025-09-07T07:05:32.5209109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5209539Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5209944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5210341Z return func(*args, **kwargs) 2025-09-07T07:05:32.5210740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5211172Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5211651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5212148Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5212335Z 2025-09-07T07:05:32.5212458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5212848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5213194Z return mod(**inputs) 2025-09-07T07:05:32.5213606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5214042Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5214469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5214878Z outputs = block( 2025-09-07T07:05:32.5215226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5215630Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5216047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5216460Z return func(*args, **kwargs) 2025-09-07T07:05:32.5216859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5217297Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5217722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5218129Z return func(*args, **kwargs) 2025-09-07T07:05:32.5218553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5218988Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5219486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5220171Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5220350Z 2025-09-07T07:05:32.5220473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5220879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5221227Z return mod(**inputs) 2025-09-07T07:05:32.5221620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5222050Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5222483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5222878Z outputs = block( 2025-09-07T07:05:32.5223234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5223631Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5224049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5224455Z return func(*args, **kwargs) 2025-09-07T07:05:32.5224905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5225341Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5225832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5226248Z return func(*args, **kwargs) 2025-09-07T07:05:32.5226655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5227087Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5227483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5227916Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5228106Z 2025-09-07T07:05:32.5228220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5228585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5228967Z return mod(**inputs) 2025-09-07T07:05:32.5229346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5229758Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5230157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5230531Z outputs = block( 2025-09-07T07:05:32.5230868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5231246Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5231638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5232022Z return func(*args, **kwargs) 2025-09-07T07:05:32.5232413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5232843Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5233269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5233677Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5234071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5234481Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5234711Z 2025-09-07T07:05:32.5234820Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5235192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5235526Z return mod(**inputs) 2025-09-07T07:05:32.5235887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5236294Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5236685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5237066Z outputs = block( 2025-09-07T07:05:32.5237393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5237765Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5238159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5238545Z return func(*args, **kwargs) 2025-09-07T07:05:32.5238925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5239444Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5239850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5240245Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5240595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5241045Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5241283Z 2025-09-07T07:05:32.5241386Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5241752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5242078Z return mod(**inputs) 2025-09-07T07:05:32.5242439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5242826Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5243212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5243600Z outputs = block( 2025-09-07T07:05:32.5243919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5244275Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5244653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5245025Z return func(*args, **kwargs) 2025-09-07T07:05:32.5245391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5245795Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5246187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5246579Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5246941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5247336Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5247504Z 2025-09-07T07:05:32.5247613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5247981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5248309Z return mod(**inputs) 2025-09-07T07:05:32.5248695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5249101Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5249499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5249875Z outputs = block( 2025-09-07T07:05:32.5250216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5250609Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5251016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5251396Z return func(*args, **kwargs) 2025-09-07T07:05:32.5251777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5252241Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5252632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5253005Z return func(*args, **kwargs) 2025-09-07T07:05:32.5253367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5253896Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5254356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5254748Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5254918Z 2025-09-07T07:05:32.5255007Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5255213Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5255424Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5255630Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5255864Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5256215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5256540Z return mod(**inputs) 2025-09-07T07:05:32.5256904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5257323Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5257713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5258077Z outputs = block( 2025-09-07T07:05:32.5258402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5258765Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5259148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5259513Z return func(*args, **kwargs) 2025-09-07T07:05:32.5259878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5260275Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5260665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5261045Z return func(*args, **kwargs) 2025-09-07T07:05:32.5261418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5261818Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5262290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5262816Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5263002Z 2025-09-07T07:05:32.5263116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5263478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5263818Z return mod(**inputs) 2025-09-07T07:05:32.5264192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5264600Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5265011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5265413Z outputs = block( 2025-09-07T07:05:32.5265846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5266257Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5266694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5267092Z return func(*args, **kwargs) 2025-09-07T07:05:32.5267493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5267951Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5268367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5268744Z return func(*args, **kwargs) 2025-09-07T07:05:32.5269118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5269530Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5269985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5270457Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5270620Z 2025-09-07T07:05:32.5270726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5271093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5271424Z return mod(**inputs) 2025-09-07T07:05:32.5271792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5272221Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5272611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5273003Z outputs = block( 2025-09-07T07:05:32.5273331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5273693Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5274079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5274459Z return func(*args, **kwargs) 2025-09-07T07:05:32.5274833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5275237Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5275621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5275996Z return func(*args, **kwargs) 2025-09-07T07:05:32.5276389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5276788Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5277153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5277569Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5277753Z 2025-09-07T07:05:32.5277862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5278237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5278573Z return mod(**inputs) 2025-09-07T07:05:32.5278940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5279350Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5279766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5280173Z outputs = block( 2025-09-07T07:05:32.5280528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5280926Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5281319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5281706Z return func(*args, **kwargs) 2025-09-07T07:05:32.5282089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5282521Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5282939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5283334Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5283701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5284104Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5284278Z 2025-09-07T07:05:32.5284385Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5284766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5285113Z return mod(**inputs) 2025-09-07T07:05:32.5285466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5285862Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5286270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5286648Z outputs = block( 2025-09-07T07:05:32.5286982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5287367Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5287768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5288150Z return func(*args, **kwargs) 2025-09-07T07:05:32.5288527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5288941Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5289355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5289743Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5290099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5290557Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5290811Z 2025-09-07T07:05:32.5290927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5291296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5291638Z return mod(**inputs) 2025-09-07T07:05:32.5292010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5292426Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5292815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5293187Z outputs = block( 2025-09-07T07:05:32.5293503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5293863Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5294241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5294611Z return func(*args, **kwargs) 2025-09-07T07:05:32.5294975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5295380Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5295790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5296194Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5296593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5296983Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5297161Z 2025-09-07T07:05:32.5297269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5297636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5297960Z return mod(**inputs) 2025-09-07T07:05:32.5298315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5298713Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5299113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5299496Z outputs = block( 2025-09-07T07:05:32.5299831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5300209Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5300596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5301001Z return func(*args, **kwargs) 2025-09-07T07:05:32.5301404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-09-07T07:05:32.5301852Z hidden_states = residual + feed_forward_hidden_states 2025-09-07T07:05:32.5302025Z 2025-09-07T07:05:32.5302138Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5302529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5302876Z return mod(**inputs) 2025-09-07T07:05:32.5303291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5303714Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5304136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5304532Z outputs = block( 2025-09-07T07:05:32.5304879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5305317Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5305807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5306262Z return func(*args, **kwargs) 2025-09-07T07:05:32.5306677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5307135Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5307549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5307932Z return func(*args, **kwargs) 2025-09-07T07:05:32.5308309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5308819Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5309293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5309693Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5309877Z 2025-09-07T07:05:32.5309963Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5310185Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5310404Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5310616Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5310875Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5311250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5311583Z return mod(**inputs) 2025-09-07T07:05:32.5311952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5312350Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5312749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5313128Z outputs = block( 2025-09-07T07:05:32.5313463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5313832Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5314212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5314591Z return func(*args, **kwargs) 2025-09-07T07:05:32.5314986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5315391Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5315779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5316167Z return func(*args, **kwargs) 2025-09-07T07:05:32.5316545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5316954Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5317422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5317892Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5318082Z 2025-09-07T07:05:32.5318184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5318543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5318863Z return mod(**inputs) 2025-09-07T07:05:32.5319223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5319888Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5320292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5320693Z outputs = block( 2025-09-07T07:05:32.5321030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5321401Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5321780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5322154Z return func(*args, **kwargs) 2025-09-07T07:05:32.5322523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5322920Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5323301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5323671Z return func(*args, **kwargs) 2025-09-07T07:05:32.5324037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5324443Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5324891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5325337Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5325540Z 2025-09-07T07:05:32.5325645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5326004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5326323Z return mod(**inputs) 2025-09-07T07:05:32.5326671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5327067Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5327454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5327825Z outputs = block( 2025-09-07T07:05:32.5328151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5328503Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5328885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5329283Z return func(*args, **kwargs) 2025-09-07T07:05:32.5329647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5330035Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5330423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5330792Z return func(*args, **kwargs) 2025-09-07T07:05:32.5331160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5331546Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5331892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5332286Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5332467Z 2025-09-07T07:05:32.5332574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5332932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5333256Z return mod(**inputs) 2025-09-07T07:05:32.5333625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5334017Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5334409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5334781Z outputs = block( 2025-09-07T07:05:32.5335088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5335445Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5335819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5336191Z return func(*args, **kwargs) 2025-09-07T07:05:32.5336553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5336961Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5337356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5337733Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5338081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5338460Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5338624Z 2025-09-07T07:05:32.5338728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5339115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5339443Z return mod(**inputs) 2025-09-07T07:05:32.5339803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5340188Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5340575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5340948Z outputs = block( 2025-09-07T07:05:32.5341269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5341629Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5342005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5342375Z return func(*args, **kwargs) 2025-09-07T07:05:32.5342743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5343194Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5343599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5343977Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5344329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5344786Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5345023Z 2025-09-07T07:05:32.5345137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5345505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5345928Z return mod(**inputs) 2025-09-07T07:05:32.5346324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5346756Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5347159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5347516Z outputs = block( 2025-09-07T07:05:32.5347862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5348227Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5348625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5349003Z return func(*args, **kwargs) 2025-09-07T07:05:32.5349382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5349790Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5350195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5350595Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5350948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5351345Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5351523Z 2025-09-07T07:05:32.5351628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5351995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5352316Z return mod(**inputs) 2025-09-07T07:05:32.5352666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5353075Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5353457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5353823Z outputs = block( 2025-09-07T07:05:32.5354148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5354507Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5354895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5355267Z return func(*args, **kwargs) 2025-09-07T07:05:32.5355633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5356023Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5356454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5356871Z return func(*args, **kwargs) 2025-09-07T07:05:32.5357316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5357883Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5358371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5358780Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5358960Z 2025-09-07T07:05:32.5359044Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5359270Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5359478Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5359689Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5359931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5360303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5360634Z return mod(**inputs) 2025-09-07T07:05:32.5360994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5361402Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5361816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5362186Z outputs = block( 2025-09-07T07:05:32.5362501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5362882Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5363265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5363638Z return func(*args, **kwargs) 2025-09-07T07:05:32.5364010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5364402Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5364792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5365161Z return func(*args, **kwargs) 2025-09-07T07:05:32.5365526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5365928Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5366366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5366847Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5367034Z 2025-09-07T07:05:32.5367138Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5367520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5367838Z return mod(**inputs) 2025-09-07T07:05:32.5368197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5368588Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5368994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5369376Z outputs = block( 2025-09-07T07:05:32.5369702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5370081Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5370475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5370861Z return func(*args, **kwargs) 2025-09-07T07:05:32.5371230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5371668Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5372080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5372457Z return func(*args, **kwargs) 2025-09-07T07:05:32.5372825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5373264Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5373724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5374193Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5374356Z 2025-09-07T07:05:32.5374475Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5374841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5375176Z return mod(**inputs) 2025-09-07T07:05:32.5375545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5375946Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5376365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5376755Z outputs = block( 2025-09-07T07:05:32.5377101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5377474Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5377861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5378247Z return func(*args, **kwargs) 2025-09-07T07:05:32.5378613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5379018Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5379412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5379794Z return func(*args, **kwargs) 2025-09-07T07:05:32.5380172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5380568Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5380936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5381350Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5381526Z 2025-09-07T07:05:32.5381669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5382033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5382363Z return mod(**inputs) 2025-09-07T07:05:32.5382729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5383157Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5383552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5383917Z outputs = block( 2025-09-07T07:05:32.5384266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5384644Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5385043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5385456Z return func(*args, **kwargs) 2025-09-07T07:05:32.5385966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5386437Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5386905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5387354Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5387738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5388174Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5388370Z 2025-09-07T07:05:32.5388487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5388874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5389214Z return mod(**inputs) 2025-09-07T07:05:32.5389575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5389983Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5390381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5390764Z outputs = block( 2025-09-07T07:05:32.5391119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5391482Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5391883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5392271Z return func(*args, **kwargs) 2025-09-07T07:05:32.5392648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5393067Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5393489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5393891Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5394251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5394719Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5394962Z 2025-09-07T07:05:32.5395071Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5395445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5395778Z return mod(**inputs) 2025-09-07T07:05:32.5396146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5396580Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5396973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5397354Z outputs = block( 2025-09-07T07:05:32.5397682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5398059Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5398442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5398883Z return func(*args, **kwargs) 2025-09-07T07:05:32.5399263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5399683Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5400104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5400565Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5400939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5401348Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5401522Z 2025-09-07T07:05:32.5401636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5402002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5402327Z return mod(**inputs) 2025-09-07T07:05:32.5402692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5403097Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5403493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5403887Z outputs = block( 2025-09-07T07:05:32.5404205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5404567Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5404957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5405341Z return func(*args, **kwargs) 2025-09-07T07:05:32.5405779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-09-07T07:05:32.5406236Z hidden_states = residual + feed_forward_hidden_states 2025-09-07T07:05:32.5406416Z 2025-09-07T07:05:32.5406528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5406928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5407246Z return mod(**inputs) 2025-09-07T07:05:32.5407607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5408004Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5408395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5408766Z outputs = block( 2025-09-07T07:05:32.5409082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5409445Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5409820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5410195Z return func(*args, **kwargs) 2025-09-07T07:05:32.5410562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5410985Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5411374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5411748Z return func(*args, **kwargs) 2025-09-07T07:05:32.5412112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5412607Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5413058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5413445Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5413610Z 2025-09-07T07:05:32.5413698Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5413912Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5414113Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5414334Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5414564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5414929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5415247Z return mod(**inputs) 2025-09-07T07:05:32.5415611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5416007Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5416397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5416766Z outputs = block( 2025-09-07T07:05:32.5417082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5417447Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5417818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5418184Z return func(*args, **kwargs) 2025-09-07T07:05:32.5418539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5418926Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5419329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5419872Z return func(*args, **kwargs) 2025-09-07T07:05:32.5420287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5420683Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5421129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5421613Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5421793Z 2025-09-07T07:05:32.5422070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5422431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5422749Z return mod(**inputs) 2025-09-07T07:05:32.5423110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5423518Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5423933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5424336Z outputs = block( 2025-09-07T07:05:32.5424681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5425110Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5425533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5425609Z return func(*args, **kwargs) 2025-09-07T07:05:32.5425947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5426050Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5426320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5426397Z return func(*args, **kwargs) 2025-09-07T07:05:32.5426665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5426775Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5427096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5427249Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5427253Z 2025-09-07T07:05:32.5427357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5427561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5427640Z return mod(**inputs) 2025-09-07T07:05:32.5427898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5427998Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5428248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5428314Z outputs = block( 2025-09-07T07:05:32.5428550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5428632Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5428883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5428952Z return func(*args, **kwargs) 2025-09-07T07:05:32.5429235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5429323Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5429560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5429656Z return func(*args, **kwargs) 2025-09-07T07:05:32.5429901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5429991Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5430211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5430331Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5430335Z 2025-09-07T07:05:32.5430445Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5430642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5430718Z return mod(**inputs) 2025-09-07T07:05:32.5430969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5431059Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5431301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5431363Z outputs = block( 2025-09-07T07:05:32.5431596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5431694Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5431936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5432004Z return func(*args, **kwargs) 2025-09-07T07:05:32.5432248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5432359Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5432601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5432686Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5432898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5433013Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5433024Z 2025-09-07T07:05:32.5433128Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5433342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5433413Z return mod(**inputs) 2025-09-07T07:05:32.5433667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5433754Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5434002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5434066Z outputs = block( 2025-09-07T07:05:32.5434290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5434368Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5434619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5434692Z return func(*args, **kwargs) 2025-09-07T07:05:32.5434939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5435046Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5435308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5435395Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5435624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5435814Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5435817Z 2025-09-07T07:05:32.5435921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5436119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5436192Z return mod(**inputs) 2025-09-07T07:05:32.5436444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5436531Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5436777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5436839Z outputs = block( 2025-09-07T07:05:32.5437069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5437147Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5437394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5437461Z return func(*args, **kwargs) 2025-09-07T07:05:32.5437719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5437829Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5438074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5438168Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5438382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5438506Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5438511Z 2025-09-07T07:05:32.5438613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5438811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5438883Z return mod(**inputs) 2025-09-07T07:05:32.5439132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5439241Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5439489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5439551Z outputs = block( 2025-09-07T07:05:32.5439780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5439859Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5440116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5440182Z return func(*args, **kwargs) 2025-09-07T07:05:32.5440421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5440516Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5440761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5440838Z return func(*args, **kwargs) 2025-09-07T07:05:32.5441068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5441268Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5441479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5441614Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5441617Z 2025-09-07T07:05:32.5441703Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5441782Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5441865Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5441941Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5442041Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5442247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5442310Z return mod(**inputs) 2025-09-07T07:05:32.5442563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5442645Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5442883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5442953Z outputs = block( 2025-09-07T07:05:32.5443165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5443248Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5443483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5443581Z return func(*args, **kwargs) 2025-09-07T07:05:32.5443827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5443912Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5444160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5444228Z return func(*args, **kwargs) 2025-09-07T07:05:32.5444481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5444577Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5444871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5445012Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5445035Z 2025-09-07T07:05:32.5445137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5445342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5445406Z return mod(**inputs) 2025-09-07T07:05:32.5445664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5445745Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5445992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5446063Z outputs = block( 2025-09-07T07:05:32.5446288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5446375Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5446624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5446696Z return func(*args, **kwargs) 2025-09-07T07:05:32.5446957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5447044Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5447322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5447391Z return func(*args, **kwargs) 2025-09-07T07:05:32.5447649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5447754Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5448043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5448162Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5448167Z 2025-09-07T07:05:32.5448267Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5448474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5448539Z return mod(**inputs) 2025-09-07T07:05:32.5448791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5448880Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5449129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5449197Z outputs = block( 2025-09-07T07:05:32.5449416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5449520Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5449764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5449831Z return func(*args, **kwargs) 2025-09-07T07:05:32.5450081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5450164Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5450401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5450473Z return func(*args, **kwargs) 2025-09-07T07:05:32.5450715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5450801Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5451015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5451139Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5451161Z 2025-09-07T07:05:32.5451263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5451458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5451532Z return mod(**inputs) 2025-09-07T07:05:32.5451783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5451871Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5452116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5452177Z outputs = block( 2025-09-07T07:05:32.5452404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5452483Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5452730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5452801Z return func(*args, **kwargs) 2025-09-07T07:05:32.5453050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5453179Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5453417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5453502Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5453728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5453848Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5453852Z 2025-09-07T07:05:32.5453954Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5454148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5454221Z return mod(**inputs) 2025-09-07T07:05:32.5454462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5454551Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5454793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5454854Z outputs = block( 2025-09-07T07:05:32.5455081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5455158Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5455407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5455495Z return func(*args, **kwargs) 2025-09-07T07:05:32.5455752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5455867Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5456132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5456228Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5456459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5456660Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5456663Z 2025-09-07T07:05:32.5456773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5456988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5457069Z return mod(**inputs) 2025-09-07T07:05:32.5457358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5457451Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5457717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5457792Z outputs = block( 2025-09-07T07:05:32.5458034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5458120Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5458393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5458469Z return func(*args, **kwargs) 2025-09-07T07:05:32.5458747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5458860Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5459135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5459238Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5459493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5459630Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5459634Z 2025-09-07T07:05:32.5459766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5459991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5460071Z return mod(**inputs) 2025-09-07T07:05:32.5460351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5460452Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5460731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5460808Z outputs = block( 2025-09-07T07:05:32.5461051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5461140Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5461415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5461493Z return func(*args, **kwargs) 2025-09-07T07:05:32.5461774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-09-07T07:05:32.5461891Z hidden_states = residual + feed_forward_hidden_states 2025-09-07T07:05:32.5461914Z 2025-09-07T07:05:32.5462028Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5462253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5462325Z return mod(**inputs) 2025-09-07T07:05:32.5462610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5462700Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5462979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5463055Z outputs = block( 2025-09-07T07:05:32.5463300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5463397Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5463665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5463750Z return func(*args, **kwargs) 2025-09-07T07:05:32.5464041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5464136Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5464409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5464485Z return func(*args, **kwargs) 2025-09-07T07:05:32.5464763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5464971Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5465214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5465348Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5465353Z 2025-09-07T07:05:32.5465443Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5465539Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5465708Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5465807Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5465935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5467365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5467462Z return mod(**inputs) 2025-09-07T07:05:32.5467745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5467837Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5468089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5468153Z outputs = block( 2025-09-07T07:05:32.5468390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5468476Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5468730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5468802Z return func(*args, **kwargs) 2025-09-07T07:05:32.5469055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5469152Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5469399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5469476Z return func(*args, **kwargs) 2025-09-07T07:05:32.5469726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5469842Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5470150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5470284Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5470287Z 2025-09-07T07:05:32.5470399Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5470601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5470677Z return mod(**inputs) 2025-09-07T07:05:32.5470936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5471019Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5471278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5471345Z outputs = block( 2025-09-07T07:05:32.5471579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5471679Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5471919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5471997Z return func(*args, **kwargs) 2025-09-07T07:05:32.5472246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5472342Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5472587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5472658Z return func(*args, **kwargs) 2025-09-07T07:05:32.5472913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5473012Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5473316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5473439Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5473442Z 2025-09-07T07:05:32.5473570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5473772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5473836Z return mod(**inputs) 2025-09-07T07:05:32.5474105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5474188Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5474437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5474500Z outputs = block( 2025-09-07T07:05:32.5474718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5474804Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5475039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5475115Z return func(*args, **kwargs) 2025-09-07T07:05:32.5475359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5475454Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5475691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5475759Z return func(*args, **kwargs) 2025-09-07T07:05:32.5476010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5476114Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5476336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5476452Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5476455Z 2025-09-07T07:05:32.5476557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5476763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5476828Z return mod(**inputs) 2025-09-07T07:05:32.5477085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5477168Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5477408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5477481Z outputs = block( 2025-09-07T07:05:32.5477717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5477803Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5478040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5478114Z return func(*args, **kwargs) 2025-09-07T07:05:32.5478356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5478459Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5478709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5478787Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5479014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5479127Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5479131Z 2025-09-07T07:05:32.5479231Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5479435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5479516Z return mod(**inputs) 2025-09-07T07:05:32.5479771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5479865Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5480119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5480181Z outputs = block( 2025-09-07T07:05:32.5480401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5480487Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5480730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5480806Z return func(*args, **kwargs) 2025-09-07T07:05:32.5481052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5481156Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5481411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5481492Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5481717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5481892Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5481914Z 2025-09-07T07:05:32.5482024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5482224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5482288Z return mod(**inputs) 2025-09-07T07:05:32.5482549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5482631Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5482885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5482947Z outputs = block( 2025-09-07T07:05:32.5483173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5483257Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5483500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5483590Z return func(*args, **kwargs) 2025-09-07T07:05:32.5483840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5483941Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5484200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5484285Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5484510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5484625Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5484628Z 2025-09-07T07:05:32.5484736Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5484938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5485004Z return mod(**inputs) 2025-09-07T07:05:32.5485263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5485344Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5485615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5485678Z outputs = block( 2025-09-07T07:05:32.5485916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5486004Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5486248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5486324Z return func(*args, **kwargs) 2025-09-07T07:05:32.5486573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5486661Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5486905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5486972Z return func(*args, **kwargs) 2025-09-07T07:05:32.5487225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5487411Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5487634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5487749Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5487753Z 2025-09-07T07:05:32.5487852Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5487940Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5488018Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5488102Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5488205Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5488409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5488486Z return mod(**inputs) 2025-09-07T07:05:32.5488752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5488846Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5489103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5489166Z outputs = block( 2025-09-07T07:05:32.5489405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5489487Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5489763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5489833Z return func(*args, **kwargs) 2025-09-07T07:05:32.5490089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5490183Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5490420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5490498Z return func(*args, **kwargs) 2025-09-07T07:05:32.5490740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5490842Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5491136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5491270Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5491273Z 2025-09-07T07:05:32.5491383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5491602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5491682Z return mod(**inputs) 2025-09-07T07:05:32.5491937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5492034Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5492295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5492359Z outputs = block( 2025-09-07T07:05:32.5492589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5492672Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5492921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5492993Z return func(*args, **kwargs) 2025-09-07T07:05:32.5493249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5493343Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5493589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5493666Z return func(*args, **kwargs) 2025-09-07T07:05:32.5493916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5494013Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5494337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5494451Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5494455Z 2025-09-07T07:05:32.5494565Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5494769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5494843Z return mod(**inputs) 2025-09-07T07:05:32.5495141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5495224Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5495479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5495541Z outputs = block( 2025-09-07T07:05:32.5495769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5495866Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5496111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5496187Z return func(*args, **kwargs) 2025-09-07T07:05:32.5496439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5496533Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5496779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5496849Z return func(*args, **kwargs) 2025-09-07T07:05:32.5497106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5497192Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5497420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5497538Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5497542Z 2025-09-07T07:05:32.5497654Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5497869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5497936Z return mod(**inputs) 2025-09-07T07:05:32.5498223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5498308Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5498568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5498633Z outputs = block( 2025-09-07T07:05:32.5498856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5498946Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5499194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5499271Z return func(*args, **kwargs) 2025-09-07T07:05:32.5499521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5499625Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5499884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5499966Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5500198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5500334Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5500340Z 2025-09-07T07:05:32.5500453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5500667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5500739Z return mod(**inputs) 2025-09-07T07:05:32.5501017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5501105Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5501376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5501444Z outputs = block( 2025-09-07T07:05:32.5501681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5501773Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5502035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5502136Z return func(*args, **kwargs) 2025-09-07T07:05:32.5502401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5502510Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5502786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5502872Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5503108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5503304Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5503309Z 2025-09-07T07:05:32.5503426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5503641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5503713Z return mod(**inputs) 2025-09-07T07:05:32.5503990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5504078Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5504374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5504442Z outputs = block( 2025-09-07T07:05:32.5504695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5504787Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5505044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5505128Z return func(*args, **kwargs) 2025-09-07T07:05:32.5505392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5505507Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5505860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5505964Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5506208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5506337Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5506341Z 2025-09-07T07:05:32.5506466Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5506685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5506783Z return mod(**inputs) 2025-09-07T07:05:32.5507070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5507162Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5507440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5507511Z outputs = block( 2025-09-07T07:05:32.5507765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5507860Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5508119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5508202Z return func(*args, **kwargs) 2025-09-07T07:05:32.5508467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-09-07T07:05:32.5508593Z hidden_states = residual + feed_forward_hidden_states 2025-09-07T07:05:32.5508617Z 2025-09-07T07:05:32.5508730Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5508943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5509021Z return mod(**inputs) 2025-09-07T07:05:32.5509303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5509399Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5509683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5509750Z outputs = block( 2025-09-07T07:05:32.5510007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5510094Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5510367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5510446Z return func(*args, **kwargs) 2025-09-07T07:05:32.5510734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5510848Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5511124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5511205Z return func(*args, **kwargs) 2025-09-07T07:05:32.5511507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-09-07T07:05:32.5511727Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-09-07T07:05:32.5511972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5512100Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5512104Z 2025-09-07T07:05:32.5512200Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5512287Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5512380Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5512466Z cudagraph partition due to non gpu ops 2025-09-07T07:05:32.5512579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5512808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5512883Z return mod(**inputs) 2025-09-07T07:05:32.5513167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5513257Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5513564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5513640Z outputs = block( 2025-09-07T07:05:32.5513882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5513975Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5514242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5514324Z return func(*args, **kwargs) 2025-09-07T07:05:32.5514605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5514702Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5514974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5515051Z return func(*args, **kwargs) 2025-09-07T07:05:32.5515328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5515456Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5515784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:05:32.5515941Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:05:32.5515944Z 2025-09-07T07:05:32.5516058Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5516288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5516361Z return mod(**inputs) 2025-09-07T07:05:32.5516651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5516742Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5517019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5517097Z outputs = block( 2025-09-07T07:05:32.5517343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5517435Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5517752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5517830Z return func(*args, **kwargs) 2025-09-07T07:05:32.5518134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5518232Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5518505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5518584Z return func(*args, **kwargs) 2025-09-07T07:05:32.5518853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-09-07T07:05:32.5518969Z attn_output, attn_weights = attention_interface( 2025-09-07T07:05:32.5519294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:05:32.5519426Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:05:32.5519430Z 2025-09-07T07:05:32.5519540Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5519952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5520031Z return mod(**inputs) 2025-09-07T07:05:32.5520311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5520457Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5520736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5520814Z outputs = block( 2025-09-07T07:05:32.5521062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5521148Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5521427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5521503Z return func(*args, **kwargs) 2025-09-07T07:05:32.5521785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-09-07T07:05:32.5521882Z attn_output, self_attn_weights = self.attn( 2025-09-07T07:05:32.5522149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5522236Z return func(*args, **kwargs) 2025-09-07T07:05:32.5522543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-09-07T07:05:32.5522645Z attn_output = self.c_proj(attn_output) 2025-09-07T07:05:32.5522890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5523029Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5523033Z 2025-09-07T07:05:32.5523146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5523371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5523453Z return mod(**inputs) 2025-09-07T07:05:32.5523737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5523840Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5524117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5524186Z outputs = block( 2025-09-07T07:05:32.5524438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5524525Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5524832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5524908Z return func(*args, **kwargs) 2025-09-07T07:05:32.5525219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5525335Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5525601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-09-07T07:05:32.5525690Z hidden_states = self.c_fc(hidden_states) 2025-09-07T07:05:32.5525911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5526036Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5526039Z 2025-09-07T07:05:32.5526144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5526348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5526421Z return mod(**inputs) 2025-09-07T07:05:32.5526679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5526767Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5527018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5527101Z outputs = block( 2025-09-07T07:05:32.5527335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5527414Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5527664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5527734Z return func(*args, **kwargs) 2025-09-07T07:05:32.5527995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5528099Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5528349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-09-07T07:05:32.5528437Z hidden_states = self.act(hidden_states) 2025-09-07T07:05:32.5528652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:05:32.5528861Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:05:32.5528865Z 2025-09-07T07:05:32.5528968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5529173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5529248Z return mod(**inputs) 2025-09-07T07:05:32.5529503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-09-07T07:05:32.5529596Z transformer_outputs = self.transformer( 2025-09-07T07:05:32.5529844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-09-07T07:05:32.5529914Z outputs = block( 2025-09-07T07:05:32.5530133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:05:32.5530215Z return super().__call__(*args, **kwargs) 2025-09-07T07:05:32.5530469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:05:32.5530539Z return func(*args, **kwargs) 2025-09-07T07:05:32.5530816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-09-07T07:05:32.5530920Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-09-07T07:05:32.5531181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-09-07T07:05:32.5531279Z hidden_states = self.c_proj(hidden_states) 2025-09-07T07:05:32.5531502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-09-07T07:05:32.5531629Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-09-07T07:05:32.5531635Z 2025-09-07T07:05:32.5531738Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5531950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5532017Z return mod(**inputs) 2025-09-07T07:05:32.5532272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1494, in forward 2025-09-07T07:05:32.5532362Z logits = self.score(hidden_states) 2025-09-07T07:05:32.5532366Z 2025-09-07T07:05:32.5532469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5532679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5532745Z return mod(**inputs) 2025-09-07T07:05:32.5533002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-09-07T07:05:32.5533174Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-09-07T07:05:32.5533179Z 2025-09-07T07:05:32.5533281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:05:32.5533490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:05:32.5533554Z return mod(**inputs) 2025-09-07T07:05:32.5533810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-09-07T07:05:32.5533960Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-09-07T07:05:32.5533965Z 2025-09-07T07:05:46.5908597Z Compilation time (from dynamo_timed): 19.752150542 2025-09-07T07:05:46.5912802Z pass 2025-09-07T07:05:46.5913835Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:46.5914737Z TIMING: _recursive_pre_grad_passes:0.01423 _recursive_joint_graph_passes:0.58737 _recursive_post_grad_passes:0.07845 async_compile.wait:0.74095 code_gen:11.09933 inductor_compile:12.30123 backend_compile:15.56614 gc:0.00179 entire_frame_compile:19.75215 total_wall_time:19.75215 2025-09-07T07:05:46.5917735Z STATS: call_* op count: 1138 | FakeTensorMode.__torch_dispatch__:12455 | FakeTensor.__torch_dispatch__:4284 | ProxyTorchDispatchMode.__torch_dispatch__:4144 2025-09-07T07:05:46.5918304Z Dynamo produced 2 graphs covering 1138 ops with 0 graph breaks (0 unique) 2025-09-07T07:05:49.2371720Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:05:49.2372681Z import pynvml # type: ignore[import] 2025-09-07T07:05:52.0131673Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:05:52.0132898Z from pkg_resources import resource_filename 2025-09-07T07:05:52.7287886Z 2025-09-07T07:05:53.7567172Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:05:53.7572295Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:05:53.7579598Z cpu eval GoogleFnet 2025-09-07T07:05:54.1946251Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:54.3640541Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:05:54.5330085Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:00.2703201Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2703848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2704349Z return mod(**inputs) 2025-09-07T07:06:00.2704916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2705368Z outputs = self.fnet( 2025-09-07T07:06:00.2706039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2706597Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2707030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2707498Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2707903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2708283Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2709003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2709481Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2709941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2710381Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2710821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2711303Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2711493Z 2025-09-07T07:06:00.2711620Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2712030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2712398Z return mod(**inputs) 2025-09-07T07:06:00.2712795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2713315Z outputs = self.fnet( 2025-09-07T07:06:00.2713678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2714072Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2714460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2714858Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2715244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2715625Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2716026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2716439Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2716854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2717256Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2717666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2718142Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2718311Z 2025-09-07T07:06:00.2718426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2718864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2719217Z return mod(**inputs) 2025-09-07T07:06:00.2719817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2720266Z outputs = self.fnet( 2025-09-07T07:06:00.2720656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2721068Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2721465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2721905Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2722296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2722691Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2723105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2723544Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2723977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2724435Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2724852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2725295Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2725464Z 2025-09-07T07:06:00.2725587Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2725978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2726322Z return mod(**inputs) 2025-09-07T07:06:00.2726703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2727106Z outputs = self.fnet( 2025-09-07T07:06:00.2727486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2727889Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2728301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2728752Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2729147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2729544Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2729951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2730389Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2730822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2731243Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2731645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2732066Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2732243Z 2025-09-07T07:06:00.2732357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2732745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2733093Z return mod(**inputs) 2025-09-07T07:06:00.2733494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2733903Z outputs = self.fnet( 2025-09-07T07:06:00.2734312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2734726Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2735111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2735513Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2735892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2736341Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2736734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2737155Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2737618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2738048Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2738467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2738906Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2739074Z 2025-09-07T07:06:00.2739211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2739602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2739955Z return mod(**inputs) 2025-09-07T07:06:00.2740334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2740734Z outputs = self.fnet( 2025-09-07T07:06:00.2741118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2741536Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2741956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2742399Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2742803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2743203Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2743708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2744174Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2744626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2745063Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2745495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2746122Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2746301Z 2025-09-07T07:06:00.2746417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2746818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2747186Z return mod(**inputs) 2025-09-07T07:06:00.2747575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2747953Z outputs = self.fnet( 2025-09-07T07:06:00.2748319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2748740Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2749127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2749528Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2749917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2750288Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2750683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2751102Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2751518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2751910Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2752303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2752720Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2752877Z 2025-09-07T07:06:00.2752994Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2753361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2753694Z return mod(**inputs) 2025-09-07T07:06:00.2754054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2754450Z outputs = self.fnet( 2025-09-07T07:06:00.2754823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2755206Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2755588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2756019Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2756414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2756813Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2757228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2757638Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2758050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2758485Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2758891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2759352Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2759525Z 2025-09-07T07:06:00.2759641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2760039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2760398Z return mod(**inputs) 2025-09-07T07:06:00.2760810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2761254Z outputs = self.fnet( 2025-09-07T07:06:00.2761673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2762123Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2762554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2763015Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2763449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2763876Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2764381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2764856Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2765337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2765785Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2766209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2766657Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2766826Z 2025-09-07T07:06:00.2766938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2767336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2767692Z return mod(**inputs) 2025-09-07T07:06:00.2768082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2768472Z outputs = self.fnet( 2025-09-07T07:06:00.2768833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2769230Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2769619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2770052Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2770440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2770821Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2771224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2771663Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2772094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2772504Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2772919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2773357Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2773527Z 2025-09-07T07:06:00.2773673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2774063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2774406Z return mod(**inputs) 2025-09-07T07:06:00.2774789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2775191Z outputs = self.fnet( 2025-09-07T07:06:00.2775574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2775976Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2776385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2776813Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2777216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2777609Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2778017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2778459Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2778916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2779339Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2779769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2780212Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2780389Z 2025-09-07T07:06:00.2780500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2780886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2781240Z return mod(**inputs) 2025-09-07T07:06:00.2781614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2782015Z outputs = self.fnet( 2025-09-07T07:06:00.2782398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2782808Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2783210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2783622Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2784014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2784427Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2784846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2785277Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2785846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2786298Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2786731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2787203Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2787387Z 2025-09-07T07:06:00.2787501Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2787894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2788288Z return mod(**inputs) 2025-09-07T07:06:00.2788673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2789133Z outputs = self.fnet( 2025-09-07T07:06:00.2789507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 512, in forward 2025-09-07T07:06:00.2789928Z embedding_output = self.embeddings( 2025-09-07T07:06:00.2790343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 142, in forward 2025-09-07T07:06:00.2790764Z embeddings = self.projection(embeddings) 2025-09-07T07:06:00.2790915Z 2025-09-07T07:06:00.2791008Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.2791270Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2791656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2792006Z return mod(**inputs) 2025-09-07T07:06:00.2792387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2792785Z outputs = self.fnet( 2025-09-07T07:06:00.2793165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2793584Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2794005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2794425Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2794839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2795230Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2795645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2796083Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2796510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2796910Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2797299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2797720Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2797878Z 2025-09-07T07:06:00.2797990Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2798350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2798683Z return mod(**inputs) 2025-09-07T07:06:00.2799044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2799453Z outputs = self.fnet( 2025-09-07T07:06:00.2799802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2800190Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2800567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2800976Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2801341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2801693Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2802076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2802489Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2802897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2803295Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2803697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2804112Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2804275Z 2025-09-07T07:06:00.2804378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2804734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2805049Z return mod(**inputs) 2025-09-07T07:06:00.2805399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2805771Z outputs = self.fnet( 2025-09-07T07:06:00.2806125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2806508Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2806880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2807279Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2807641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2808016Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2808395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2808810Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2809223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2809621Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2810015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2810465Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2810649Z 2025-09-07T07:06:00.2810772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2811160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2811517Z return mod(**inputs) 2025-09-07T07:06:00.2811914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2812331Z outputs = self.fnet( 2025-09-07T07:06:00.2812715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2813148Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2813567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2814029Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2814434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2814842Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2815271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2815729Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2816170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2816616Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2817043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2817503Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2817678Z 2025-09-07T07:06:00.2817803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2818218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2818592Z return mod(**inputs) 2025-09-07T07:06:00.2818988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2819414Z outputs = self.fnet( 2025-09-07T07:06:00.2819992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2820417Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2820841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2821278Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2821685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2822086Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2822515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2822961Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2823468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2823911Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2824394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2824889Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2825351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.2825833Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2825989Z 2025-09-07T07:06:00.2826112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2826497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2826858Z return mod(**inputs) 2025-09-07T07:06:00.2827255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2827670Z outputs = self.fnet( 2025-09-07T07:06:00.2828062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2828475Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2828882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2829315Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2829745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2830130Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2830550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2830976Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2831411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2831845Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2832293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2832783Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2833240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.2833697Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.2834136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.2834634Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.2834899Z 2025-09-07T07:06:00.2835019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2835410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2835765Z return mod(**inputs) 2025-09-07T07:06:00.2836152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2836542Z outputs = self.fnet( 2025-09-07T07:06:00.2836893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2837276Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2837655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2838043Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2838409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2838796Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2839190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2839607Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2840013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2840411Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2840823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.2841296Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.2841722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.2842116Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2842260Z 2025-09-07T07:06:00.2842606Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.2842859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2843243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2843571Z return mod(**inputs) 2025-09-07T07:06:00.2843937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2844385Z outputs = self.fnet( 2025-09-07T07:06:00.2844769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2845154Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2845520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2845912Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2846277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2846637Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2847014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2847417Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2847819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2848209Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2848627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2849037Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2849204Z 2025-09-07T07:06:00.2849310Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2849681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2850018Z return mod(**inputs) 2025-09-07T07:06:00.2850395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2850767Z outputs = self.fnet( 2025-09-07T07:06:00.2851130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2851524Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2851913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2852316Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2852698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2853099Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2853496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2853948Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2854405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2854807Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2855201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2855622Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2855782Z 2025-09-07T07:06:00.2855898Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2856258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2856592Z return mod(**inputs) 2025-09-07T07:06:00.2856958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2857339Z outputs = self.fnet( 2025-09-07T07:06:00.2857695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2858085Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2858470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2858928Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2859302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2859672Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2860088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2860534Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2860971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2861395Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2861811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2862253Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2862422Z 2025-09-07T07:06:00.2862544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2862954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2863295Z return mod(**inputs) 2025-09-07T07:06:00.2863675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2864074Z outputs = self.fnet( 2025-09-07T07:06:00.2864450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2864855Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2865250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2865742Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2866163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2866571Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2866982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2867417Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2867876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2868273Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2868687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2869103Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2869272Z 2025-09-07T07:06:00.2869380Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2869755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2870092Z return mod(**inputs) 2025-09-07T07:06:00.2870456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2870830Z outputs = self.fnet( 2025-09-07T07:06:00.2871192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2871581Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2871962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2872354Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2872729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2873093Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2873486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2873910Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2874320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2874727Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2875149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2875611Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2876042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.2876435Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2876585Z 2025-09-07T07:06:00.2876693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2877074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2877429Z return mod(**inputs) 2025-09-07T07:06:00.2877786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2878171Z outputs = self.fnet( 2025-09-07T07:06:00.2878538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2878930Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2879309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2879706Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2880079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2880454Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2880846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2881246Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2881652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2882058Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2882498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2882961Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2883396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.2883833Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.2884211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.2884668Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.2884910Z 2025-09-07T07:06:00.2885023Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2885387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2885721Z return mod(**inputs) 2025-09-07T07:06:00.2886091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2886496Z outputs = self.fnet( 2025-09-07T07:06:00.2886881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2887265Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2887657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2888086Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2888468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2889004Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2889399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2889818Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2890223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2890621Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2891034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.2891499Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.2891948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.2892374Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2892514Z 2025-09-07T07:06:00.2892606Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.2892845Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2893216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2893548Z return mod(**inputs) 2025-09-07T07:06:00.2893913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2894287Z outputs = self.fnet( 2025-09-07T07:06:00.2894647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2895036Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2895416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2895818Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2896180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2896549Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2896959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2897378Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2897811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2898203Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2898597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2899034Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2899206Z 2025-09-07T07:06:00.2899327Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2899718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2900066Z return mod(**inputs) 2025-09-07T07:06:00.2900451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2900852Z outputs = self.fnet( 2025-09-07T07:06:00.2901239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2901645Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2902050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2902493Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2902891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2903280Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2903708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2904158Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2904610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2905044Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2905465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2905992Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2906181Z 2025-09-07T07:06:00.2906297Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2906735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2907087Z return mod(**inputs) 2025-09-07T07:06:00.2907483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2907896Z outputs = self.fnet( 2025-09-07T07:06:00.2908277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2908669Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2909061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2909463Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2909839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2910210Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2910605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2911015Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2911453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2911849Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2912269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2912721Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2912894Z 2025-09-07T07:06:00.2913000Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2913368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2913697Z return mod(**inputs) 2025-09-07T07:06:00.2914061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2914442Z outputs = self.fnet( 2025-09-07T07:06:00.2914797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2915187Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2915568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2915971Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2916338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2916712Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2917104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2917547Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2917945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2918323Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2918706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2919110Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2919264Z 2025-09-07T07:06:00.2919377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2919840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2920171Z return mod(**inputs) 2025-09-07T07:06:00.2920534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2920924Z outputs = self.fnet( 2025-09-07T07:06:00.2921452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2921823Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2922199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2922591Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2922962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2923327Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2923702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2924098Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2924507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2924910Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2925322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2925769Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2926210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.2926600Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2926737Z 2025-09-07T07:06:00.2926872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2927224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2927546Z return mod(**inputs) 2025-09-07T07:06:00.2927900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2928277Z outputs = self.fnet( 2025-09-07T07:06:00.2928629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2929005Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2929393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2929796Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2930172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2930547Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2930919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2931313Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2931748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2932148Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2932560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2933023Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2933444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.2933870Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.2934254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.2934713Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.2934963Z 2025-09-07T07:06:00.2935069Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2935462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2935790Z return mod(**inputs) 2025-09-07T07:06:00.2936154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2936530Z outputs = self.fnet( 2025-09-07T07:06:00.2936890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2937283Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2937665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2938062Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2938435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2938805Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2939198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2939601Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2940065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2940477Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2940921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.2941397Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.2941836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.2942232Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2942386Z 2025-09-07T07:06:00.2942478Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.2942735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2943123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2943447Z return mod(**inputs) 2025-09-07T07:06:00.2943810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2944204Z outputs = self.fnet( 2025-09-07T07:06:00.2944588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2944999Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2945396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2945915Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2946331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2946740Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2947154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2947564Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2947976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2948367Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2948758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2949172Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2949344Z 2025-09-07T07:06:00.2949452Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2949818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2950172Z return mod(**inputs) 2025-09-07T07:06:00.2950534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2950904Z outputs = self.fnet( 2025-09-07T07:06:00.2951265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2951652Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2952037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2952438Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2952803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2953179Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2953557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2953960Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2954371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2954759Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2955139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2955565Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2955722Z 2025-09-07T07:06:00.2955832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2956180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2956509Z return mod(**inputs) 2025-09-07T07:06:00.2956867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2957260Z outputs = self.fnet( 2025-09-07T07:06:00.2957610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2957988Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2958360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2958756Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2959122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2959478Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2959861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2960282Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2960685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2961074Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2961451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2961868Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2962029Z 2025-09-07T07:06:00.2962134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2962492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2962813Z return mod(**inputs) 2025-09-07T07:06:00.2963155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2963529Z outputs = self.fnet( 2025-09-07T07:06:00.2963906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2964285Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2964649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2965044Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2965418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2965791Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2966185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2966597Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2966998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2967381Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2967761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2968164Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2968316Z 2025-09-07T07:06:00.2968439Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2968796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2969175Z return mod(**inputs) 2025-09-07T07:06:00.2969556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2969954Z outputs = self.fnet( 2025-09-07T07:06:00.2970348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2970774Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2971192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2971627Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2972017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2972412Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2972808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2973209Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2973621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2974050Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2974522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2975006Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2975433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.2975829Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2975977Z 2025-09-07T07:06:00.2976085Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2976453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2976786Z return mod(**inputs) 2025-09-07T07:06:00.2977142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2977514Z outputs = self.fnet( 2025-09-07T07:06:00.2977871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2978287Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2978669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2979127Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2979528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2979912Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2980331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2980749Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2981178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2981611Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2982054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.2982542Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.2982992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.2983452Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.2983871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.2984388Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.2984643Z 2025-09-07T07:06:00.2984766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2985160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2985510Z return mod(**inputs) 2025-09-07T07:06:00.2985975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2986400Z outputs = self.fnet( 2025-09-07T07:06:00.2986807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2987217Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2987619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2988028Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2988408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2988781Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2989168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.2989597Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.2990008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.2990419Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.2990846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.2991318Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.2991766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.2992173Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.2992313Z 2025-09-07T07:06:00.2992405Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.2992647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2993017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2993371Z return mod(**inputs) 2025-09-07T07:06:00.2993734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.2994122Z outputs = self.fnet( 2025-09-07T07:06:00.2994481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.2994873Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.2995275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.2995671Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.2996038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.2996392Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.2996776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.2997181Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.2997583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.2997982Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.2998368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.2998790Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.2998948Z 2025-09-07T07:06:00.2999061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.2999416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.2999734Z return mod(**inputs) 2025-09-07T07:06:00.3000097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3000479Z outputs = self.fnet( 2025-09-07T07:06:00.3000839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3001229Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3001606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3002009Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3002390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3002763Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3003145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3003583Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3003996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3004394Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3004783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3005192Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3005358Z 2025-09-07T07:06:00.3005464Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3005834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3006167Z return mod(**inputs) 2025-09-07T07:06:00.3006520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3006902Z outputs = self.fnet( 2025-09-07T07:06:00.3007288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3007674Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3008059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3008449Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3008845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3009239Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3009655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3010095Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3010523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3010942Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3011350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3011780Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3011948Z 2025-09-07T07:06:00.3012088Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3012474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3012832Z return mod(**inputs) 2025-09-07T07:06:00.3013242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3013657Z outputs = self.fnet( 2025-09-07T07:06:00.3014037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3014454Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3014863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3015289Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3015701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3016099Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3016525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3016979Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3017437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3017864Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3018316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3018762Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3018934Z 2025-09-07T07:06:00.3019057Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3019460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3019972Z return mod(**inputs) 2025-09-07T07:06:00.3020372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3020799Z outputs = self.fnet( 2025-09-07T07:06:00.3021194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3021629Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3022034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3022516Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3022920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3023315Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3023729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3024166Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3024613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3025059Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3025516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3026090Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3026559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3027004Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3027153Z 2025-09-07T07:06:00.3027271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3027689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3028041Z return mod(**inputs) 2025-09-07T07:06:00.3028454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3028864Z outputs = self.fnet( 2025-09-07T07:06:00.3029256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3029641Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3030054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3030486Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3030878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3031252Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3031640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3032044Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3032459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3032870Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3033296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3033775Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3034206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3034643Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3035032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3035491Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3035729Z 2025-09-07T07:06:00.3035838Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3036207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3036538Z return mod(**inputs) 2025-09-07T07:06:00.3036904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3037281Z outputs = self.fnet( 2025-09-07T07:06:00.3037664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3038055Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3038440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3038847Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3039214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3039589Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3039999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3040474Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3040888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3041288Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3041709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3042181Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3042646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3043037Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3043199Z 2025-09-07T07:06:00.3043284Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.3043526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3043889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3044220Z return mod(**inputs) 2025-09-07T07:06:00.3044576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3044956Z outputs = self.fnet( 2025-09-07T07:06:00.3045321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3045713Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3046108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3046538Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3046912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3047281Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3047677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3048107Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3048514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3048905Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3049292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3049708Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3049868Z 2025-09-07T07:06:00.3049973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3050343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3050674Z return mod(**inputs) 2025-09-07T07:06:00.3051033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3051414Z outputs = self.fnet( 2025-09-07T07:06:00.3051798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3052225Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3052608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3053013Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3053390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3053752Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3054133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3054540Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3054952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3055347Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3055731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3056139Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3056295Z 2025-09-07T07:06:00.3056423Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3056782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3057104Z return mod(**inputs) 2025-09-07T07:06:00.3057479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3057870Z outputs = self.fnet( 2025-09-07T07:06:00.3058233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3058618Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3059005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3059401Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3059779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3060174Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3060587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3061008Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3061441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3061857Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3062286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3062727Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3062902Z 2025-09-07T07:06:00.3063014Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3063401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3063754Z return mod(**inputs) 2025-09-07T07:06:00.3064127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3064532Z outputs = self.fnet( 2025-09-07T07:06:00.3064911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3065326Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3065801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3066237Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3066661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3067054Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3067473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3067906Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3068343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3068766Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3069185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3069627Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3069794Z 2025-09-07T07:06:00.3069910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3070301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3070651Z return mod(**inputs) 2025-09-07T07:06:00.3071052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3071456Z outputs = self.fnet( 2025-09-07T07:06:00.3071829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3072264Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3072672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3073098Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3073486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3073879Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3074293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3074718Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3075163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3075593Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3076005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3076456Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3076877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3077284Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3077420Z 2025-09-07T07:06:00.3077524Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3077878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3078198Z return mod(**inputs) 2025-09-07T07:06:00.3078552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3078932Z outputs = self.fnet( 2025-09-07T07:06:00.3079296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3079683Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3080062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3080474Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3080831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3081216Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3081609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3082017Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3082433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3082838Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3083267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3083736Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3084159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3084571Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3084954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3085417Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3085666Z 2025-09-07T07:06:00.3085795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3086167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3086520Z return mod(**inputs) 2025-09-07T07:06:00.3086882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3087253Z outputs = self.fnet( 2025-09-07T07:06:00.3087607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3087997Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3088374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3088778Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3089174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3089563Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3089974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3090391Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3090803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3091212Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3091653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3092128Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3092573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3092978Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3093120Z 2025-09-07T07:06:00.3093211Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.3093457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3093820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3094153Z return mod(**inputs) 2025-09-07T07:06:00.3094517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3094901Z outputs = self.fnet( 2025-09-07T07:06:00.3095263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3095677Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3096065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3096472Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3096850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3097209Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3097604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3098020Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3098439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3098864Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3099273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3099720Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3099896Z 2025-09-07T07:06:00.3100027Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3100418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3100764Z return mod(**inputs) 2025-09-07T07:06:00.3101161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3101568Z outputs = self.fnet( 2025-09-07T07:06:00.3101954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3102366Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3102766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3103191Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3103589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3103991Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3104403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3104843Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3105278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3105828Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3106290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3106734Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3106909Z 2025-09-07T07:06:00.3107021Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3107408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3107766Z return mod(**inputs) 2025-09-07T07:06:00.3108160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3108566Z outputs = self.fnet( 2025-09-07T07:06:00.3108952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3109363Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3109768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3110192Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3110453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3110539Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3110814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3110921Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3111191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3111277Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3111539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3111654Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3111659Z 2025-09-07T07:06:00.3111769Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3111990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3112061Z return mod(**inputs) 2025-09-07T07:06:00.3112325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3112418Z outputs = self.fnet( 2025-09-07T07:06:00.3112682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3112788Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3113054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3113152Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3113391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3113479Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3113751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3113856Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3114135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3114219Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3114487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3114602Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3114605Z 2025-09-07T07:06:00.3114715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3114959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3115032Z return mod(**inputs) 2025-09-07T07:06:00.3115301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3115371Z outputs = self.fnet( 2025-09-07T07:06:00.3115637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3115725Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3115988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3116088Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3116329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3116413Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3116684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3116796Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3117088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3117173Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3117474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3117607Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3117872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3117971Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3117975Z 2025-09-07T07:06:00.3118084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3118310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3118382Z return mod(**inputs) 2025-09-07T07:06:00.3118646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3118724Z outputs = self.fnet( 2025-09-07T07:06:00.3119016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3119102Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3119386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3119479Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3119878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3119972Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3120248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3120341Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3120635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3120720Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3121022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3121160Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3121428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3121555Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3121830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3122025Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3122036Z 2025-09-07T07:06:00.3122147Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3122362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3122442Z return mod(**inputs) 2025-09-07T07:06:00.3122703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3122779Z outputs = self.fnet( 2025-09-07T07:06:00.3123038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3123125Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3123372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3123486Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3123716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3123799Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3124043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3124140Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3124413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3124507Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3124809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3124961Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3125230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3125324Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3125327Z 2025-09-07T07:06:00.3125427Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.3125563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3125790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3125860Z return mod(**inputs) 2025-09-07T07:06:00.3126149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3126229Z outputs = self.fnet( 2025-09-07T07:06:00.3126498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3126585Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3126851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3126943Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3127185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3127263Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3127507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3127606Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3127852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3127929Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3128182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3128292Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3128296Z 2025-09-07T07:06:00.3128396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3128600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3128666Z return mod(**inputs) 2025-09-07T07:06:00.3128909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3128979Z outputs = self.fnet( 2025-09-07T07:06:00.3129222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3129301Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3129549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3129638Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3129884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3129973Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3130218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3130317Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3130564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3130645Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3130885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3130993Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3130998Z 2025-09-07T07:06:00.3131096Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3131302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3131364Z return mod(**inputs) 2025-09-07T07:06:00.3131604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3131693Z outputs = self.fnet( 2025-09-07T07:06:00.3131935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3132028Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3132272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3132363Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3132581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3132661Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3132908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3133002Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3133251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3133331Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3133577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3133685Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3133689Z 2025-09-07T07:06:00.3133790Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3133993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3134076Z return mod(**inputs) 2025-09-07T07:06:00.3134330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3134403Z outputs = self.fnet( 2025-09-07T07:06:00.3134653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3134744Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3134989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3135079Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3135299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3135376Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3135629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3135768Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3136018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3136097Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3136338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3136445Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3136448Z 2025-09-07T07:06:00.3136549Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3136752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3136817Z return mod(**inputs) 2025-09-07T07:06:00.3137064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3137131Z outputs = self.fnet( 2025-09-07T07:06:00.3137373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3137454Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3137712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3137804Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3138039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3138119Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3138369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3138451Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3138719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3138798Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3139072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3139196Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3139439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3139533Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3139537Z 2025-09-07T07:06:00.3139641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3139850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3139915Z return mod(**inputs) 2025-09-07T07:06:00.3140185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3140260Z outputs = self.fnet( 2025-09-07T07:06:00.3140506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3140587Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3140835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3140923Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3141156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3141236Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3141490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3141575Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3141884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3141968Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3142268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3142399Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3142664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3142791Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3143017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3143214Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3143226Z 2025-09-07T07:06:00.3143339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3143553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3143629Z return mod(**inputs) 2025-09-07T07:06:00.3143912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3143989Z outputs = self.fnet( 2025-09-07T07:06:00.3144255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3145529Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3145907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3146008Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3146267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3146359Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3146630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3146732Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3147011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3147095Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3147371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3147511Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3147754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3147862Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3147870Z 2025-09-07T07:06:00.3147960Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.3148061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3148265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3148329Z return mod(**inputs) 2025-09-07T07:06:00.3148573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3148646Z outputs = self.fnet( 2025-09-07T07:06:00.3148888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3148966Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3149207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3149292Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3149538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3149618Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3149867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3149965Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3150213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3150294Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3150535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3150642Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3150647Z 2025-09-07T07:06:00.3150746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3150955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3151019Z return mod(**inputs) 2025-09-07T07:06:00.3151262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3151354Z outputs = self.fnet( 2025-09-07T07:06:00.3151596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3151677Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3151936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3152023Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3152251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3152331Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3152582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3152681Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3152941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3153021Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3153267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3153377Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3153380Z 2025-09-07T07:06:00.3153480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3153683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3153764Z return mod(**inputs) 2025-09-07T07:06:00.3154005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3154077Z outputs = self.fnet( 2025-09-07T07:06:00.3154321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3154403Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3154647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3154739Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3154956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3155033Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3155281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3155396Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3155704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3155784Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3156047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3156155Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3156159Z 2025-09-07T07:06:00.3156261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3156465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3156529Z return mod(**inputs) 2025-09-07T07:06:00.3156794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3156868Z outputs = self.fnet( 2025-09-07T07:06:00.3157114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3157196Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3157457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3157550Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3157789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3157872Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3158119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3158213Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3158464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3158546Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3158791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3158900Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3158905Z 2025-09-07T07:06:00.3159007Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3159214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3159280Z return mod(**inputs) 2025-09-07T07:06:00.3159541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3159604Z outputs = self.fnet( 2025-09-07T07:06:00.3159867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3159946Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3160189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3160278Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3160497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3160573Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3160824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3160906Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3161170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3161249Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3161535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3161673Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3161916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3162006Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3162009Z 2025-09-07T07:06:00.3162108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3162312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3162375Z return mod(**inputs) 2025-09-07T07:06:00.3162617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3162689Z outputs = self.fnet( 2025-09-07T07:06:00.3162938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3163017Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3163259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3163343Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3163582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3163660Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3163920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3164006Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3164271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3164350Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3164629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3164750Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3164998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3165116Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3165329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3165511Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3165524Z 2025-09-07T07:06:00.3165626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3165855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3165930Z return mod(**inputs) 2025-09-07T07:06:00.3166176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3166250Z outputs = self.fnet( 2025-09-07T07:06:00.3166492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3166565Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3166816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3166901Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3167126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3167202Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3167444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3167554Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3167808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3167892Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3168167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3168294Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3168561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3168648Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3168651Z 2025-09-07T07:06:00.3168759Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.3168871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3169098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3169164Z return mod(**inputs) 2025-09-07T07:06:00.3169411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3169486Z outputs = self.fnet( 2025-09-07T07:06:00.3169750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3169835Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3170100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3170200Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3170427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3170507Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3170760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3170859Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3171113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3171192Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3171437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3171547Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3171550Z 2025-09-07T07:06:00.3171651Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3171859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3171943Z return mod(**inputs) 2025-09-07T07:06:00.3172189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3172264Z outputs = self.fnet( 2025-09-07T07:06:00.3172507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3172590Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3172830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3172917Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3173142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3173221Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3173468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3173583Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3173831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3173910Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3174153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3174261Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3174265Z 2025-09-07T07:06:00.3174367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3174571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3174634Z return mod(**inputs) 2025-09-07T07:06:00.3174876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3174952Z outputs = self.fnet( 2025-09-07T07:06:00.3175194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3175272Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3175541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3175633Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3175850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3175943Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3176196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3176290Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3176541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3176621Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3176862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3176970Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3176975Z 2025-09-07T07:06:00.3177078Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3177285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3177351Z return mod(**inputs) 2025-09-07T07:06:00.3177603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3177677Z outputs = self.fnet( 2025-09-07T07:06:00.3177924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3178023Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3178281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3178371Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3178595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3178673Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3178935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3179033Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3179290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3179374Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3179624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3179756Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3179760Z 2025-09-07T07:06:00.3179862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3180075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3180143Z return mod(**inputs) 2025-09-07T07:06:00.3180402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3180468Z outputs = self.fnet( 2025-09-07T07:06:00.3180720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3180802Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3181054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3181149Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3181372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3181450Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3181721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3181807Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3182093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3182176Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3182461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3182586Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3182844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3182936Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3182940Z 2025-09-07T07:06:00.3183044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3183252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3183318Z return mod(**inputs) 2025-09-07T07:06:00.3183569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3183641Z outputs = self.fnet( 2025-09-07T07:06:00.3183891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3198997Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3199459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3199574Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3199813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3199898Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3200160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3200251Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3200523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3200617Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3200902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3201028Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3201380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3201490Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3201718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3201906Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3201913Z 2025-09-07T07:06:00.3202040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3202247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3202323Z return mod(**inputs) 2025-09-07T07:06:00.3202573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3202648Z outputs = self.fnet( 2025-09-07T07:06:00.3202903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3202982Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3203268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3203363Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3203614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3203702Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3203953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3204041Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3204317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3204397Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3204684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3204819Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3205070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3205164Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3205169Z 2025-09-07T07:06:00.3205252Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.3205367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3205570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3205669Z return mod(**inputs) 2025-09-07T07:06:00.3205925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3205993Z outputs = self.fnet( 2025-09-07T07:06:00.3206247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3206324Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3206569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3206665Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3206887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3206976Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3207218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3207348Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3207588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3207669Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3207923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3208029Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3208033Z 2025-09-07T07:06:00.3208147Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3208350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3208415Z return mod(**inputs) 2025-09-07T07:06:00.3208666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3208734Z outputs = self.fnet( 2025-09-07T07:06:00.3208985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3209059Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3209311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3209414Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3209637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3209739Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3209985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3210089Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3210339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3210422Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3210669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3210773Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3210777Z 2025-09-07T07:06:00.3210882Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3211093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3211160Z return mod(**inputs) 2025-09-07T07:06:00.3211411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3211476Z outputs = self.fnet( 2025-09-07T07:06:00.3211718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3211817Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3212057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3212148Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3212367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3212453Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3212702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3212799Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3213048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3213131Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3213379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3213494Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3213497Z 2025-09-07T07:06:00.3213600Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3213806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3213870Z return mod(**inputs) 2025-09-07T07:06:00.3214118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3214184Z outputs = self.fnet( 2025-09-07T07:06:00.3217389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3217486Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3217746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3217843Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3218066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3218148Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3218403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3218501Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3218777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3218861Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3219160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3219265Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3219271Z 2025-09-07T07:06:00.3219387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3219840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3219923Z return mod(**inputs) 2025-09-07T07:06:00.3220174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3220240Z outputs = self.fnet( 2025-09-07T07:06:00.3220501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3220576Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3220844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3220936Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3221233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3221329Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3221593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3221694Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3221977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3222063Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3222382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3222515Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3222803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3222896Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3222943Z 2025-09-07T07:06:00.3223067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3223285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3223354Z return mod(**inputs) 2025-09-07T07:06:00.3223628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3223699Z outputs = self.fnet( 2025-09-07T07:06:00.3223972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3224051Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3224403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3224506Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3224742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3224836Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3225099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3225194Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3225485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3225613Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3225998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3226136Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3226418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3226549Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3226780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3226991Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3226995Z 2025-09-07T07:06:00.3227109Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3227335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3227408Z return mod(**inputs) 2025-09-07T07:06:00.3227685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3227763Z outputs = self.fnet( 2025-09-07T07:06:00.3228020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3228104Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3228341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3228432Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3228643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3228720Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3228964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3229045Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3229303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3229378Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3229655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3229817Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3230083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3230182Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3230186Z 2025-09-07T07:06:00.3230277Z cudagraph partition due to non gpu ops 2025-09-07T07:06:00.3230399Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3230617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3230688Z return mod(**inputs) 2025-09-07T07:06:00.3230994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3231068Z outputs = self.fnet( 2025-09-07T07:06:00.3231341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3231417Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3231678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3231777Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3232012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3232122Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3232389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3232504Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3232769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3232857Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3233126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3233237Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3233241Z 2025-09-07T07:06:00.3233357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3233572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3233644Z return mod(**inputs) 2025-09-07T07:06:00.3233915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3233987Z outputs = self.fnet( 2025-09-07T07:06:00.3234259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3234358Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3234628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3234726Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3234964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3235058Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3235297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3235397Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3235637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3235716Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3235977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3236100Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3236104Z 2025-09-07T07:06:00.3236213Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3236426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3236494Z return mod(**inputs) 2025-09-07T07:06:00.3236755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3236824Z outputs = self.fnet( 2025-09-07T07:06:00.3237076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3237171Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3237428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3237515Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3237736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3237821Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3238065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3238167Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3238430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3238512Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3238766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3238867Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3238871Z 2025-09-07T07:06:00.3238981Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3239177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3239249Z return mod(**inputs) 2025-09-07T07:06:00.3239491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3239555Z outputs = self.fnet( 2025-09-07T07:06:00.3239806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3239878Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3240129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3240214Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3240449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3240536Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3240778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-09-07T07:06:00.3240881Z self_fourier_outputs = self.fourier(hidden_states) 2025-09-07T07:06:00.3241122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-09-07T07:06:00.3241201Z self_outputs = self.self(hidden_states) 2025-09-07T07:06:00.3241457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-09-07T07:06:00.3241554Z outputs = self.fourier_transform(hidden_states).real 2025-09-07T07:06:00.3241560Z 2025-09-07T07:06:00.3241667Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3241862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3241947Z return mod(**inputs) 2025-09-07T07:06:00.3242182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3242245Z outputs = self.fnet( 2025-09-07T07:06:00.3242490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3242561Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3242816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3242901Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3243135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3243222Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3243465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3243555Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3243811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3243887Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3244168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3244303Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3244555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-09-07T07:06:00.3244639Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3244642Z 2025-09-07T07:06:00.3244754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3244950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3245017Z return mod(**inputs) 2025-09-07T07:06:00.3245267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3245333Z outputs = self.fnet( 2025-09-07T07:06:00.3245581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3245652Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3245893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3245984Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3246209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3246317Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3246567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3246659Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3246929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3247007Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3247303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-09-07T07:06:00.3247417Z intermediate_output = self.intermediate(fourier_output) 2025-09-07T07:06:00.3247667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-09-07T07:06:00.3247774Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:00.3247986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-09-07T07:06:00.3248190Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-09-07T07:06:00.3248194Z 2025-09-07T07:06:00.3248297Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3248503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3248574Z return mod(**inputs) 2025-09-07T07:06:00.3248843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-09-07T07:06:00.3248913Z outputs = self.fnet( 2025-09-07T07:06:00.3249207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-09-07T07:06:00.3249291Z encoder_outputs = self.encoder( 2025-09-07T07:06:00.3249539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-09-07T07:06:00.3249634Z layer_outputs = layer_module(hidden_states) 2025-09-07T07:06:00.3249854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:00.3249933Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:00.3250188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-09-07T07:06:00.3250279Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:00.3250557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:00.3250634Z return forward_fn(*input_tensors) 2025-09-07T07:06:00.3250924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-09-07T07:06:00.3251056Z layer_output = self.output(intermediate_output, fourier_output) 2025-09-07T07:06:00.3251301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-09-07T07:06:00.3251391Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3251395Z 2025-09-07T07:06:00.3251495Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3251698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3251762Z return mod(**inputs) 2025-09-07T07:06:00.3252009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-09-07T07:06:00.3252112Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:06:00.3252357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-09-07T07:06:00.3252518Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:06:00.3252761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-09-07T07:06:00.3252861Z hidden_states = self.transform(hidden_states) 2025-09-07T07:06:00.3253101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 321, in forward 2025-09-07T07:06:00.3253181Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:00.3253184Z 2025-09-07T07:06:00.3253292Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3253490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3253562Z return mod(**inputs) 2025-09-07T07:06:00.3253804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-09-07T07:06:00.3253897Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:06:00.3254144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-09-07T07:06:00.3254271Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:06:00.3254522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 341, in forward 2025-09-07T07:06:00.3254608Z hidden_states = self.decoder(hidden_states) 2025-09-07T07:06:00.3254611Z 2025-09-07T07:06:00.3254719Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:00.3254916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:00.3254982Z return mod(**inputs) 2025-09-07T07:06:00.3255262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 686, in forward 2025-09-07T07:06:00.3255459Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:06:00.3255466Z 2025-09-07T07:06:10.9517725Z Compilation time (from dynamo_timed): 15.132767618 2025-09-07T07:06:10.9578967Z pass 2025-09-07T07:06:10.9579357Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:10.9580171Z TIMING: _recursive_pre_grad_passes:0.00632 _recursive_joint_graph_passes:0.4957 _recursive_post_grad_passes:0.07324 async_compile.wait:0.74256 code_gen:10.23976 inductor_compile:11.1597 backend_compile:13.33076 gc:0.00134 entire_frame_compile:15.13277 total_wall_time:15.13277 2025-09-07T07:06:10.9581400Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:7515 | FakeTensor.__torch_dispatch__:3268 | ProxyTorchDispatchMode.__torch_dispatch__:2859 2025-09-07T07:06:10.9581977Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-09-07T07:06:13.5487782Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:06:13.5489497Z import pynvml # type: ignore[import] 2025-09-07T07:06:16.3223216Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:06:16.3224216Z from pkg_resources import resource_filename 2025-09-07T07:06:16.9947327Z 2025-09-07T07:06:18.2566588Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:06:18.2570015Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:06:18.2579559Z cpu eval LayoutLMForMaskedLM 2025-09-07T07:06:18.8813256Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:19.1706931Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:19.4240636Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:28.0940683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.0941394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.0941789Z return mod(**inputs) 2025-09-07T07:06:28.0942271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0942715Z return func(*args, **kwargs) 2025-09-07T07:06:28.0943200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0943609Z return func(*args, **kwargs) 2025-09-07T07:06:28.0944009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0944769Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0945238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.0945905Z outputs = self.layoutlm( 2025-09-07T07:06:28.0946331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0946762Z return func(*args, **kwargs) 2025-09-07T07:06:28.0947183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0947676Z return func(*args, **kwargs) 2025-09-07T07:06:28.0948134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0948540Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0948993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.0949440Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.0949861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0950281Z return func(*args, **kwargs) 2025-09-07T07:06:28.0950719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0951134Z return func(*args, **kwargs) 2025-09-07T07:06:28.0951616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0952037Z return func(*args, **kwargs) 2025-09-07T07:06:28.0952246Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.0952646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0953053Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0953529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.0953987Z layer_outputs = layer_module( 2025-09-07T07:06:28.0954368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.0954773Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.0955189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0955601Z return func(*args, **kwargs) 2025-09-07T07:06:28.0955995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0956517Z return func(*args, **kwargs) 2025-09-07T07:06:28.0957002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0957415Z return func(*args, **kwargs) 2025-09-07T07:06:28.0957850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.0958311Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.0958741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0959156Z return func(*args, **kwargs) 2025-09-07T07:06:28.0959551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0959950Z return func(*args, **kwargs) 2025-09-07T07:06:28.0960348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0960749Z return func(*args, **kwargs) 2025-09-07T07:06:28.0961180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.0961659Z self_outputs = self.self( 2025-09-07T07:06:28.0962059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0962462Z return func(*args, **kwargs) 2025-09-07T07:06:28.0962852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0963259Z return func(*args, **kwargs) 2025-09-07T07:06:28.0963651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0964053Z return func(*args, **kwargs) 2025-09-07T07:06:28.0964556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.0965106Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.0965321Z 2025-09-07T07:06:28.0965442Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.0965814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.0966152Z return mod(**inputs) 2025-09-07T07:06:28.0966517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0966896Z return func(*args, **kwargs) 2025-09-07T07:06:28.0967293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0967665Z return func(*args, **kwargs) 2025-09-07T07:06:28.0968011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0968400Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0968822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.0969251Z outputs = self.layoutlm( 2025-09-07T07:06:28.0969640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0970019Z return func(*args, **kwargs) 2025-09-07T07:06:28.0970410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0970843Z return func(*args, **kwargs) 2025-09-07T07:06:28.0971208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0971593Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0972032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.0972492Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.0972888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0973277Z return func(*args, **kwargs) 2025-09-07T07:06:28.0973642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0974023Z return func(*args, **kwargs) 2025-09-07T07:06:28.0974376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0974757Z return func(*args, **kwargs) 2025-09-07T07:06:28.0974957Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.0975434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0975830Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0976267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.0976730Z layer_outputs = layer_module( 2025-09-07T07:06:28.0977108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.0977547Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.0977959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0978384Z return func(*args, **kwargs) 2025-09-07T07:06:28.0978786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0979198Z return func(*args, **kwargs) 2025-09-07T07:06:28.0979609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0980004Z return func(*args, **kwargs) 2025-09-07T07:06:28.0980421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.0980877Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.0981293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0981692Z return func(*args, **kwargs) 2025-09-07T07:06:28.0982076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0982474Z return func(*args, **kwargs) 2025-09-07T07:06:28.0982882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0983279Z return func(*args, **kwargs) 2025-09-07T07:06:28.0983695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.0984148Z self_outputs = self.self( 2025-09-07T07:06:28.0984557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0984951Z return func(*args, **kwargs) 2025-09-07T07:06:28.0985337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0985796Z return func(*args, **kwargs) 2025-09-07T07:06:28.0986197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0986607Z return func(*args, **kwargs) 2025-09-07T07:06:28.0987046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.0987553Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.0987794Z 2025-09-07T07:06:28.0987918Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.0988326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.0988685Z return mod(**inputs) 2025-09-07T07:06:28.0989073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0989470Z return func(*args, **kwargs) 2025-09-07T07:06:28.0989876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0990297Z return func(*args, **kwargs) 2025-09-07T07:06:28.0990666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0991050Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0991489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.0991925Z outputs = self.layoutlm( 2025-09-07T07:06:28.0992336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0992734Z return func(*args, **kwargs) 2025-09-07T07:06:28.0993120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0993520Z return func(*args, **kwargs) 2025-09-07T07:06:28.0993891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0994279Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0994728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.0995194Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.0995614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0996032Z return func(*args, **kwargs) 2025-09-07T07:06:28.0996439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0996843Z return func(*args, **kwargs) 2025-09-07T07:06:28.0997226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.0997629Z return func(*args, **kwargs) 2025-09-07T07:06:28.0997843Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.0998248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.0998631Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.0999074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.0999517Z layer_outputs = layer_module( 2025-09-07T07:06:28.0999905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1000310Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1000728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1001148Z return func(*args, **kwargs) 2025-09-07T07:06:28.1001539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1001946Z return func(*args, **kwargs) 2025-09-07T07:06:28.1002340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1002743Z return func(*args, **kwargs) 2025-09-07T07:06:28.1003171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1003641Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1004055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1004446Z return func(*args, **kwargs) 2025-09-07T07:06:28.1004833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1005236Z return func(*args, **kwargs) 2025-09-07T07:06:28.1005629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1006041Z return func(*args, **kwargs) 2025-09-07T07:06:28.1006467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1006912Z self_outputs = self.self( 2025-09-07T07:06:28.1007309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1007737Z return func(*args, **kwargs) 2025-09-07T07:06:28.1008126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1008541Z return func(*args, **kwargs) 2025-09-07T07:06:28.1008937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1009351Z return func(*args, **kwargs) 2025-09-07T07:06:28.1009784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1010311Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1010543Z 2025-09-07T07:06:28.1010656Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1010900Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1011171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1011578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1011960Z return mod(**inputs) 2025-09-07T07:06:28.1012356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1012778Z return func(*args, **kwargs) 2025-09-07T07:06:28.1013180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1013592Z return func(*args, **kwargs) 2025-09-07T07:06:28.1013989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1014390Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1014848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1015291Z outputs = self.layoutlm( 2025-09-07T07:06:28.1015692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1016181Z return func(*args, **kwargs) 2025-09-07T07:06:28.1016583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1017005Z return func(*args, **kwargs) 2025-09-07T07:06:28.1017379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1017788Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1018245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1018697Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1019113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1019544Z return func(*args, **kwargs) 2025-09-07T07:06:28.1020271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1020698Z return func(*args, **kwargs) 2025-09-07T07:06:28.1021095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1021511Z return func(*args, **kwargs) 2025-09-07T07:06:28.1021735Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1022135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1022534Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1022984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1023425Z layer_outputs = layer_module( 2025-09-07T07:06:28.1023824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1024308Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1024727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1025128Z return func(*args, **kwargs) 2025-09-07T07:06:28.1025526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1026004Z return func(*args, **kwargs) 2025-09-07T07:06:28.1026412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1026821Z return func(*args, **kwargs) 2025-09-07T07:06:28.1027284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1027755Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1028186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1028596Z return func(*args, **kwargs) 2025-09-07T07:06:28.1028995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1029400Z return func(*args, **kwargs) 2025-09-07T07:06:28.1029796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1030229Z return func(*args, **kwargs) 2025-09-07T07:06:28.1030660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1031172Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1031695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1032148Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1032314Z 2025-09-07T07:06:28.1032431Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1032806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1033141Z return mod(**inputs) 2025-09-07T07:06:28.1033520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1033932Z return func(*args, **kwargs) 2025-09-07T07:06:28.1034333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1034733Z return func(*args, **kwargs) 2025-09-07T07:06:28.1035089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1035511Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1035948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1036382Z outputs = self.layoutlm( 2025-09-07T07:06:28.1036773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1037184Z return func(*args, **kwargs) 2025-09-07T07:06:28.1037566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1037972Z return func(*args, **kwargs) 2025-09-07T07:06:28.1038334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1038713Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1039153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1039610Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1040022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1040392Z return func(*args, **kwargs) 2025-09-07T07:06:28.1040760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1041135Z return func(*args, **kwargs) 2025-09-07T07:06:28.1041500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1041876Z return func(*args, **kwargs) 2025-09-07T07:06:28.1042073Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1042450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1042814Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1043232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1043639Z layer_outputs = layer_module( 2025-09-07T07:06:28.1043999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1044373Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1044762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1045153Z return func(*args, **kwargs) 2025-09-07T07:06:28.1045513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1045891Z return func(*args, **kwargs) 2025-09-07T07:06:28.1046281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1046687Z return func(*args, **kwargs) 2025-09-07T07:06:28.1047107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1047559Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1048002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1048413Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1048888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1049410Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1049902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1050373Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1050526Z 2025-09-07T07:06:28.1050653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1051049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1051395Z return mod(**inputs) 2025-09-07T07:06:28.1051776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1052180Z return func(*args, **kwargs) 2025-09-07T07:06:28.1052570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1052979Z return func(*args, **kwargs) 2025-09-07T07:06:28.1053340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1053725Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1054164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1054617Z outputs = self.layoutlm( 2025-09-07T07:06:28.1054994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1055403Z return func(*args, **kwargs) 2025-09-07T07:06:28.1055786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1056192Z return func(*args, **kwargs) 2025-09-07T07:06:28.1056554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1056927Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1057380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1057821Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1058223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1058618Z return func(*args, **kwargs) 2025-09-07T07:06:28.1059001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1059401Z return func(*args, **kwargs) 2025-09-07T07:06:28.1059788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1060196Z return func(*args, **kwargs) 2025-09-07T07:06:28.1060427Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1060792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1061154Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1061566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1061975Z layer_outputs = layer_module( 2025-09-07T07:06:28.1062329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1062699Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1063084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1063458Z return func(*args, **kwargs) 2025-09-07T07:06:28.1063812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1064189Z return func(*args, **kwargs) 2025-09-07T07:06:28.1064553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1064928Z return func(*args, **kwargs) 2025-09-07T07:06:28.1065350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1065861Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1066308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1066754Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1067243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1067770Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1068222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1068666Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1069052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1069402Z return self.act(input) 2025-09-07T07:06:28.1069540Z 2025-09-07T07:06:28.1069645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1070008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1070334Z return mod(**inputs) 2025-09-07T07:06:28.1070682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1071048Z return func(*args, **kwargs) 2025-09-07T07:06:28.1071407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1071774Z return func(*args, **kwargs) 2025-09-07T07:06:28.1072128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1072487Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1072892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1073313Z outputs = self.layoutlm( 2025-09-07T07:06:28.1073669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1074039Z return func(*args, **kwargs) 2025-09-07T07:06:28.1074394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1074760Z return func(*args, **kwargs) 2025-09-07T07:06:28.1075114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1075466Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1075867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1076264Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1076635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1077018Z return func(*args, **kwargs) 2025-09-07T07:06:28.1077384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1077764Z return func(*args, **kwargs) 2025-09-07T07:06:28.1078130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1078500Z return func(*args, **kwargs) 2025-09-07T07:06:28.1078699Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1079054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1079402Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1079850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1080264Z layer_outputs = layer_module( 2025-09-07T07:06:28.1080623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1080995Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1081395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1081791Z return func(*args, **kwargs) 2025-09-07T07:06:28.1082161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1082535Z return func(*args, **kwargs) 2025-09-07T07:06:28.1082901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1083270Z return func(*args, **kwargs) 2025-09-07T07:06:28.1083668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1084118Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1084560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1084984Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1085453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1085988Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1086500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1086978Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1087135Z 2025-09-07T07:06:28.1087251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1087650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1088003Z return mod(**inputs) 2025-09-07T07:06:28.1088389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1088799Z return func(*args, **kwargs) 2025-09-07T07:06:28.1089181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1089583Z return func(*args, **kwargs) 2025-09-07T07:06:28.1089965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1090364Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1090789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1091223Z outputs = self.layoutlm( 2025-09-07T07:06:28.1091622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1092028Z return func(*args, **kwargs) 2025-09-07T07:06:28.1092420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1092820Z return func(*args, **kwargs) 2025-09-07T07:06:28.1093179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1093556Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1093966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1094378Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1094778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1095203Z return func(*args, **kwargs) 2025-09-07T07:06:28.1095591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1096001Z return func(*args, **kwargs) 2025-09-07T07:06:28.1096380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1096778Z return func(*args, **kwargs) 2025-09-07T07:06:28.1096994Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1097376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1097748Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1098163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1098574Z layer_outputs = layer_module( 2025-09-07T07:06:28.1098933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1099329Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1099711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1100100Z return func(*args, **kwargs) 2025-09-07T07:06:28.1100483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1100900Z return func(*args, **kwargs) 2025-09-07T07:06:28.1101287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1101674Z return func(*args, **kwargs) 2025-09-07T07:06:28.1102148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1102599Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1103013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1103401Z return func(*args, **kwargs) 2025-09-07T07:06:28.1103785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1104180Z return func(*args, **kwargs) 2025-09-07T07:06:28.1104568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1104981Z return func(*args, **kwargs) 2025-09-07T07:06:28.1105396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1105935Z self_outputs = self.self( 2025-09-07T07:06:28.1106333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1106743Z return func(*args, **kwargs) 2025-09-07T07:06:28.1107132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1107534Z return func(*args, **kwargs) 2025-09-07T07:06:28.1107926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1108327Z return func(*args, **kwargs) 2025-09-07T07:06:28.1108754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1109269Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1109499Z 2025-09-07T07:06:28.1109617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1110016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1110427Z return mod(**inputs) 2025-09-07T07:06:28.1110813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1111207Z return func(*args, **kwargs) 2025-09-07T07:06:28.1111597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1111993Z return func(*args, **kwargs) 2025-09-07T07:06:28.1112358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1112741Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1113187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1113625Z outputs = self.layoutlm( 2025-09-07T07:06:28.1114017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1114436Z return func(*args, **kwargs) 2025-09-07T07:06:28.1114812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1115209Z return func(*args, **kwargs) 2025-09-07T07:06:28.1115569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1115960Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1116390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1116824Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1117246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1117647Z return func(*args, **kwargs) 2025-09-07T07:06:28.1118036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1118432Z return func(*args, **kwargs) 2025-09-07T07:06:28.1118819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1119215Z return func(*args, **kwargs) 2025-09-07T07:06:28.1119426Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1119973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1120377Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1120874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1121324Z layer_outputs = layer_module( 2025-09-07T07:06:28.1121705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1122100Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1122513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1122916Z return func(*args, **kwargs) 2025-09-07T07:06:28.1123308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1123718Z return func(*args, **kwargs) 2025-09-07T07:06:28.1124115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1124513Z return func(*args, **kwargs) 2025-09-07T07:06:28.1124913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1125340Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1125756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1126140Z return func(*args, **kwargs) 2025-09-07T07:06:28.1126507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1126937Z return func(*args, **kwargs) 2025-09-07T07:06:28.1127336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1127724Z return func(*args, **kwargs) 2025-09-07T07:06:28.1128124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1128537Z self_outputs = self.self( 2025-09-07T07:06:28.1128937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1129335Z return func(*args, **kwargs) 2025-09-07T07:06:28.1129736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1130186Z return func(*args, **kwargs) 2025-09-07T07:06:28.1130594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1131010Z return func(*args, **kwargs) 2025-09-07T07:06:28.1131437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1131955Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1132176Z 2025-09-07T07:06:28.1132290Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1132697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1133035Z return mod(**inputs) 2025-09-07T07:06:28.1133395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1133804Z return func(*args, **kwargs) 2025-09-07T07:06:28.1134193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1134572Z return func(*args, **kwargs) 2025-09-07T07:06:28.1134936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1135332Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1135784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1136195Z outputs = self.layoutlm( 2025-09-07T07:06:28.1136564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1136934Z return func(*args, **kwargs) 2025-09-07T07:06:28.1137297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1137676Z return func(*args, **kwargs) 2025-09-07T07:06:28.1138016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1138367Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1138779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1139187Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1139565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1139938Z return func(*args, **kwargs) 2025-09-07T07:06:28.1140295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1140688Z return func(*args, **kwargs) 2025-09-07T07:06:28.1141052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1141430Z return func(*args, **kwargs) 2025-09-07T07:06:28.1141625Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1141985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1142349Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1142762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1143179Z layer_outputs = layer_module( 2025-09-07T07:06:28.1143550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1143948Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1144366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1144801Z return func(*args, **kwargs) 2025-09-07T07:06:28.1145194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1145597Z return func(*args, **kwargs) 2025-09-07T07:06:28.1146073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1146496Z return func(*args, **kwargs) 2025-09-07T07:06:28.1146935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1147355Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1147809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1148221Z return func(*args, **kwargs) 2025-09-07T07:06:28.1148610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1149020Z return func(*args, **kwargs) 2025-09-07T07:06:28.1149458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1149870Z return func(*args, **kwargs) 2025-09-07T07:06:28.1150292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1150736Z self_outputs = self.self( 2025-09-07T07:06:28.1151150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1151565Z return func(*args, **kwargs) 2025-09-07T07:06:28.1151965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1152379Z return func(*args, **kwargs) 2025-09-07T07:06:28.1152777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1153172Z return func(*args, **kwargs) 2025-09-07T07:06:28.1153607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1154130Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1154353Z 2025-09-07T07:06:28.1154451Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1154688Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1154944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1155338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1155667Z return mod(**inputs) 2025-09-07T07:06:28.1156018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1156402Z return func(*args, **kwargs) 2025-09-07T07:06:28.1156758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1157130Z return func(*args, **kwargs) 2025-09-07T07:06:28.1157466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1157823Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1158223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1158624Z outputs = self.layoutlm( 2025-09-07T07:06:28.1158984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1159352Z return func(*args, **kwargs) 2025-09-07T07:06:28.1159705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1160093Z return func(*args, **kwargs) 2025-09-07T07:06:28.1160429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1160782Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1161186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1161586Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1161959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1162321Z return func(*args, **kwargs) 2025-09-07T07:06:28.1162704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1163083Z return func(*args, **kwargs) 2025-09-07T07:06:28.1163442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1163806Z return func(*args, **kwargs) 2025-09-07T07:06:28.1164001Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1164348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1164692Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1165092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1165504Z layer_outputs = layer_module( 2025-09-07T07:06:28.1165851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1166215Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1166609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1166989Z return func(*args, **kwargs) 2025-09-07T07:06:28.1167357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1167735Z return func(*args, **kwargs) 2025-09-07T07:06:28.1168096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1168479Z return func(*args, **kwargs) 2025-09-07T07:06:28.1168886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1169316Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1169716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1170088Z return func(*args, **kwargs) 2025-09-07T07:06:28.1170465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1170831Z return func(*args, **kwargs) 2025-09-07T07:06:28.1171187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1171546Z return func(*args, **kwargs) 2025-09-07T07:06:28.1171929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1172384Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1172852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1173259Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1173400Z 2025-09-07T07:06:28.1173508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1173869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1174221Z return mod(**inputs) 2025-09-07T07:06:28.1174571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1174943Z return func(*args, **kwargs) 2025-09-07T07:06:28.1175293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1175660Z return func(*args, **kwargs) 2025-09-07T07:06:28.1176000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1176357Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1176772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1177176Z outputs = self.layoutlm( 2025-09-07T07:06:28.1177536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1177909Z return func(*args, **kwargs) 2025-09-07T07:06:28.1178266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1178626Z return func(*args, **kwargs) 2025-09-07T07:06:28.1178963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1179318Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1179735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1180131Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1180499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1180864Z return func(*args, **kwargs) 2025-09-07T07:06:28.1181225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1181602Z return func(*args, **kwargs) 2025-09-07T07:06:28.1181960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1182339Z return func(*args, **kwargs) 2025-09-07T07:06:28.1182539Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1182899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1183260Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1183675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1184082Z layer_outputs = layer_module( 2025-09-07T07:06:28.1184458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1184833Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1185212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1185612Z return func(*args, **kwargs) 2025-09-07T07:06:28.1186097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1186510Z return func(*args, **kwargs) 2025-09-07T07:06:28.1186904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1187357Z return func(*args, **kwargs) 2025-09-07T07:06:28.1187758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1188191Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1188606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1189110Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1189552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1190049Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1190512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1190938Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1191082Z 2025-09-07T07:06:28.1191190Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1191600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1191942Z return mod(**inputs) 2025-09-07T07:06:28.1192308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1192694Z return func(*args, **kwargs) 2025-09-07T07:06:28.1193062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1193507Z return func(*args, **kwargs) 2025-09-07T07:06:28.1193850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1194214Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1194637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1195053Z outputs = self.layoutlm( 2025-09-07T07:06:28.1195427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1195806Z return func(*args, **kwargs) 2025-09-07T07:06:28.1196176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1196550Z return func(*args, **kwargs) 2025-09-07T07:06:28.1196895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1197260Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1197671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1198081Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1198462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1198830Z return func(*args, **kwargs) 2025-09-07T07:06:28.1199185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1199571Z return func(*args, **kwargs) 2025-09-07T07:06:28.1199919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1200282Z return func(*args, **kwargs) 2025-09-07T07:06:28.1200478Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1200832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1201175Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1201579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1201978Z layer_outputs = layer_module( 2025-09-07T07:06:28.1202323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1202690Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1203619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1204011Z return func(*args, **kwargs) 2025-09-07T07:06:28.1204370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1204735Z return func(*args, **kwargs) 2025-09-07T07:06:28.1205089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1205444Z return func(*args, **kwargs) 2025-09-07T07:06:28.1205842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1206263Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1206701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1207109Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1207541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1208026Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1208471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1208914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1209303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1209643Z return self.act(input) 2025-09-07T07:06:28.1209764Z 2025-09-07T07:06:28.1209871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1210232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1210559Z return mod(**inputs) 2025-09-07T07:06:28.1210903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1211274Z return func(*args, **kwargs) 2025-09-07T07:06:28.1211642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1212024Z return func(*args, **kwargs) 2025-09-07T07:06:28.1212353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1212706Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1213105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1213506Z outputs = self.layoutlm( 2025-09-07T07:06:28.1213865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1214258Z return func(*args, **kwargs) 2025-09-07T07:06:28.1214616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1214980Z return func(*args, **kwargs) 2025-09-07T07:06:28.1215312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1215653Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1216051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1216452Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1216816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1217183Z return func(*args, **kwargs) 2025-09-07T07:06:28.1217531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1217926Z return func(*args, **kwargs) 2025-09-07T07:06:28.1218283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1218651Z return func(*args, **kwargs) 2025-09-07T07:06:28.1218847Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1219195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1219681Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1220101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1220518Z layer_outputs = layer_module( 2025-09-07T07:06:28.1220929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1221333Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1221743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1222154Z return func(*args, **kwargs) 2025-09-07T07:06:28.1222539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1222939Z return func(*args, **kwargs) 2025-09-07T07:06:28.1223325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1223731Z return func(*args, **kwargs) 2025-09-07T07:06:28.1224182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1224639Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1225090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1225528Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1226052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1226593Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1227103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1227555Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1227713Z 2025-09-07T07:06:28.1227831Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1228215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1228535Z return mod(**inputs) 2025-09-07T07:06:28.1228873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1229271Z return func(*args, **kwargs) 2025-09-07T07:06:28.1229622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1229982Z return func(*args, **kwargs) 2025-09-07T07:06:28.1230303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1230646Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1231044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1231431Z outputs = self.layoutlm( 2025-09-07T07:06:28.1231780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1232135Z return func(*args, **kwargs) 2025-09-07T07:06:28.1232479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1232868Z return func(*args, **kwargs) 2025-09-07T07:06:28.1233201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1233554Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1233944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1234345Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1234720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1235082Z return func(*args, **kwargs) 2025-09-07T07:06:28.1235450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1235814Z return func(*args, **kwargs) 2025-09-07T07:06:28.1236177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1236563Z return func(*args, **kwargs) 2025-09-07T07:06:28.1236761Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1237104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1237457Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1237855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1238275Z layer_outputs = layer_module( 2025-09-07T07:06:28.1238627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1239000Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1239383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1239762Z return func(*args, **kwargs) 2025-09-07T07:06:28.1240128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1240494Z return func(*args, **kwargs) 2025-09-07T07:06:28.1240867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1241241Z return func(*args, **kwargs) 2025-09-07T07:06:28.1241635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1242058Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1242449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1242821Z return func(*args, **kwargs) 2025-09-07T07:06:28.1243207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1243584Z return func(*args, **kwargs) 2025-09-07T07:06:28.1243944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1244322Z return func(*args, **kwargs) 2025-09-07T07:06:28.1244718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1245133Z self_outputs = self.self( 2025-09-07T07:06:28.1245505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1245876Z return func(*args, **kwargs) 2025-09-07T07:06:28.1246242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1246619Z return func(*args, **kwargs) 2025-09-07T07:06:28.1246984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1247391Z return func(*args, **kwargs) 2025-09-07T07:06:28.1247794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1248281Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1248490Z 2025-09-07T07:06:28.1248607Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1248979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1249303Z return mod(**inputs) 2025-09-07T07:06:28.1249680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1250060Z return func(*args, **kwargs) 2025-09-07T07:06:28.1250434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1250796Z return func(*args, **kwargs) 2025-09-07T07:06:28.1251128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1251481Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1251884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1252283Z outputs = self.layoutlm( 2025-09-07T07:06:28.1252650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1253021Z return func(*args, **kwargs) 2025-09-07T07:06:28.1253381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1253751Z return func(*args, **kwargs) 2025-09-07T07:06:28.1254084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1254433Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1254830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1255233Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1255611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1256019Z return func(*args, **kwargs) 2025-09-07T07:06:28.1256420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1256824Z return func(*args, **kwargs) 2025-09-07T07:06:28.1257217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1257614Z return func(*args, **kwargs) 2025-09-07T07:06:28.1257812Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1258172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1258541Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1258941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1259334Z layer_outputs = layer_module( 2025-09-07T07:06:28.1259690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1260060Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1260446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1260820Z return func(*args, **kwargs) 2025-09-07T07:06:28.1261184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1261582Z return func(*args, **kwargs) 2025-09-07T07:06:28.1261946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1262320Z return func(*args, **kwargs) 2025-09-07T07:06:28.1262707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1263130Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1263526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1263903Z return func(*args, **kwargs) 2025-09-07T07:06:28.1264284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1264655Z return func(*args, **kwargs) 2025-09-07T07:06:28.1265022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1265404Z return func(*args, **kwargs) 2025-09-07T07:06:28.1265914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1266363Z self_outputs = self.self( 2025-09-07T07:06:28.1266752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1267166Z return func(*args, **kwargs) 2025-09-07T07:06:28.1267592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1268007Z return func(*args, **kwargs) 2025-09-07T07:06:28.1268404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1268830Z return func(*args, **kwargs) 2025-09-07T07:06:28.1269276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1269807Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1270026Z 2025-09-07T07:06:28.1270151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1270554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1270920Z return mod(**inputs) 2025-09-07T07:06:28.1271316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1271735Z return func(*args, **kwargs) 2025-09-07T07:06:28.1272144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1272583Z return func(*args, **kwargs) 2025-09-07T07:06:28.1272958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1273364Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1273812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1274251Z outputs = self.layoutlm( 2025-09-07T07:06:28.1274652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1275069Z return func(*args, **kwargs) 2025-09-07T07:06:28.1275472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1275882Z return func(*args, **kwargs) 2025-09-07T07:06:28.1276248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1276642Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1277112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1277561Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1277932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1278313Z return func(*args, **kwargs) 2025-09-07T07:06:28.1278682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1279060Z return func(*args, **kwargs) 2025-09-07T07:06:28.1279424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1279811Z return func(*args, **kwargs) 2025-09-07T07:06:28.1280014Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1280374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1280740Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1281152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1281567Z layer_outputs = layer_module( 2025-09-07T07:06:28.1281924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1282302Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1282708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1283077Z return func(*args, **kwargs) 2025-09-07T07:06:28.1283446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1283844Z return func(*args, **kwargs) 2025-09-07T07:06:28.1284232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1284631Z return func(*args, **kwargs) 2025-09-07T07:06:28.1285059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1285516Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1285938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1286355Z return func(*args, **kwargs) 2025-09-07T07:06:28.1286744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1287138Z return func(*args, **kwargs) 2025-09-07T07:06:28.1287498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1287899Z return func(*args, **kwargs) 2025-09-07T07:06:28.1288312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1288751Z self_outputs = self.self( 2025-09-07T07:06:28.1289144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1289543Z return func(*args, **kwargs) 2025-09-07T07:06:28.1289938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1290341Z return func(*args, **kwargs) 2025-09-07T07:06:28.1290737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1291113Z return func(*args, **kwargs) 2025-09-07T07:06:28.1291514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1292036Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1292254Z 2025-09-07T07:06:28.1292344Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1292578Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1292841Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1293238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1293583Z return mod(**inputs) 2025-09-07T07:06:28.1293965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1294364Z return func(*args, **kwargs) 2025-09-07T07:06:28.1294770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1295169Z return func(*args, **kwargs) 2025-09-07T07:06:28.1295525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1295913Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1296353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1296785Z outputs = self.layoutlm( 2025-09-07T07:06:28.1297164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1297563Z return func(*args, **kwargs) 2025-09-07T07:06:28.1297971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1298370Z return func(*args, **kwargs) 2025-09-07T07:06:28.1298737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1299118Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1299562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1300004Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1300409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1300812Z return func(*args, **kwargs) 2025-09-07T07:06:28.1301196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1301599Z return func(*args, **kwargs) 2025-09-07T07:06:28.1301990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1302394Z return func(*args, **kwargs) 2025-09-07T07:06:28.1302598Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1303012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1303395Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1303828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1304260Z layer_outputs = layer_module( 2025-09-07T07:06:28.1304626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1305027Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1305444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1305939Z return func(*args, **kwargs) 2025-09-07T07:06:28.1306340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1306751Z return func(*args, **kwargs) 2025-09-07T07:06:28.1307150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1307593Z return func(*args, **kwargs) 2025-09-07T07:06:28.1308016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1308458Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1308873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1309273Z return func(*args, **kwargs) 2025-09-07T07:06:28.1309664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1310056Z return func(*args, **kwargs) 2025-09-07T07:06:28.1310463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1310864Z return func(*args, **kwargs) 2025-09-07T07:06:28.1311286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1311788Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1312287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1312767Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1312929Z 2025-09-07T07:06:28.1313048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1313468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1313832Z return mod(**inputs) 2025-09-07T07:06:28.1314208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1314609Z return func(*args, **kwargs) 2025-09-07T07:06:28.1314999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1315396Z return func(*args, **kwargs) 2025-09-07T07:06:28.1315754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1316137Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1316575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1317006Z outputs = self.layoutlm( 2025-09-07T07:06:28.1317395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1317789Z return func(*args, **kwargs) 2025-09-07T07:06:28.1318175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1318594Z return func(*args, **kwargs) 2025-09-07T07:06:28.1318960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1319333Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1319918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1320364Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1320768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1321169Z return func(*args, **kwargs) 2025-09-07T07:06:28.1321547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1321947Z return func(*args, **kwargs) 2025-09-07T07:06:28.1322334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1322774Z return func(*args, **kwargs) 2025-09-07T07:06:28.1322979Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1323337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1323696Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1324109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1324518Z layer_outputs = layer_module( 2025-09-07T07:06:28.1324872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1325245Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1325664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1326042Z return func(*args, **kwargs) 2025-09-07T07:06:28.1326412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1326783Z return func(*args, **kwargs) 2025-09-07T07:06:28.1327153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1327536Z return func(*args, **kwargs) 2025-09-07T07:06:28.1327936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1328383Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1328807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1329217Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1329656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1330150Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1330602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1331020Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1331170Z 2025-09-07T07:06:28.1331280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1331650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1331984Z return mod(**inputs) 2025-09-07T07:06:28.1332336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1332714Z return func(*args, **kwargs) 2025-09-07T07:06:28.1333078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1333511Z return func(*args, **kwargs) 2025-09-07T07:06:28.1333855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1334221Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1334638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1335047Z outputs = self.layoutlm( 2025-09-07T07:06:28.1335416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1335789Z return func(*args, **kwargs) 2025-09-07T07:06:28.1336156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1336540Z return func(*args, **kwargs) 2025-09-07T07:06:28.1336882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1337262Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1337689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1338111Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1338482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1338851Z return func(*args, **kwargs) 2025-09-07T07:06:28.1339207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1339574Z return func(*args, **kwargs) 2025-09-07T07:06:28.1339956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1340320Z return func(*args, **kwargs) 2025-09-07T07:06:28.1340515Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1340861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1341211Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1341618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1342027Z layer_outputs = layer_module( 2025-09-07T07:06:28.1342374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1342794Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1343183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1343566Z return func(*args, **kwargs) 2025-09-07T07:06:28.1343944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1344319Z return func(*args, **kwargs) 2025-09-07T07:06:28.1344697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1345079Z return func(*args, **kwargs) 2025-09-07T07:06:28.1345485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1345577Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1345956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1346058Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1346401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1346556Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1346867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1347003Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1347235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1347312Z return self.act(input) 2025-09-07T07:06:28.1347317Z 2025-09-07T07:06:28.1347448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1347653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1347729Z return mod(**inputs) 2025-09-07T07:06:28.1347971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1348042Z return func(*args, **kwargs) 2025-09-07T07:06:28.1348288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1348374Z return func(*args, **kwargs) 2025-09-07T07:06:28.1348596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1348672Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1348945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1349017Z outputs = self.layoutlm( 2025-09-07T07:06:28.1349254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1349328Z return func(*args, **kwargs) 2025-09-07T07:06:28.1349583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1349657Z return func(*args, **kwargs) 2025-09-07T07:06:28.1349872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1349950Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1350225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1350298Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1350541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1350608Z return func(*args, **kwargs) 2025-09-07T07:06:28.1350868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1350943Z return func(*args, **kwargs) 2025-09-07T07:06:28.1351179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1351254Z return func(*args, **kwargs) 2025-09-07T07:06:28.1351331Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1351543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1351625Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1351890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1351970Z layer_outputs = layer_module( 2025-09-07T07:06:28.1352189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1352277Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1352515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1352582Z return func(*args, **kwargs) 2025-09-07T07:06:28.1352840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1352908Z return func(*args, **kwargs) 2025-09-07T07:06:28.1353149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1353214Z return func(*args, **kwargs) 2025-09-07T07:06:28.1353482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1353576Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1353836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1353920Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1354219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1354361Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1354645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1354726Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1354730Z 2025-09-07T07:06:28.1354844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1355046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1355119Z return mod(**inputs) 2025-09-07T07:06:28.1355357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1355423Z return func(*args, **kwargs) 2025-09-07T07:06:28.1355680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1355751Z return func(*args, **kwargs) 2025-09-07T07:06:28.1355970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1356045Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1356309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1356384Z outputs = self.layoutlm( 2025-09-07T07:06:28.1356618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1356691Z return func(*args, **kwargs) 2025-09-07T07:06:28.1356945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1357019Z return func(*args, **kwargs) 2025-09-07T07:06:28.1357238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1357313Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1357585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1357660Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1357906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1357972Z return func(*args, **kwargs) 2025-09-07T07:06:28.1358211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1358284Z return func(*args, **kwargs) 2025-09-07T07:06:28.1358523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1358598Z return func(*args, **kwargs) 2025-09-07T07:06:28.1358678Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1358916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1359000Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1359272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1359352Z layer_outputs = layer_module( 2025-09-07T07:06:28.1359576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1359656Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1359905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1359973Z return func(*args, **kwargs) 2025-09-07T07:06:28.1360223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1360292Z return func(*args, **kwargs) 2025-09-07T07:06:28.1360540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1360622Z return func(*args, **kwargs) 2025-09-07T07:06:28.1360892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1360985Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1361226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1361300Z return func(*args, **kwargs) 2025-09-07T07:06:28.1361549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1361615Z return func(*args, **kwargs) 2025-09-07T07:06:28.1361878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1361947Z return func(*args, **kwargs) 2025-09-07T07:06:28.1362229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1362302Z self_outputs = self.self( 2025-09-07T07:06:28.1362538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1362612Z return func(*args, **kwargs) 2025-09-07T07:06:28.1362848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1362941Z return func(*args, **kwargs) 2025-09-07T07:06:28.1363187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1363264Z return func(*args, **kwargs) 2025-09-07T07:06:28.1363548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1363696Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1363700Z 2025-09-07T07:06:28.1363814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1364014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1364086Z return mod(**inputs) 2025-09-07T07:06:28.1364327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1364396Z return func(*args, **kwargs) 2025-09-07T07:06:28.1364646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1364712Z return func(*args, **kwargs) 2025-09-07T07:06:28.1364939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1366013Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1366298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1366378Z outputs = self.layoutlm( 2025-09-07T07:06:28.1366626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1366701Z return func(*args, **kwargs) 2025-09-07T07:06:28.1366948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1367024Z return func(*args, **kwargs) 2025-09-07T07:06:28.1367258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1367332Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1367610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1367709Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1367958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1368028Z return func(*args, **kwargs) 2025-09-07T07:06:28.1368268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1368342Z return func(*args, **kwargs) 2025-09-07T07:06:28.1368586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1368661Z return func(*args, **kwargs) 2025-09-07T07:06:28.1368740Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1368978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1369066Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1369346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1369430Z layer_outputs = layer_module( 2025-09-07T07:06:28.1369658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1369741Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1369992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1370062Z return func(*args, **kwargs) 2025-09-07T07:06:28.1370337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1370405Z return func(*args, **kwargs) 2025-09-07T07:06:28.1370655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1370724Z return func(*args, **kwargs) 2025-09-07T07:06:28.1371005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1371103Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1371359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1371438Z return func(*args, **kwargs) 2025-09-07T07:06:28.1371694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1371765Z return func(*args, **kwargs) 2025-09-07T07:06:28.1372036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1372109Z return func(*args, **kwargs) 2025-09-07T07:06:28.1372402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1372499Z self_outputs = self.self( 2025-09-07T07:06:28.1372757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1372838Z return func(*args, **kwargs) 2025-09-07T07:06:28.1373080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1373154Z return func(*args, **kwargs) 2025-09-07T07:06:28.1373400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1373473Z return func(*args, **kwargs) 2025-09-07T07:06:28.1373748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1373891Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1373896Z 2025-09-07T07:06:28.1374009Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1374233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1374308Z return mod(**inputs) 2025-09-07T07:06:28.1374553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1374621Z return func(*args, **kwargs) 2025-09-07T07:06:28.1374871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1374940Z return func(*args, **kwargs) 2025-09-07T07:06:28.1375167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1375267Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1375541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1375622Z outputs = self.layoutlm( 2025-09-07T07:06:28.1375864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1375942Z return func(*args, **kwargs) 2025-09-07T07:06:28.1376181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1376254Z return func(*args, **kwargs) 2025-09-07T07:06:28.1376491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1376571Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1376861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1376937Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1377200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1377276Z return func(*args, **kwargs) 2025-09-07T07:06:28.1377530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1377611Z return func(*args, **kwargs) 2025-09-07T07:06:28.1377865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1377944Z return func(*args, **kwargs) 2025-09-07T07:06:28.1378027Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1378260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1378349Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1378638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1378742Z layer_outputs = layer_module( 2025-09-07T07:06:28.1378979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1379079Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1379334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1379406Z return func(*args, **kwargs) 2025-09-07T07:06:28.1379669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1379742Z return func(*args, **kwargs) 2025-09-07T07:06:28.1380004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1380077Z return func(*args, **kwargs) 2025-09-07T07:06:28.1380365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1380487Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1380747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1380827Z return func(*args, **kwargs) 2025-09-07T07:06:28.1381092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1381165Z return func(*args, **kwargs) 2025-09-07T07:06:28.1381444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1381519Z return func(*args, **kwargs) 2025-09-07T07:06:28.1381846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1381928Z self_outputs = self.self( 2025-09-07T07:06:28.1382201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1382276Z return func(*args, **kwargs) 2025-09-07T07:06:28.1382539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1382622Z return func(*args, **kwargs) 2025-09-07T07:06:28.1382884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1382964Z return func(*args, **kwargs) 2025-09-07T07:06:28.1383285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1383454Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1383460Z 2025-09-07T07:06:28.1383558Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1383649Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1383775Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1384001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1384074Z return mod(**inputs) 2025-09-07T07:06:28.1384350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1384424Z return func(*args, **kwargs) 2025-09-07T07:06:28.1384696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1384771Z return func(*args, **kwargs) 2025-09-07T07:06:28.1385008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1385100Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1385398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1385501Z outputs = self.layoutlm( 2025-09-07T07:06:28.1385854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1385941Z return func(*args, **kwargs) 2025-09-07T07:06:28.1386209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1386283Z return func(*args, **kwargs) 2025-09-07T07:06:28.1386532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1386620Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1386928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1387010Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1387282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1387388Z return func(*args, **kwargs) 2025-09-07T07:06:28.1387660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1387741Z return func(*args, **kwargs) 2025-09-07T07:06:28.1388010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1388083Z return func(*args, **kwargs) 2025-09-07T07:06:28.1388175Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1388420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1388509Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1388829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1388911Z layer_outputs = layer_module( 2025-09-07T07:06:28.1389169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1389259Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1389543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1389617Z return func(*args, **kwargs) 2025-09-07T07:06:28.1389891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1389998Z return func(*args, **kwargs) 2025-09-07T07:06:28.1390263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1390348Z return func(*args, **kwargs) 2025-09-07T07:06:28.1390646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1390749Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1391013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1391085Z return func(*args, **kwargs) 2025-09-07T07:06:28.1391355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1391430Z return func(*args, **kwargs) 2025-09-07T07:06:28.1391704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1391776Z return func(*args, **kwargs) 2025-09-07T07:06:28.1392076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1392229Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1392547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1392651Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1392655Z 2025-09-07T07:06:28.1392772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1393000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1393075Z return mod(**inputs) 2025-09-07T07:06:28.1393338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1393420Z return func(*args, **kwargs) 2025-09-07T07:06:28.1393684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1393764Z return func(*args, **kwargs) 2025-09-07T07:06:28.1394002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1394104Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1394414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1394492Z outputs = self.layoutlm( 2025-09-07T07:06:28.1394765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1394838Z return func(*args, **kwargs) 2025-09-07T07:06:28.1395111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1395184Z return func(*args, **kwargs) 2025-09-07T07:06:28.1395441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1395527Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1395804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1395885Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1396142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1396213Z return func(*args, **kwargs) 2025-09-07T07:06:28.1396479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1396548Z return func(*args, **kwargs) 2025-09-07T07:06:28.1396832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1396906Z return func(*args, **kwargs) 2025-09-07T07:06:28.1396992Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1397233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1397315Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1397615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1397691Z layer_outputs = layer_module( 2025-09-07T07:06:28.1397932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1398024Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1398283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1398363Z return func(*args, **kwargs) 2025-09-07T07:06:28.1398629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1398705Z return func(*args, **kwargs) 2025-09-07T07:06:28.1398962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1399032Z return func(*args, **kwargs) 2025-09-07T07:06:28.1399324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1399418Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1399705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1399791Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1400117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1400257Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1400546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1400646Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1400678Z 2025-09-07T07:06:28.1400791Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1401015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1401087Z return mod(**inputs) 2025-09-07T07:06:28.1401346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1401427Z return func(*args, **kwargs) 2025-09-07T07:06:28.1401692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1401767Z return func(*args, **kwargs) 2025-09-07T07:06:28.1402003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1402080Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1402364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1402436Z outputs = self.layoutlm( 2025-09-07T07:06:28.1402686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1402755Z return func(*args, **kwargs) 2025-09-07T07:06:28.1402995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1403069Z return func(*args, **kwargs) 2025-09-07T07:06:28.1403303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1403391Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1403685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1403767Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1404029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1404104Z return func(*args, **kwargs) 2025-09-07T07:06:28.1404365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1404436Z return func(*args, **kwargs) 2025-09-07T07:06:28.1404699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1404770Z return func(*args, **kwargs) 2025-09-07T07:06:28.1404855Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1405095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1405176Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1405488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1405567Z layer_outputs = layer_module( 2025-09-07T07:06:28.1405802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1405898Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1406155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1406235Z return func(*args, **kwargs) 2025-09-07T07:06:28.1406493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1406566Z return func(*args, **kwargs) 2025-09-07T07:06:28.1406829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1406903Z return func(*args, **kwargs) 2025-09-07T07:06:28.1407200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1407310Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1407597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1407675Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1407985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1408121Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1408400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1408546Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1408762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1408835Z return self.act(input) 2025-09-07T07:06:28.1408839Z 2025-09-07T07:06:28.1408954Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1409159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1409233Z return mod(**inputs) 2025-09-07T07:06:28.1409478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1409555Z return func(*args, **kwargs) 2025-09-07T07:06:28.1409814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1409885Z return func(*args, **kwargs) 2025-09-07T07:06:28.1410119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1410198Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1410491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1410568Z outputs = self.layoutlm( 2025-09-07T07:06:28.1410831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1410910Z return func(*args, **kwargs) 2025-09-07T07:06:28.1411168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1411246Z return func(*args, **kwargs) 2025-09-07T07:06:28.1411484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1411577Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1411863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1411955Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1412211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1412279Z return func(*args, **kwargs) 2025-09-07T07:06:28.1412522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1412598Z return func(*args, **kwargs) 2025-09-07T07:06:28.1412841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1412918Z return func(*args, **kwargs) 2025-09-07T07:06:28.1412998Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1413230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1413306Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1413580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1413679Z layer_outputs = layer_module( 2025-09-07T07:06:28.1413912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1414007Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1414268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1414339Z return func(*args, **kwargs) 2025-09-07T07:06:28.1414608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1414679Z return func(*args, **kwargs) 2025-09-07T07:06:28.1414965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1415039Z return func(*args, **kwargs) 2025-09-07T07:06:28.1415334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1415429Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1415696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1415783Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1416112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1416284Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1416578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1416668Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1416672Z 2025-09-07T07:06:28.1416793Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1417007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1417085Z return mod(**inputs) 2025-09-07T07:06:28.1417342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1417414Z return func(*args, **kwargs) 2025-09-07T07:06:28.1417677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1417750Z return func(*args, **kwargs) 2025-09-07T07:06:28.1417989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1418072Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1418359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1418463Z outputs = self.layoutlm( 2025-09-07T07:06:28.1418719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1418799Z return func(*args, **kwargs) 2025-09-07T07:06:28.1419055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1419131Z return func(*args, **kwargs) 2025-09-07T07:06:28.1419364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1419447Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1419884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1419969Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1420237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1420357Z return func(*args, **kwargs) 2025-09-07T07:06:28.1420615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1420695Z return func(*args, **kwargs) 2025-09-07T07:06:28.1420960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1421042Z return func(*args, **kwargs) 2025-09-07T07:06:28.1421127Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1421371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1421463Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1421791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1421881Z layer_outputs = layer_module( 2025-09-07T07:06:28.1422130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1422227Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1422495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1422570Z return func(*args, **kwargs) 2025-09-07T07:06:28.1422852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1422953Z return func(*args, **kwargs) 2025-09-07T07:06:28.1423219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1423293Z return func(*args, **kwargs) 2025-09-07T07:06:28.1423581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1423686Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1423944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1424026Z return func(*args, **kwargs) 2025-09-07T07:06:28.1424289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1424362Z return func(*args, **kwargs) 2025-09-07T07:06:28.1424646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1424721Z return func(*args, **kwargs) 2025-09-07T07:06:28.1425029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1425111Z self_outputs = self.self( 2025-09-07T07:06:28.1425420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1425501Z return func(*args, **kwargs) 2025-09-07T07:06:28.1425831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1425921Z return func(*args, **kwargs) 2025-09-07T07:06:28.1426184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1426266Z return func(*args, **kwargs) 2025-09-07T07:06:28.1426568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1426735Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1426739Z 2025-09-07T07:06:28.1426866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1427087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1427195Z return mod(**inputs) 2025-09-07T07:06:28.1427471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1427544Z return func(*args, **kwargs) 2025-09-07T07:06:28.1427809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1427880Z return func(*args, **kwargs) 2025-09-07T07:06:28.1428126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1428209Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1428524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1428601Z outputs = self.layoutlm( 2025-09-07T07:06:28.1428858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1428942Z return func(*args, **kwargs) 2025-09-07T07:06:28.1429199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1429276Z return func(*args, **kwargs) 2025-09-07T07:06:28.1429509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1429590Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1429902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1429984Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1430255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1430328Z return func(*args, **kwargs) 2025-09-07T07:06:28.1430595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1430680Z return func(*args, **kwargs) 2025-09-07T07:06:28.1430938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1431015Z return func(*args, **kwargs) 2025-09-07T07:06:28.1431099Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1431334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1431422Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1431715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1431800Z layer_outputs = layer_module( 2025-09-07T07:06:28.1432040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1432154Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1432412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1432483Z return func(*args, **kwargs) 2025-09-07T07:06:28.1432746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1432816Z return func(*args, **kwargs) 2025-09-07T07:06:28.1433079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1433151Z return func(*args, **kwargs) 2025-09-07T07:06:28.1433440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1433538Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1433798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1433894Z return func(*args, **kwargs) 2025-09-07T07:06:28.1434155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1434225Z return func(*args, **kwargs) 2025-09-07T07:06:28.1434494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1434573Z return func(*args, **kwargs) 2025-09-07T07:06:28.1434863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1434935Z self_outputs = self.self( 2025-09-07T07:06:28.1435205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1435275Z return func(*args, **kwargs) 2025-09-07T07:06:28.1435516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1435593Z return func(*args, **kwargs) 2025-09-07T07:06:28.1435842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1435918Z return func(*args, **kwargs) 2025-09-07T07:06:28.1436206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1436381Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1436386Z 2025-09-07T07:06:28.1436505Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1436724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1436805Z return mod(**inputs) 2025-09-07T07:06:28.1437064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1437140Z return func(*args, **kwargs) 2025-09-07T07:06:28.1437400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1437472Z return func(*args, **kwargs) 2025-09-07T07:06:28.1437711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1437795Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1438093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1438169Z outputs = self.layoutlm( 2025-09-07T07:06:28.1438430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1438530Z return func(*args, **kwargs) 2025-09-07T07:06:28.1438785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1438866Z return func(*args, **kwargs) 2025-09-07T07:06:28.1439100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1439181Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1439484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1439563Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1439830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1439904Z return func(*args, **kwargs) 2025-09-07T07:06:28.1440171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1440251Z return func(*args, **kwargs) 2025-09-07T07:06:28.1440509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1440584Z return func(*args, **kwargs) 2025-09-07T07:06:28.1440664Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1440883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1440964Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1441237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1441316Z layer_outputs = layer_module( 2025-09-07T07:06:28.1441556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1441646Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1441894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1441979Z return func(*args, **kwargs) 2025-09-07T07:06:28.1442244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1442315Z return func(*args, **kwargs) 2025-09-07T07:06:28.1442588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1442658Z return func(*args, **kwargs) 2025-09-07T07:06:28.1442963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1443067Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1443327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1443408Z return func(*args, **kwargs) 2025-09-07T07:06:28.1443673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1443747Z return func(*args, **kwargs) 2025-09-07T07:06:28.1444017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1444086Z return func(*args, **kwargs) 2025-09-07T07:06:28.1444366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1444439Z self_outputs = self.self( 2025-09-07T07:06:28.1444692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1444761Z return func(*args, **kwargs) 2025-09-07T07:06:28.1445008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1445099Z return func(*args, **kwargs) 2025-09-07T07:06:28.1445345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1445420Z return func(*args, **kwargs) 2025-09-07T07:06:28.1445695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1445852Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1445856Z 2025-09-07T07:06:28.1445951Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1446040Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1446161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1446389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1446456Z return mod(**inputs) 2025-09-07T07:06:28.1446709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1446795Z return func(*args, **kwargs) 2025-09-07T07:06:28.1447043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1447110Z return func(*args, **kwargs) 2025-09-07T07:06:28.1447342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1447417Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1447692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1447770Z outputs = self.layoutlm( 2025-09-07T07:06:28.1448065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1448142Z return func(*args, **kwargs) 2025-09-07T07:06:28.1448384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1448454Z return func(*args, **kwargs) 2025-09-07T07:06:28.1448680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1448756Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1449038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1449112Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1449382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1449458Z return func(*args, **kwargs) 2025-09-07T07:06:28.1449706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1449784Z return func(*args, **kwargs) 2025-09-07T07:06:28.1450025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1450091Z return func(*args, **kwargs) 2025-09-07T07:06:28.1450174Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1450392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1450474Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1450749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1450830Z layer_outputs = layer_module( 2025-09-07T07:06:28.1451057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1451138Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1451410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1451483Z return func(*args, **kwargs) 2025-09-07T07:06:28.1451751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1451823Z return func(*args, **kwargs) 2025-09-07T07:06:28.1452083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1452162Z return func(*args, **kwargs) 2025-09-07T07:06:28.1452451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1452549Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1452805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1452879Z return func(*args, **kwargs) 2025-09-07T07:06:28.1453140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1453231Z return func(*args, **kwargs) 2025-09-07T07:06:28.1453496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1453566Z return func(*args, **kwargs) 2025-09-07T07:06:28.1453862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1454005Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1454296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1454409Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1454413Z 2025-09-07T07:06:28.1454529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1454755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1454830Z return mod(**inputs) 2025-09-07T07:06:28.1455091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1455172Z return func(*args, **kwargs) 2025-09-07T07:06:28.1455427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1455508Z return func(*args, **kwargs) 2025-09-07T07:06:28.1455757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1455840Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1456135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1456212Z outputs = self.layoutlm( 2025-09-07T07:06:28.1456477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1456550Z return func(*args, **kwargs) 2025-09-07T07:06:28.1456815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1456885Z return func(*args, **kwargs) 2025-09-07T07:06:28.1457117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1457205Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1457494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1457581Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1457838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1457928Z return func(*args, **kwargs) 2025-09-07T07:06:28.1458202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1458274Z return func(*args, **kwargs) 2025-09-07T07:06:28.1458541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1458613Z return func(*args, **kwargs) 2025-09-07T07:06:28.1458694Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1458939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1459017Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1459322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1459400Z layer_outputs = layer_module( 2025-09-07T07:06:28.1459636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1459749Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1460008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1460089Z return func(*args, **kwargs) 2025-09-07T07:06:28.1460356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1460435Z return func(*args, **kwargs) 2025-09-07T07:06:28.1460705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1460776Z return func(*args, **kwargs) 2025-09-07T07:06:28.1461091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1461186Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1461480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1461563Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1461891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1462036Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1462366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1462477Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1462481Z 2025-09-07T07:06:28.1462596Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1462818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1462892Z return mod(**inputs) 2025-09-07T07:06:28.1463180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1463262Z return func(*args, **kwargs) 2025-09-07T07:06:28.1463535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1463612Z return func(*args, **kwargs) 2025-09-07T07:06:28.1463848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1463929Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1464233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1464312Z outputs = self.layoutlm( 2025-09-07T07:06:28.1464596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1464689Z return func(*args, **kwargs) 2025-09-07T07:06:28.1464976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1465055Z return func(*args, **kwargs) 2025-09-07T07:06:28.1465300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1465389Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1465775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1465872Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1466141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1466217Z return func(*args, **kwargs) 2025-09-07T07:06:28.1466492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1466589Z return func(*args, **kwargs) 2025-09-07T07:06:28.1466878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1466950Z return func(*args, **kwargs) 2025-09-07T07:06:28.1467035Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1467277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1467356Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1467658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1467736Z layer_outputs = layer_module( 2025-09-07T07:06:28.1467994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1468095Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1468355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1468436Z return func(*args, **kwargs) 2025-09-07T07:06:28.1468703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1468776Z return func(*args, **kwargs) 2025-09-07T07:06:28.1469041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1469114Z return func(*args, **kwargs) 2025-09-07T07:06:28.1469414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1469501Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1469771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1469852Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1470157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1470291Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1470563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1470686Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1470902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1470977Z return self.act(input) 2025-09-07T07:06:28.1470990Z 2025-09-07T07:06:28.1471098Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1471309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1471402Z return mod(**inputs) 2025-09-07T07:06:28.1471653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1471730Z return func(*args, **kwargs) 2025-09-07T07:06:28.1471979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1472046Z return func(*args, **kwargs) 2025-09-07T07:06:28.1472278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1472359Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1472650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1472721Z outputs = self.layoutlm( 2025-09-07T07:06:28.1472973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1473066Z return func(*args, **kwargs) 2025-09-07T07:06:28.1473308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1473383Z return func(*args, **kwargs) 2025-09-07T07:06:28.1473603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1473679Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1473960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1474035Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1474310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1474379Z return func(*args, **kwargs) 2025-09-07T07:06:28.1474623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1474700Z return func(*args, **kwargs) 2025-09-07T07:06:28.1474943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1475018Z return func(*args, **kwargs) 2025-09-07T07:06:28.1475097Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1475323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1475398Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1475691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1475774Z layer_outputs = layer_module( 2025-09-07T07:06:28.1476000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1476089Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1476335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1476404Z return func(*args, **kwargs) 2025-09-07T07:06:28.1476654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1476721Z return func(*args, **kwargs) 2025-09-07T07:06:28.1476970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1477041Z return func(*args, **kwargs) 2025-09-07T07:06:28.1477315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1477407Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1477685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1477771Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1478076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1478221Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1478492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1478574Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1478579Z 2025-09-07T07:06:28.1478693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1478896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1478969Z return mod(**inputs) 2025-09-07T07:06:28.1479213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1479297Z return func(*args, **kwargs) 2025-09-07T07:06:28.1479547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1479615Z return func(*args, **kwargs) 2025-09-07T07:06:28.1479842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1479918Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1480201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1480272Z outputs = self.layoutlm( 2025-09-07T07:06:28.1480533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1480613Z return func(*args, **kwargs) 2025-09-07T07:06:28.1480870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1480949Z return func(*args, **kwargs) 2025-09-07T07:06:28.1481179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1481259Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1481554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1481631Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1481911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1481986Z return func(*args, **kwargs) 2025-09-07T07:06:28.1482245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1482327Z return func(*args, **kwargs) 2025-09-07T07:06:28.1482594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1482670Z return func(*args, **kwargs) 2025-09-07T07:06:28.1482750Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1482970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1483053Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1483335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1483419Z layer_outputs = layer_module( 2025-09-07T07:06:28.1483649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1483737Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1483998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1484068Z return func(*args, **kwargs) 2025-09-07T07:06:28.1484315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1484383Z return func(*args, **kwargs) 2025-09-07T07:06:28.1484628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1484696Z return func(*args, **kwargs) 2025-09-07T07:06:28.1484967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1485060Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1485302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1485380Z return func(*args, **kwargs) 2025-09-07T07:06:28.1485623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1485713Z return func(*args, **kwargs) 2025-09-07T07:06:28.1485985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1486057Z return func(*args, **kwargs) 2025-09-07T07:06:28.1486364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1486441Z self_outputs = self.self( 2025-09-07T07:06:28.1486715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1486788Z return func(*args, **kwargs) 2025-09-07T07:06:28.1487071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1487153Z return func(*args, **kwargs) 2025-09-07T07:06:28.1487418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1487508Z return func(*args, **kwargs) 2025-09-07T07:06:28.1487780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1487932Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1487936Z 2025-09-07T07:06:28.1488049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1488286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1488362Z return mod(**inputs) 2025-09-07T07:06:28.1488606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1488678Z return func(*args, **kwargs) 2025-09-07T07:06:28.1488944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1489017Z return func(*args, **kwargs) 2025-09-07T07:06:28.1489258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1489346Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1489626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1489698Z outputs = self.layoutlm( 2025-09-07T07:06:28.1489943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1490019Z return func(*args, **kwargs) 2025-09-07T07:06:28.1490261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1490351Z return func(*args, **kwargs) 2025-09-07T07:06:28.1490574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1490651Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1490936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1491010Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1491261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1491330Z return func(*args, **kwargs) 2025-09-07T07:06:28.1491574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1491651Z return func(*args, **kwargs) 2025-09-07T07:06:28.1491901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1491979Z return func(*args, **kwargs) 2025-09-07T07:06:28.1492077Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1492297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1492379Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1492652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1492731Z layer_outputs = layer_module( 2025-09-07T07:06:28.1492958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1493045Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1493301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1493372Z return func(*args, **kwargs) 2025-09-07T07:06:28.1493621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1493690Z return func(*args, **kwargs) 2025-09-07T07:06:28.1493936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1494004Z return func(*args, **kwargs) 2025-09-07T07:06:28.1494273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1494365Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1494621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1494694Z return func(*args, **kwargs) 2025-09-07T07:06:28.1494937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1495007Z return func(*args, **kwargs) 2025-09-07T07:06:28.1495256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1495327Z return func(*args, **kwargs) 2025-09-07T07:06:28.1495621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1495696Z self_outputs = self.self( 2025-09-07T07:06:28.1495959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1496032Z return func(*args, **kwargs) 2025-09-07T07:06:28.1496288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1496368Z return func(*args, **kwargs) 2025-09-07T07:06:28.1496630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1496727Z return func(*args, **kwargs) 2025-09-07T07:06:28.1497030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1497187Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1497191Z 2025-09-07T07:06:28.1497306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1497508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1497583Z return mod(**inputs) 2025-09-07T07:06:28.1497839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1497910Z return func(*args, **kwargs) 2025-09-07T07:06:28.1498178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1498251Z return func(*args, **kwargs) 2025-09-07T07:06:28.1498512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1498592Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1498885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1498961Z outputs = self.layoutlm( 2025-09-07T07:06:28.1499215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1499298Z return func(*args, **kwargs) 2025-09-07T07:06:28.1499553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1499648Z return func(*args, **kwargs) 2025-09-07T07:06:28.1499884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1499969Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1500266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1500345Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1500616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1500688Z return func(*args, **kwargs) 2025-09-07T07:06:28.1500977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1501060Z return func(*args, **kwargs) 2025-09-07T07:06:28.1501332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1501412Z return func(*args, **kwargs) 2025-09-07T07:06:28.1501498Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1501731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1501821Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1502111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1502196Z layer_outputs = layer_module( 2025-09-07T07:06:28.1502433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1502525Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1502791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1502862Z return func(*args, **kwargs) 2025-09-07T07:06:28.1503135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1503229Z return func(*args, **kwargs) 2025-09-07T07:06:28.1503494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1503565Z return func(*args, **kwargs) 2025-09-07T07:06:28.1503869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1503966Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1504234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1504315Z return func(*args, **kwargs) 2025-09-07T07:06:28.1504571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1504643Z return func(*args, **kwargs) 2025-09-07T07:06:28.1504925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1505016Z return func(*args, **kwargs) 2025-09-07T07:06:28.1505311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1505386Z self_outputs = self.self( 2025-09-07T07:06:28.1505888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1505970Z return func(*args, **kwargs) 2025-09-07T07:06:28.1506241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1506323Z return func(*args, **kwargs) 2025-09-07T07:06:28.1506612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1506692Z return func(*args, **kwargs) 2025-09-07T07:06:28.1506999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1507165Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1507169Z 2025-09-07T07:06:28.1507265Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1507351Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1507473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1507690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1507763Z return mod(**inputs) 2025-09-07T07:06:28.1508059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1508134Z return func(*args, **kwargs) 2025-09-07T07:06:28.1508409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1508485Z return func(*args, **kwargs) 2025-09-07T07:06:28.1508731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1508812Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1509111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1509194Z outputs = self.layoutlm( 2025-09-07T07:06:28.1509463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1509541Z return func(*args, **kwargs) 2025-09-07T07:06:28.1509823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1509898Z return func(*args, **kwargs) 2025-09-07T07:06:28.1510146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1510244Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1510539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1510617Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1510873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1510950Z return func(*args, **kwargs) 2025-09-07T07:06:28.1511206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1511287Z return func(*args, **kwargs) 2025-09-07T07:06:28.1511543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1511615Z return func(*args, **kwargs) 2025-09-07T07:06:28.1511704Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1511934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1512035Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1512323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1512406Z layer_outputs = layer_module( 2025-09-07T07:06:28.1512644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1512728Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1512995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1513066Z return func(*args, **kwargs) 2025-09-07T07:06:28.1513343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1513420Z return func(*args, **kwargs) 2025-09-07T07:06:28.1513681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1513758Z return func(*args, **kwargs) 2025-09-07T07:06:28.1514049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1514146Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1514403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1514494Z return func(*args, **kwargs) 2025-09-07T07:06:28.1514756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1514829Z return func(*args, **kwargs) 2025-09-07T07:06:28.1515094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1515168Z return func(*args, **kwargs) 2025-09-07T07:06:28.1515467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1515610Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1515918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1516019Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1516023Z 2025-09-07T07:06:28.1516140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1516381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1516454Z return mod(**inputs) 2025-09-07T07:06:28.1516711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1516810Z return func(*args, **kwargs) 2025-09-07T07:06:28.1517067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1517147Z return func(*args, **kwargs) 2025-09-07T07:06:28.1517379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1517458Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1517753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1517830Z outputs = self.layoutlm( 2025-09-07T07:06:28.1518094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1518168Z return func(*args, **kwargs) 2025-09-07T07:06:28.1518428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1518520Z return func(*args, **kwargs) 2025-09-07T07:06:28.1518756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1518845Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1519135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1519220Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1519478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1519723Z return func(*args, **kwargs) 2025-09-07T07:06:28.1520043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1520119Z return func(*args, **kwargs) 2025-09-07T07:06:28.1520393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1520468Z return func(*args, **kwargs) 2025-09-07T07:06:28.1520554Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1520808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1520890Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1521203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1521280Z layer_outputs = layer_module( 2025-09-07T07:06:28.1521567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1521658Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1521916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1521998Z return func(*args, **kwargs) 2025-09-07T07:06:28.1522256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1522334Z return func(*args, **kwargs) 2025-09-07T07:06:28.1522598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1522668Z return func(*args, **kwargs) 2025-09-07T07:06:28.1522970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1523067Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1523373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1523458Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1523826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1523972Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1524268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1524367Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1524371Z 2025-09-07T07:06:28.1524485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1524716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1524792Z return mod(**inputs) 2025-09-07T07:06:28.1525064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1525149Z return func(*args, **kwargs) 2025-09-07T07:06:28.1525420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1525529Z return func(*args, **kwargs) 2025-09-07T07:06:28.1525770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1525853Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1526160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1526235Z outputs = self.layoutlm( 2025-09-07T07:06:28.1526512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1526586Z return func(*args, **kwargs) 2025-09-07T07:06:28.1526868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1526951Z return func(*args, **kwargs) 2025-09-07T07:06:28.1527197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1527287Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1527585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1527671Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1527934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1528007Z return func(*args, **kwargs) 2025-09-07T07:06:28.1528297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1528371Z return func(*args, **kwargs) 2025-09-07T07:06:28.1528642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1528718Z return func(*args, **kwargs) 2025-09-07T07:06:28.1528804Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1529055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1529135Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1529438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1529517Z layer_outputs = layer_module( 2025-09-07T07:06:28.1529760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1529856Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1530119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1530203Z return func(*args, **kwargs) 2025-09-07T07:06:28.1530485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1530561Z return func(*args, **kwargs) 2025-09-07T07:06:28.1530834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1530908Z return func(*args, **kwargs) 2025-09-07T07:06:28.1531202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1531288Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1531563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1531643Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1531952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1532084Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1532374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1532501Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1532720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1532793Z return self.act(input) 2025-09-07T07:06:28.1532804Z 2025-09-07T07:06:28.1532911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1533117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1533192Z return mod(**inputs) 2025-09-07T07:06:28.1533454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1533532Z return func(*args, **kwargs) 2025-09-07T07:06:28.1533777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1533848Z return func(*args, **kwargs) 2025-09-07T07:06:28.1534079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1534153Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1534438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1534510Z outputs = self.layoutlm( 2025-09-07T07:06:28.1534770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1534848Z return func(*args, **kwargs) 2025-09-07T07:06:28.1535091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1535167Z return func(*args, **kwargs) 2025-09-07T07:06:28.1535386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1535468Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1535768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1535847Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1536113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1536185Z return func(*args, **kwargs) 2025-09-07T07:06:28.1536449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1536521Z return func(*args, **kwargs) 2025-09-07T07:06:28.1536781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1536880Z return func(*args, **kwargs) 2025-09-07T07:06:28.1536966Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1537210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1537290Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1537585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1537669Z layer_outputs = layer_module( 2025-09-07T07:06:28.1537913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1538008Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1538271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1538345Z return func(*args, **kwargs) 2025-09-07T07:06:28.1538617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1538707Z return func(*args, **kwargs) 2025-09-07T07:06:28.1538973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1539045Z return func(*args, **kwargs) 2025-09-07T07:06:28.1539333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1539433Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1539717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1539808Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1540153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1540315Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1540612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1540701Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1540705Z 2025-09-07T07:06:28.1540823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1541042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1541119Z return mod(**inputs) 2025-09-07T07:06:28.1541399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1541472Z return func(*args, **kwargs) 2025-09-07T07:06:28.1541762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1541835Z return func(*args, **kwargs) 2025-09-07T07:06:28.1542077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1542157Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1542465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1542540Z outputs = self.layoutlm( 2025-09-07T07:06:28.1542809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1542887Z return func(*args, **kwargs) 2025-09-07T07:06:28.1543155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1543234Z return func(*args, **kwargs) 2025-09-07T07:06:28.1543468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1543566Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1543878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1543957Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1544226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1544298Z return func(*args, **kwargs) 2025-09-07T07:06:28.1544566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1544647Z return func(*args, **kwargs) 2025-09-07T07:06:28.1544903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1544983Z return func(*args, **kwargs) 2025-09-07T07:06:28.1545065Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1545299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1545404Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1545755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1545849Z layer_outputs = layer_module( 2025-09-07T07:06:28.1546087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1546180Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1546452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1546526Z return func(*args, **kwargs) 2025-09-07T07:06:28.1546815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1546894Z return func(*args, **kwargs) 2025-09-07T07:06:28.1547183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1547259Z return func(*args, **kwargs) 2025-09-07T07:06:28.1547560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1547657Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1547907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1548009Z return func(*args, **kwargs) 2025-09-07T07:06:28.1548253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1548323Z return func(*args, **kwargs) 2025-09-07T07:06:28.1548574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1548646Z return func(*args, **kwargs) 2025-09-07T07:06:28.1548951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1549028Z self_outputs = self.self( 2025-09-07T07:06:28.1549306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1549377Z return func(*args, **kwargs) 2025-09-07T07:06:28.1549644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1549725Z return func(*args, **kwargs) 2025-09-07T07:06:28.1549983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1550061Z return func(*args, **kwargs) 2025-09-07T07:06:28.1550380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1550543Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1550548Z 2025-09-07T07:06:28.1550668Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1550885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1550964Z return mod(**inputs) 2025-09-07T07:06:28.1551231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1551305Z return func(*args, **kwargs) 2025-09-07T07:06:28.1551589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1551662Z return func(*args, **kwargs) 2025-09-07T07:06:28.1551903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1551984Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1552332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1552407Z outputs = self.layoutlm( 2025-09-07T07:06:28.1552661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1552740Z return func(*args, **kwargs) 2025-09-07T07:06:28.1552992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1553073Z return func(*args, **kwargs) 2025-09-07T07:06:28.1553304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1553422Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1553722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1553803Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1554068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1554141Z return func(*args, **kwargs) 2025-09-07T07:06:28.1554397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1554479Z return func(*args, **kwargs) 2025-09-07T07:06:28.1554753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1554836Z return func(*args, **kwargs) 2025-09-07T07:06:28.1554918Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1555166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1555249Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1555551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1555637Z layer_outputs = layer_module( 2025-09-07T07:06:28.1555878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1555972Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1556237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1556311Z return func(*args, **kwargs) 2025-09-07T07:06:28.1556581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1556655Z return func(*args, **kwargs) 2025-09-07T07:06:28.1556923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1557021Z return func(*args, **kwargs) 2025-09-07T07:06:28.1557316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1557415Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1557676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1557753Z return func(*args, **kwargs) 2025-09-07T07:06:28.1558015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1558086Z return func(*args, **kwargs) 2025-09-07T07:06:28.1558354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1558426Z return func(*args, **kwargs) 2025-09-07T07:06:28.1558730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1558825Z self_outputs = self.self( 2025-09-07T07:06:28.1559094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1559160Z return func(*args, **kwargs) 2025-09-07T07:06:28.1559401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1559477Z return func(*args, **kwargs) 2025-09-07T07:06:28.1559720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1559794Z return func(*args, **kwargs) 2025-09-07T07:06:28.1560081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1560227Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1560236Z 2025-09-07T07:06:28.1560349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1560554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1560629Z return mod(**inputs) 2025-09-07T07:06:28.1560874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1560951Z return func(*args, **kwargs) 2025-09-07T07:06:28.1561216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1561287Z return func(*args, **kwargs) 2025-09-07T07:06:28.1561520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1561596Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1561880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1561952Z outputs = self.layoutlm( 2025-09-07T07:06:28.1562194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1562270Z return func(*args, **kwargs) 2025-09-07T07:06:28.1562514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1562591Z return func(*args, **kwargs) 2025-09-07T07:06:28.1562814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1562890Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1563174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1563268Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1563520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1563589Z return func(*args, **kwargs) 2025-09-07T07:06:28.1563832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1563909Z return func(*args, **kwargs) 2025-09-07T07:06:28.1564149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1564228Z return func(*args, **kwargs) 2025-09-07T07:06:28.1564311Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1564553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1564633Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1564923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1565028Z layer_outputs = layer_module( 2025-09-07T07:06:28.1565268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1565361Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1565628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1565699Z return func(*args, **kwargs) 2025-09-07T07:06:28.1565977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1566048Z return func(*args, **kwargs) 2025-09-07T07:06:28.1566342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1566416Z return func(*args, **kwargs) 2025-09-07T07:06:28.1566724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1566825Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1567095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1567172Z return func(*args, **kwargs) 2025-09-07T07:06:28.1567433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1567509Z return func(*args, **kwargs) 2025-09-07T07:06:28.1567820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1567894Z return func(*args, **kwargs) 2025-09-07T07:06:28.1568201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1568278Z self_outputs = self.self( 2025-09-07T07:06:28.1568552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1568626Z return func(*args, **kwargs) 2025-09-07T07:06:28.1568889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1568967Z return func(*args, **kwargs) 2025-09-07T07:06:28.1569227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1569300Z return func(*args, **kwargs) 2025-09-07T07:06:28.1569577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1569734Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1569745Z 2025-09-07T07:06:28.1569854Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1569939Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1570062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1570280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1570350Z return mod(**inputs) 2025-09-07T07:06:28.1570626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1570697Z return func(*args, **kwargs) 2025-09-07T07:06:28.1570967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1571039Z return func(*args, **kwargs) 2025-09-07T07:06:28.1571279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1571359Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1571657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1571757Z outputs = self.layoutlm( 2025-09-07T07:06:28.1572000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1572073Z return func(*args, **kwargs) 2025-09-07T07:06:28.1572315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1572382Z return func(*args, **kwargs) 2025-09-07T07:06:28.1572612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1572689Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1573003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1573085Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1573350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1573441Z return func(*args, **kwargs) 2025-09-07T07:06:28.1573682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1573754Z return func(*args, **kwargs) 2025-09-07T07:06:28.1573996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1574067Z return func(*args, **kwargs) 2025-09-07T07:06:28.1574162Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1574380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1574462Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1574733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1574814Z layer_outputs = layer_module( 2025-09-07T07:06:28.1575037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1575116Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1575378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1575448Z return func(*args, **kwargs) 2025-09-07T07:06:28.1575713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1575784Z return func(*args, **kwargs) 2025-09-07T07:06:28.1576040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1576119Z return func(*args, **kwargs) 2025-09-07T07:06:28.1576430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1576528Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1576782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1576854Z return func(*args, **kwargs) 2025-09-07T07:06:28.1577116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1577186Z return func(*args, **kwargs) 2025-09-07T07:06:28.1577450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1577521Z return func(*args, **kwargs) 2025-09-07T07:06:28.1577816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1577960Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1578263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1578363Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1578366Z 2025-09-07T07:06:28.1578478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1578701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1578771Z return mod(**inputs) 2025-09-07T07:06:28.1579031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1579110Z return func(*args, **kwargs) 2025-09-07T07:06:28.1579384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1579465Z return func(*args, **kwargs) 2025-09-07T07:06:28.1579702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1579793Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1580084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1580160Z outputs = self.layoutlm( 2025-09-07T07:06:28.1580424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1580494Z return func(*args, **kwargs) 2025-09-07T07:06:28.1580777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1580851Z return func(*args, **kwargs) 2025-09-07T07:06:28.1581084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1581175Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1581465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1581553Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1581811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1581881Z return func(*args, **kwargs) 2025-09-07T07:06:28.1582143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1582215Z return func(*args, **kwargs) 2025-09-07T07:06:28.1582480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1582549Z return func(*args, **kwargs) 2025-09-07T07:06:28.1582633Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1582893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1582973Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1583268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1583344Z layer_outputs = layer_module( 2025-09-07T07:06:28.1583587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1583672Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1583928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1584008Z return func(*args, **kwargs) 2025-09-07T07:06:28.1584265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1584345Z return func(*args, **kwargs) 2025-09-07T07:06:28.1584603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1584693Z return func(*args, **kwargs) 2025-09-07T07:06:28.1584990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1585081Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1585371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1585454Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1585855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1586030Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1586331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1586437Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1586441Z 2025-09-07T07:06:28.1586557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1586787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1586862Z return mod(**inputs) 2025-09-07T07:06:28.1587136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1587219Z return func(*args, **kwargs) 2025-09-07T07:06:28.1587501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1587583Z return func(*args, **kwargs) 2025-09-07T07:06:28.1587821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1587903Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1588208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1588284Z outputs = self.layoutlm( 2025-09-07T07:06:28.1588550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1588622Z return func(*args, **kwargs) 2025-09-07T07:06:28.1588886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1588957Z return func(*args, **kwargs) 2025-09-07T07:06:28.1589191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1589278Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1589570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1589680Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1589956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1590027Z return func(*args, **kwargs) 2025-09-07T07:06:28.1590295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1590365Z return func(*args, **kwargs) 2025-09-07T07:06:28.1590635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1590707Z return func(*args, **kwargs) 2025-09-07T07:06:28.1590790Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1591033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1591113Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1591415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1591509Z layer_outputs = layer_module( 2025-09-07T07:06:28.1591747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1591841Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1592094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1592170Z return func(*args, **kwargs) 2025-09-07T07:06:28.1592415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1592490Z return func(*args, **kwargs) 2025-09-07T07:06:28.1592745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1592814Z return func(*args, **kwargs) 2025-09-07T07:06:28.1593095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1593183Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1593464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1593543Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1593854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1594008Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1594284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1594408Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1594626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1594701Z return self.act(input) 2025-09-07T07:06:28.1594711Z 2025-09-07T07:06:28.1594817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1595030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1595111Z return mod(**inputs) 2025-09-07T07:06:28.1595372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1595452Z return func(*args, **kwargs) 2025-09-07T07:06:28.1595713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1595788Z return func(*args, **kwargs) 2025-09-07T07:06:28.1596032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1596138Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1596453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1596529Z outputs = self.layoutlm( 2025-09-07T07:06:28.1596808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1596889Z return func(*args, **kwargs) 2025-09-07T07:06:28.1597148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1597229Z return func(*args, **kwargs) 2025-09-07T07:06:28.1597465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1597546Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1597850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1597958Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1598222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1598293Z return func(*args, **kwargs) 2025-09-07T07:06:28.1598556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1598628Z return func(*args, **kwargs) 2025-09-07T07:06:28.1598884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1598963Z return func(*args, **kwargs) 2025-09-07T07:06:28.1599045Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1599298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1599382Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1599668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1599755Z layer_outputs = layer_module( 2025-09-07T07:06:28.1599990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1600091Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1600335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1600405Z return func(*args, **kwargs) 2025-09-07T07:06:28.1600673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1600743Z return func(*args, **kwargs) 2025-09-07T07:06:28.1600994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1601064Z return func(*args, **kwargs) 2025-09-07T07:06:28.1601355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1601445Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1601726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1601818Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1602136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1602282Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1602563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1602671Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1602675Z 2025-09-07T07:06:28.1602797Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1603012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1603090Z return mod(**inputs) 2025-09-07T07:06:28.1603346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1603421Z return func(*args, **kwargs) 2025-09-07T07:06:28.1603663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1603731Z return func(*args, **kwargs) 2025-09-07T07:06:28.1603957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1604035Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1604316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1604404Z outputs = self.layoutlm( 2025-09-07T07:06:28.1604646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1604723Z return func(*args, **kwargs) 2025-09-07T07:06:28.1604962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1605036Z return func(*args, **kwargs) 2025-09-07T07:06:28.1605257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1605333Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1605638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1605718Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1605981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1606056Z return func(*args, **kwargs) 2025-09-07T07:06:28.1606322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1606401Z return func(*args, **kwargs) 2025-09-07T07:06:28.1606665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1606742Z return func(*args, **kwargs) 2025-09-07T07:06:28.1606824Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1607083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1607165Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1607457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1607538Z layer_outputs = layer_module( 2025-09-07T07:06:28.1607763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1607850Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1608094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1608162Z return func(*args, **kwargs) 2025-09-07T07:06:28.1608411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1608480Z return func(*args, **kwargs) 2025-09-07T07:06:28.1608737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1608810Z return func(*args, **kwargs) 2025-09-07T07:06:28.1609129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1609229Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1609497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1609576Z return func(*args, **kwargs) 2025-09-07T07:06:28.1609841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1609924Z return func(*args, **kwargs) 2025-09-07T07:06:28.1610189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1610262Z return func(*args, **kwargs) 2025-09-07T07:06:28.1610565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1610640Z self_outputs = self.self( 2025-09-07T07:06:28.1610890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1610978Z return func(*args, **kwargs) 2025-09-07T07:06:28.1611227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1611305Z return func(*args, **kwargs) 2025-09-07T07:06:28.1611553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1611630Z return func(*args, **kwargs) 2025-09-07T07:06:28.1611913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1612079Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1612093Z 2025-09-07T07:06:28.1612204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1612419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1612500Z return mod(**inputs) 2025-09-07T07:06:28.1612767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1612845Z return func(*args, **kwargs) 2025-09-07T07:06:28.1613113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1613184Z return func(*args, **kwargs) 2025-09-07T07:06:28.1613442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1613525Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1613834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1613910Z outputs = self.layoutlm( 2025-09-07T07:06:28.1614178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1614259Z return func(*args, **kwargs) 2025-09-07T07:06:28.1614524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1614602Z return func(*args, **kwargs) 2025-09-07T07:06:28.1614834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1614913Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1615211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1615290Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1615554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1615647Z return func(*args, **kwargs) 2025-09-07T07:06:28.1615910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1615989Z return func(*args, **kwargs) 2025-09-07T07:06:28.1616249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1616327Z return func(*args, **kwargs) 2025-09-07T07:06:28.1616408Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1616652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1616735Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1617032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1617116Z layer_outputs = layer_module( 2025-09-07T07:06:28.1617358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1617471Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1617728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1617798Z return func(*args, **kwargs) 2025-09-07T07:06:28.1618069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1618142Z return func(*args, **kwargs) 2025-09-07T07:06:28.1618407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1618477Z return func(*args, **kwargs) 2025-09-07T07:06:28.1618787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1618887Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1619145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1619225Z return func(*args, **kwargs) 2025-09-07T07:06:28.1619476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1619701Z return func(*args, **kwargs) 2025-09-07T07:06:28.1619966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1620038Z return func(*args, **kwargs) 2025-09-07T07:06:28.1620704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1620788Z self_outputs = self.self( 2025-09-07T07:06:28.1621067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1621143Z return func(*args, **kwargs) 2025-09-07T07:06:28.1621414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1621495Z return func(*args, **kwargs) 2025-09-07T07:06:28.1621764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1621846Z return func(*args, **kwargs) 2025-09-07T07:06:28.1622151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1622308Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1622321Z 2025-09-07T07:06:28.1622439Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1622666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1622797Z return mod(**inputs) 2025-09-07T07:06:28.1623073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1623156Z return func(*args, **kwargs) 2025-09-07T07:06:28.1623426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1623500Z return func(*args, **kwargs) 2025-09-07T07:06:28.1623752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1623834Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1624145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1624223Z outputs = self.layoutlm( 2025-09-07T07:06:28.1624493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1624577Z return func(*args, **kwargs) 2025-09-07T07:06:28.1624868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1624948Z return func(*args, **kwargs) 2025-09-07T07:06:28.1625188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1625272Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1625581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1625718Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1626001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1626102Z return func(*args, **kwargs) 2025-09-07T07:06:28.1626373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1626451Z return func(*args, **kwargs) 2025-09-07T07:06:28.1626721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1626802Z return func(*args, **kwargs) 2025-09-07T07:06:28.1626885Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1627125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1627204Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1627511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1627599Z layer_outputs = layer_module( 2025-09-07T07:06:28.1627844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1627939Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1628196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1628270Z return func(*args, **kwargs) 2025-09-07T07:06:28.1628538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1628608Z return func(*args, **kwargs) 2025-09-07T07:06:28.1628869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1628946Z return func(*args, **kwargs) 2025-09-07T07:06:28.1629236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1629334Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1629591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1629688Z return func(*args, **kwargs) 2025-09-07T07:06:28.1629947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1630027Z return func(*args, **kwargs) 2025-09-07T07:06:28.1630282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1630353Z return func(*args, **kwargs) 2025-09-07T07:06:28.1630648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1630726Z self_outputs = self.self( 2025-09-07T07:06:28.1630990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1631058Z return func(*args, **kwargs) 2025-09-07T07:06:28.1631293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1631388Z return func(*args, **kwargs) 2025-09-07T07:06:28.1631623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1631696Z return func(*args, **kwargs) 2025-09-07T07:06:28.1631962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1632108Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1632119Z 2025-09-07T07:06:28.1632200Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1632279Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1632390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1632608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1632686Z return mod(**inputs) 2025-09-07T07:06:28.1632923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1632992Z return func(*args, **kwargs) 2025-09-07T07:06:28.1633233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1633301Z return func(*args, **kwargs) 2025-09-07T07:06:28.1633522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1633596Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1633881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1633958Z outputs = self.layoutlm( 2025-09-07T07:06:28.1634196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1634271Z return func(*args, **kwargs) 2025-09-07T07:06:28.1634508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1634573Z return func(*args, **kwargs) 2025-09-07T07:06:28.1634799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1634873Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1635148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1635221Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1635456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1635529Z return func(*args, **kwargs) 2025-09-07T07:06:28.1635766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1635860Z return func(*args, **kwargs) 2025-09-07T07:06:28.1636096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1636171Z return func(*args, **kwargs) 2025-09-07T07:06:28.1636246Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1636459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1636539Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1636805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1636880Z layer_outputs = layer_module( 2025-09-07T07:06:28.1637101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1637180Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1637420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1637504Z return func(*args, **kwargs) 2025-09-07T07:06:28.1637745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1637811Z return func(*args, **kwargs) 2025-09-07T07:06:28.1638047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1638121Z return func(*args, **kwargs) 2025-09-07T07:06:28.1638386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1638477Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1638729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1638806Z return func(*args, **kwargs) 2025-09-07T07:06:28.1639049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1639115Z return func(*args, **kwargs) 2025-09-07T07:06:28.1639363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1639428Z return func(*args, **kwargs) 2025-09-07T07:06:28.1639702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1639850Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1640118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1640210Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1640214Z 2025-09-07T07:06:28.1640320Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1640525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1640593Z return mod(**inputs) 2025-09-07T07:06:28.1640828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1640902Z return func(*args, **kwargs) 2025-09-07T07:06:28.1641139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1641216Z return func(*args, **kwargs) 2025-09-07T07:06:28.1641431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1641513Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1641781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1641868Z outputs = self.layoutlm( 2025-09-07T07:06:28.1642116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1642183Z return func(*args, **kwargs) 2025-09-07T07:06:28.1642433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1642500Z return func(*args, **kwargs) 2025-09-07T07:06:28.1642719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1642805Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1643077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1643158Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1643401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1643500Z return func(*args, **kwargs) 2025-09-07T07:06:28.1643743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1643811Z return func(*args, **kwargs) 2025-09-07T07:06:28.1644055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1644121Z return func(*args, **kwargs) 2025-09-07T07:06:28.1644198Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1644423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1644496Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1644784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1644858Z layer_outputs = layer_module( 2025-09-07T07:06:28.1645087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1645169Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1645430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1645508Z return func(*args, **kwargs) 2025-09-07T07:06:28.1645770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1645849Z return func(*args, **kwargs) 2025-09-07T07:06:28.1646134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1646207Z return func(*args, **kwargs) 2025-09-07T07:06:28.1646509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1646602Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1646894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1646979Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1647318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1647444Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1647720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1647822Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1647825Z 2025-09-07T07:06:28.1647931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1648138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1648223Z return mod(**inputs) 2025-09-07T07:06:28.1648474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1648559Z return func(*args, **kwargs) 2025-09-07T07:06:28.1648815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1648892Z return func(*args, **kwargs) 2025-09-07T07:06:28.1649128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1649211Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1649504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1649578Z outputs = self.layoutlm( 2025-09-07T07:06:28.1649843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1649934Z return func(*args, **kwargs) 2025-09-07T07:06:28.1650197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1650268Z return func(*args, **kwargs) 2025-09-07T07:06:28.1650501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1650587Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1650875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1650961Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1651236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1651311Z return func(*args, **kwargs) 2025-09-07T07:06:28.1651577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1651650Z return func(*args, **kwargs) 2025-09-07T07:06:28.1651916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1651988Z return func(*args, **kwargs) 2025-09-07T07:06:28.1652070Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1652308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1652390Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1652713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1652792Z layer_outputs = layer_module( 2025-09-07T07:06:28.1653030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1653122Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1653380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1653458Z return func(*args, **kwargs) 2025-09-07T07:06:28.1653715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1653796Z return func(*args, **kwargs) 2025-09-07T07:06:28.1654049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1654122Z return func(*args, **kwargs) 2025-09-07T07:06:28.1654422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1654515Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1654851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1654937Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1655264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1655404Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1655697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1655829Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1656062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1656145Z return self.act(input) 2025-09-07T07:06:28.1656151Z 2025-09-07T07:06:28.1656264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1656482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1656583Z return mod(**inputs) 2025-09-07T07:06:28.1656849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1656930Z return func(*args, **kwargs) 2025-09-07T07:06:28.1657193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1657265Z return func(*args, **kwargs) 2025-09-07T07:06:28.1657510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1657592Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1657917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1657994Z outputs = self.layoutlm( 2025-09-07T07:06:28.1658253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1658332Z return func(*args, **kwargs) 2025-09-07T07:06:28.1658587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1658667Z return func(*args, **kwargs) 2025-09-07T07:06:28.1658901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1658990Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1659295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1659375Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1659638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1659712Z return func(*args, **kwargs) 2025-09-07T07:06:28.1659973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1660047Z return func(*args, **kwargs) 2025-09-07T07:06:28.1660301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1660380Z return func(*args, **kwargs) 2025-09-07T07:06:28.1660464Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1660700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1660779Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1661067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1661151Z layer_outputs = layer_module( 2025-09-07T07:06:28.1661413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1661507Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1661761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1661832Z return func(*args, **kwargs) 2025-09-07T07:06:28.1662093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1662164Z return func(*args, **kwargs) 2025-09-07T07:06:28.1662429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1662500Z return func(*args, **kwargs) 2025-09-07T07:06:28.1662796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1662887Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1663167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1663280Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1663604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1663758Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1664046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1664138Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1664149Z 2025-09-07T07:06:28.1664263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1664493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1664573Z return mod(**inputs) 2025-09-07T07:06:28.1664831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1664912Z return func(*args, **kwargs) 2025-09-07T07:06:28.1665170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1665241Z return func(*args, **kwargs) 2025-09-07T07:06:28.1665483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1665565Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1665981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1666065Z outputs = self.layoutlm( 2025-09-07T07:06:28.1666337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1666421Z return func(*args, **kwargs) 2025-09-07T07:06:28.1666686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1666774Z return func(*args, **kwargs) 2025-09-07T07:06:28.1667022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1667104Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1667404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1667482Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1667747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1667818Z return func(*args, **kwargs) 2025-09-07T07:06:28.1668068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1668154Z return func(*args, **kwargs) 2025-09-07T07:06:28.1668397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1668471Z return func(*args, **kwargs) 2025-09-07T07:06:28.1668550Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1668778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1668858Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1669150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1669234Z layer_outputs = layer_module( 2025-09-07T07:06:28.1669473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1669568Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1669853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1669944Z return func(*args, **kwargs) 2025-09-07T07:06:28.1670221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1670294Z return func(*args, **kwargs) 2025-09-07T07:06:28.1670566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1670638Z return func(*args, **kwargs) 2025-09-07T07:06:28.1670942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1671042Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1671337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1671415Z return func(*args, **kwargs) 2025-09-07T07:06:28.1671666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1671742Z return func(*args, **kwargs) 2025-09-07T07:06:28.1671986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1672055Z return func(*args, **kwargs) 2025-09-07T07:06:28.1672336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1672429Z self_outputs = self.self( 2025-09-07T07:06:28.1672687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1672757Z return func(*args, **kwargs) 2025-09-07T07:06:28.1673004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1673084Z return func(*args, **kwargs) 2025-09-07T07:06:28.1673342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1673420Z return func(*args, **kwargs) 2025-09-07T07:06:28.1673707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1673862Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1673873Z 2025-09-07T07:06:28.1673984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1674194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1674273Z return mod(**inputs) 2025-09-07T07:06:28.1674524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1674627Z return func(*args, **kwargs) 2025-09-07T07:06:28.1674879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1674950Z return func(*args, **kwargs) 2025-09-07T07:06:28.1675195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1675274Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1675580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1675657Z outputs = self.layoutlm( 2025-09-07T07:06:28.1675940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1676021Z return func(*args, **kwargs) 2025-09-07T07:06:28.1676303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1676382Z return func(*args, **kwargs) 2025-09-07T07:06:28.1676630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1676709Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1677004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1677082Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1677357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1677426Z return func(*args, **kwargs) 2025-09-07T07:06:28.1677691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1677759Z return func(*args, **kwargs) 2025-09-07T07:06:28.1678006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1678086Z return func(*args, **kwargs) 2025-09-07T07:06:28.1678171Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1678411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1678489Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1678779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1678866Z layer_outputs = layer_module( 2025-09-07T07:06:28.1679122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1679216Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1679473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1679546Z return func(*args, **kwargs) 2025-09-07T07:06:28.1679813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1679885Z return func(*args, **kwargs) 2025-09-07T07:06:28.1680148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1680218Z return func(*args, **kwargs) 2025-09-07T07:06:28.1680508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1680609Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1680869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1680949Z return func(*args, **kwargs) 2025-09-07T07:06:28.1681205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1681306Z return func(*args, **kwargs) 2025-09-07T07:06:28.1681567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1681640Z return func(*args, **kwargs) 2025-09-07T07:06:28.1681942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1682018Z self_outputs = self.self( 2025-09-07T07:06:28.1682285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1682356Z return func(*args, **kwargs) 2025-09-07T07:06:28.1682616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1682693Z return func(*args, **kwargs) 2025-09-07T07:06:28.1682953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1683048Z return func(*args, **kwargs) 2025-09-07T07:06:28.1683342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1683494Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1683505Z 2025-09-07T07:06:28.1683618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1683834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1683915Z return mod(**inputs) 2025-09-07T07:06:28.1684178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1684275Z return func(*args, **kwargs) 2025-09-07T07:06:28.1684532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1684606Z return func(*args, **kwargs) 2025-09-07T07:06:28.1684847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1684928Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1685223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1685297Z outputs = self.layoutlm( 2025-09-07T07:06:28.1685573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1685653Z return func(*args, **kwargs) 2025-09-07T07:06:28.1685914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1685992Z return func(*args, **kwargs) 2025-09-07T07:06:28.1686229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1686312Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1686617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1686695Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1686963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1687036Z return func(*args, **kwargs) 2025-09-07T07:06:28.1687302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1687374Z return func(*args, **kwargs) 2025-09-07T07:06:28.1687637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1687739Z return func(*args, **kwargs) 2025-09-07T07:06:28.1687822Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1688069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1688149Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1688441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1688526Z layer_outputs = layer_module( 2025-09-07T07:06:28.1688763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1688856Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1689112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1689184Z return func(*args, **kwargs) 2025-09-07T07:06:28.1689451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1689542Z return func(*args, **kwargs) 2025-09-07T07:06:28.1689805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1689872Z return func(*args, **kwargs) 2025-09-07T07:06:28.1690143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1690244Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1690489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1690566Z return func(*args, **kwargs) 2025-09-07T07:06:28.1690824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1690900Z return func(*args, **kwargs) 2025-09-07T07:06:28.1691145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1691214Z return func(*args, **kwargs) 2025-09-07T07:06:28.1691501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1691573Z self_outputs = self.self( 2025-09-07T07:06:28.1691829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1691897Z return func(*args, **kwargs) 2025-09-07T07:06:28.1692156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1692234Z return func(*args, **kwargs) 2025-09-07T07:06:28.1692478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1692554Z return func(*args, **kwargs) 2025-09-07T07:06:28.1692830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1692981Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1692994Z 2025-09-07T07:06:28.1693076Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1693159Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1693274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1693479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1693556Z return mod(**inputs) 2025-09-07T07:06:28.1693803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1693873Z return func(*args, **kwargs) 2025-09-07T07:06:28.1694127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1694221Z return func(*args, **kwargs) 2025-09-07T07:06:28.1694457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1694533Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1694825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1694903Z outputs = self.layoutlm( 2025-09-07T07:06:28.1695157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1695237Z return func(*args, **kwargs) 2025-09-07T07:06:28.1695498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1695570Z return func(*args, **kwargs) 2025-09-07T07:06:28.1695817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1695952Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1696251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1696329Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1696593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1696663Z return func(*args, **kwargs) 2025-09-07T07:06:28.1696919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1696996Z return func(*args, **kwargs) 2025-09-07T07:06:28.1697266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1697346Z return func(*args, **kwargs) 2025-09-07T07:06:28.1697429Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1697674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1719757Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1720245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1720347Z layer_outputs = layer_module( 2025-09-07T07:06:28.1720613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1720882Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1721176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1721270Z return func(*args, **kwargs) 2025-09-07T07:06:28.1721542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1721627Z return func(*args, **kwargs) 2025-09-07T07:06:28.1721886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1721975Z return func(*args, **kwargs) 2025-09-07T07:06:28.1722274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1722380Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1722645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1722720Z return func(*args, **kwargs) 2025-09-07T07:06:28.1722988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1723061Z return func(*args, **kwargs) 2025-09-07T07:06:28.1723368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1723443Z return func(*args, **kwargs) 2025-09-07T07:06:28.1723755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1723899Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1724190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1724294Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1724301Z 2025-09-07T07:06:28.1724433Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1724673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1724746Z return mod(**inputs) 2025-09-07T07:06:28.1725018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1725134Z return func(*args, **kwargs) 2025-09-07T07:06:28.1725404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1725478Z return func(*args, **kwargs) 2025-09-07T07:06:28.1725731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1725817Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1726119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1726199Z outputs = self.layoutlm( 2025-09-07T07:06:28.1726498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1726582Z return func(*args, **kwargs) 2025-09-07T07:06:28.1726847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1726929Z return func(*args, **kwargs) 2025-09-07T07:06:28.1727166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1727251Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1727549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1727634Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1727923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1728000Z return func(*args, **kwargs) 2025-09-07T07:06:28.1728259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1728340Z return func(*args, **kwargs) 2025-09-07T07:06:28.1728602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1728681Z return func(*args, **kwargs) 2025-09-07T07:06:28.1728762Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1728979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1729062Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1729340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1729425Z layer_outputs = layer_module( 2025-09-07T07:06:28.1729651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1729745Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1730007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1730077Z return func(*args, **kwargs) 2025-09-07T07:06:28.1730332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1730402Z return func(*args, **kwargs) 2025-09-07T07:06:28.1730653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1730721Z return func(*args, **kwargs) 2025-09-07T07:06:28.1730997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1731095Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1731380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1731475Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1731802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1731961Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1732260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1732351Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1732356Z 2025-09-07T07:06:28.1732483Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1732706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1732774Z return mod(**inputs) 2025-09-07T07:06:28.1733046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1733116Z return func(*args, **kwargs) 2025-09-07T07:06:28.1733370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1733439Z return func(*args, **kwargs) 2025-09-07T07:06:28.1733662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1733745Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1734019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1734098Z outputs = self.layoutlm( 2025-09-07T07:06:28.1734362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1734430Z return func(*args, **kwargs) 2025-09-07T07:06:28.1734683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1734752Z return func(*args, **kwargs) 2025-09-07T07:06:28.1734978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1735055Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1735326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1735409Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1735650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1735723Z return func(*args, **kwargs) 2025-09-07T07:06:28.1735964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1736038Z return func(*args, **kwargs) 2025-09-07T07:06:28.1736279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1736365Z return func(*args, **kwargs) 2025-09-07T07:06:28.1736455Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1736676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1736756Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1737027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1737100Z layer_outputs = layer_module( 2025-09-07T07:06:28.1737340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1737423Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1737673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1737741Z return func(*args, **kwargs) 2025-09-07T07:06:28.1737984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1738075Z return func(*args, **kwargs) 2025-09-07T07:06:28.1738319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1738394Z return func(*args, **kwargs) 2025-09-07T07:06:28.1738669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1738756Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1739034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1739113Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1739446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1739576Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1739863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1739980Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1740200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1740280Z return self.act(input) 2025-09-07T07:06:28.1740285Z 2025-09-07T07:06:28.1740397Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1740631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1740702Z return mod(**inputs) 2025-09-07T07:06:28.1740947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1741025Z return func(*args, **kwargs) 2025-09-07T07:06:28.1741268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1741345Z return func(*args, **kwargs) 2025-09-07T07:06:28.1741567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1741651Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1741920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1741991Z outputs = self.layoutlm( 2025-09-07T07:06:28.1742248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1742320Z return func(*args, **kwargs) 2025-09-07T07:06:28.1742583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1742672Z return func(*args, **kwargs) 2025-09-07T07:06:28.1742921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1743012Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1743306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1743394Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1743657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1743731Z return func(*args, **kwargs) 2025-09-07T07:06:28.1743999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1744073Z return func(*args, **kwargs) 2025-09-07T07:06:28.1744343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1744447Z return func(*args, **kwargs) 2025-09-07T07:06:28.1744532Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1744774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1744852Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1745150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1745226Z layer_outputs = layer_module( 2025-09-07T07:06:28.1745478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1745564Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1745927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1746017Z return func(*args, **kwargs) 2025-09-07T07:06:28.1746273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1746356Z return func(*args, **kwargs) 2025-09-07T07:06:28.1746613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1746685Z return func(*args, **kwargs) 2025-09-07T07:06:28.1746986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1747079Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1747390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1747477Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1747799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1747958Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1748250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1748351Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1748355Z 2025-09-07T07:06:28.1748470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1748700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1748771Z return mod(**inputs) 2025-09-07T07:06:28.1749030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1749115Z return func(*args, **kwargs) 2025-09-07T07:06:28.1749369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1749469Z return func(*args, **kwargs) 2025-09-07T07:06:28.1749705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1749785Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1750083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1750157Z outputs = self.layoutlm( 2025-09-07T07:06:28.1750418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1750490Z return func(*args, **kwargs) 2025-09-07T07:06:28.1750753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1750825Z return func(*args, **kwargs) 2025-09-07T07:06:28.1751059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1751169Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1751458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1751543Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1751802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1751875Z return func(*args, **kwargs) 2025-09-07T07:06:28.1752144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1752219Z return func(*args, **kwargs) 2025-09-07T07:06:28.1752508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1752582Z return func(*args, **kwargs) 2025-09-07T07:06:28.1752671Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1752917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1753012Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1753311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1753386Z layer_outputs = layer_module( 2025-09-07T07:06:28.1753626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1753720Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1754004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1754086Z return func(*args, **kwargs) 2025-09-07T07:06:28.1754367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1754448Z return func(*args, **kwargs) 2025-09-07T07:06:28.1754733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1754805Z return func(*args, **kwargs) 2025-09-07T07:06:28.1755110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1755202Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1755477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1755550Z return func(*args, **kwargs) 2025-09-07T07:06:28.1755835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1755915Z return func(*args, **kwargs) 2025-09-07T07:06:28.1756203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1756284Z return func(*args, **kwargs) 2025-09-07T07:06:28.1756577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1756656Z self_outputs = self.self( 2025-09-07T07:06:28.1756978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1757051Z return func(*args, **kwargs) 2025-09-07T07:06:28.1757321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1757392Z return func(*args, **kwargs) 2025-09-07T07:06:28.1757654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1757726Z return func(*args, **kwargs) 2025-09-07T07:06:28.1758022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1758214Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1758218Z 2025-09-07T07:06:28.1758333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1758559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1758629Z return mod(**inputs) 2025-09-07T07:06:28.1758947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1759028Z return func(*args, **kwargs) 2025-09-07T07:06:28.1759309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1759388Z return func(*args, **kwargs) 2025-09-07T07:06:28.1759617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1759695Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1759984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1760054Z outputs = self.layoutlm( 2025-09-07T07:06:28.1760308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1760376Z return func(*args, **kwargs) 2025-09-07T07:06:28.1760651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1760720Z return func(*args, **kwargs) 2025-09-07T07:06:28.1760944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1761030Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1761306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1761390Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1761632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1761699Z return func(*args, **kwargs) 2025-09-07T07:06:28.1761952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1762021Z return func(*args, **kwargs) 2025-09-07T07:06:28.1762274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1762341Z return func(*args, **kwargs) 2025-09-07T07:06:28.1762422Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1762652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1762749Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1763032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1763104Z layer_outputs = layer_module( 2025-09-07T07:06:28.1763327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1763416Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1763657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1763733Z return func(*args, **kwargs) 2025-09-07T07:06:28.1763975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1764048Z return func(*args, **kwargs) 2025-09-07T07:06:28.1764290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1764376Z return func(*args, **kwargs) 2025-09-07T07:06:28.1764656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1764741Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1764991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1765069Z return func(*args, **kwargs) 2025-09-07T07:06:28.1765306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1765379Z return func(*args, **kwargs) 2025-09-07T07:06:28.1765629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1765706Z return func(*args, **kwargs) 2025-09-07T07:06:28.1765971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1766044Z self_outputs = self.self( 2025-09-07T07:06:28.1766283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1766348Z return func(*args, **kwargs) 2025-09-07T07:06:28.1766587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1766651Z return func(*args, **kwargs) 2025-09-07T07:06:28.1766915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1766988Z return func(*args, **kwargs) 2025-09-07T07:06:28.1767279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1767441Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1767447Z 2025-09-07T07:06:28.1767568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1767779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1767846Z return mod(**inputs) 2025-09-07T07:06:28.1768089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1768163Z return func(*args, **kwargs) 2025-09-07T07:06:28.1768406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1768480Z return func(*args, **kwargs) 2025-09-07T07:06:28.1768704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1768807Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1769084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1769156Z outputs = self.layoutlm( 2025-09-07T07:06:28.1769403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1769469Z return func(*args, **kwargs) 2025-09-07T07:06:28.1769716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1769784Z return func(*args, **kwargs) 2025-09-07T07:06:28.1770005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1770088Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1770362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1770446Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1770703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1770770Z return func(*args, **kwargs) 2025-09-07T07:06:28.1771018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1771086Z return func(*args, **kwargs) 2025-09-07T07:06:28.1771331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1771401Z return func(*args, **kwargs) 2025-09-07T07:06:28.1771482Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1771722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1771798Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1772079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1772153Z layer_outputs = layer_module( 2025-09-07T07:06:28.1772384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1772464Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1772703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1772779Z return func(*args, **kwargs) 2025-09-07T07:06:28.1773035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1773112Z return func(*args, **kwargs) 2025-09-07T07:06:28.1773356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1773425Z return func(*args, **kwargs) 2025-09-07T07:06:28.1773705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1773794Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1774044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1774112Z return func(*args, **kwargs) 2025-09-07T07:06:28.1774357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1774429Z return func(*args, **kwargs) 2025-09-07T07:06:28.1774672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1774745Z return func(*args, **kwargs) 2025-09-07T07:06:28.1775019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1775115Z self_outputs = self.self( 2025-09-07T07:06:28.1775356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1775422Z return func(*args, **kwargs) 2025-09-07T07:06:28.1775684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1775756Z return func(*args, **kwargs) 2025-09-07T07:06:28.1776017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1776089Z return func(*args, **kwargs) 2025-09-07T07:06:28.1776377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1776549Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1776555Z 2025-09-07T07:06:28.1776643Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1776755Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1776867Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1777085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1777162Z return mod(**inputs) 2025-09-07T07:06:28.1777416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1777495Z return func(*args, **kwargs) 2025-09-07T07:06:28.1777751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1777823Z return func(*args, **kwargs) 2025-09-07T07:06:28.1778077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1778160Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1778466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1778544Z outputs = self.layoutlm( 2025-09-07T07:06:28.1778811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1778880Z return func(*args, **kwargs) 2025-09-07T07:06:28.1779127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1779201Z return func(*args, **kwargs) 2025-09-07T07:06:28.1779470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1779555Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1779833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1779909Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1780163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1780232Z return func(*args, **kwargs) 2025-09-07T07:06:28.1780479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1780547Z return func(*args, **kwargs) 2025-09-07T07:06:28.1780790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1780866Z return func(*args, **kwargs) 2025-09-07T07:06:28.1780946Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1781185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1781264Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1781592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1781680Z layer_outputs = layer_module( 2025-09-07T07:06:28.1781917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1782010Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1782263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1782341Z return func(*args, **kwargs) 2025-09-07T07:06:28.1782596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1782668Z return func(*args, **kwargs) 2025-09-07T07:06:28.1782930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1783003Z return func(*args, **kwargs) 2025-09-07T07:06:28.1783299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1783409Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1783664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1783743Z return func(*args, **kwargs) 2025-09-07T07:06:28.1784002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1784081Z return func(*args, **kwargs) 2025-09-07T07:06:28.1784345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1784418Z return func(*args, **kwargs) 2025-09-07T07:06:28.1784746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1784890Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1785191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1785283Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1785287Z 2025-09-07T07:06:28.1785407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1785623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1785779Z return mod(**inputs) 2025-09-07T07:06:28.1786086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1786165Z return func(*args, **kwargs) 2025-09-07T07:06:28.1786442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1786518Z return func(*args, **kwargs) 2025-09-07T07:06:28.1786762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1786858Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1787157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1787244Z outputs = self.layoutlm( 2025-09-07T07:06:28.1787512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1787584Z return func(*args, **kwargs) 2025-09-07T07:06:28.1787851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1787924Z return func(*args, **kwargs) 2025-09-07T07:06:28.1788166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1788267Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1788576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1788655Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1788913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1788991Z return func(*args, **kwargs) 2025-09-07T07:06:28.1789247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1789325Z return func(*args, **kwargs) 2025-09-07T07:06:28.1789580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1789653Z return func(*args, **kwargs) 2025-09-07T07:06:28.1789744Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1789979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1790085Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1790375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1790452Z layer_outputs = layer_module( 2025-09-07T07:06:28.1790697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1790782Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1791045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1791118Z return func(*args, **kwargs) 2025-09-07T07:06:28.1791396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1791472Z return func(*args, **kwargs) 2025-09-07T07:06:28.1791733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1791815Z return func(*args, **kwargs) 2025-09-07T07:06:28.1792103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1792202Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1792487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1792597Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1792932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1793069Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1793365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1793456Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1793462Z 2025-09-07T07:06:28.1793581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1793799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1793869Z return mod(**inputs) 2025-09-07T07:06:28.1794133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1794206Z return func(*args, **kwargs) 2025-09-07T07:06:28.1794471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1794543Z return func(*args, **kwargs) 2025-09-07T07:06:28.1794776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1794882Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1795174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1795254Z outputs = self.layoutlm( 2025-09-07T07:06:28.1795516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1795587Z return func(*args, **kwargs) 2025-09-07T07:06:28.1795855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1795929Z return func(*args, **kwargs) 2025-09-07T07:06:28.1796170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1796250Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1796542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1796648Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1796903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1796981Z return func(*args, **kwargs) 2025-09-07T07:06:28.1797233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1797310Z return func(*args, **kwargs) 2025-09-07T07:06:28.1797569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1797639Z return func(*args, **kwargs) 2025-09-07T07:06:28.1797729Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1797977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1798066Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1798362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1798439Z layer_outputs = layer_module( 2025-09-07T07:06:28.1798694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1798781Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1799046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1799118Z return func(*args, **kwargs) 2025-09-07T07:06:28.1799390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1799472Z return func(*args, **kwargs) 2025-09-07T07:06:28.1799729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1799808Z return func(*args, **kwargs) 2025-09-07T07:06:28.1800098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1800196Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1800477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1800557Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1800890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1801021Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1801315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1801464Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1801697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1801782Z return self.act(input) 2025-09-07T07:06:28.1801786Z 2025-09-07T07:06:28.1801900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1802130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1802202Z return mod(**inputs) 2025-09-07T07:06:28.1802459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1802542Z return func(*args, **kwargs) 2025-09-07T07:06:28.1802800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1802881Z return func(*args, **kwargs) 2025-09-07T07:06:28.1803114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1803220Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1803512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1803588Z outputs = self.layoutlm( 2025-09-07T07:06:28.1803853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1803924Z return func(*args, **kwargs) 2025-09-07T07:06:28.1804187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1804258Z return func(*args, **kwargs) 2025-09-07T07:06:28.1804511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1804601Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1804891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1804978Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1805241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1805313Z return func(*args, **kwargs) 2025-09-07T07:06:28.1805575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1805647Z return func(*args, **kwargs) 2025-09-07T07:06:28.1805926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1805999Z return func(*args, **kwargs) 2025-09-07T07:06:28.1806083Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1806329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1806410Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1806711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1806788Z layer_outputs = layer_module( 2025-09-07T07:06:28.1807034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1807122Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1807380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1807461Z return func(*args, **kwargs) 2025-09-07T07:06:28.1807719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1807799Z return func(*args, **kwargs) 2025-09-07T07:06:28.1808054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1808147Z return func(*args, **kwargs) 2025-09-07T07:06:28.1808449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1808540Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1808825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1808908Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1809240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1809386Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1809676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1809776Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1809798Z 2025-09-07T07:06:28.1809912Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1810137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1810206Z return mod(**inputs) 2025-09-07T07:06:28.1810462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1810540Z return func(*args, **kwargs) 2025-09-07T07:06:28.1810801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1810880Z return func(*args, **kwargs) 2025-09-07T07:06:28.1811129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1811210Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1811507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1811581Z outputs = self.layoutlm( 2025-09-07T07:06:28.1811845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1811915Z return func(*args, **kwargs) 2025-09-07T07:06:28.1812177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1812248Z return func(*args, **kwargs) 2025-09-07T07:06:28.1812498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1812589Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1812885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1812972Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1813231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1813303Z return func(*args, **kwargs) 2025-09-07T07:06:28.1813580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1813654Z return func(*args, **kwargs) 2025-09-07T07:06:28.1813925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1813996Z return func(*args, **kwargs) 2025-09-07T07:06:28.1814080Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1814326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1814406Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1814709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1814808Z layer_outputs = layer_module( 2025-09-07T07:06:28.1815052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1815146Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1815423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1815502Z return func(*args, **kwargs) 2025-09-07T07:06:28.1815780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1815859Z return func(*args, **kwargs) 2025-09-07T07:06:28.1816134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1816204Z return func(*args, **kwargs) 2025-09-07T07:06:28.1816512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1816620Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1816893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1816964Z return func(*args, **kwargs) 2025-09-07T07:06:28.1817226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1817306Z return func(*args, **kwargs) 2025-09-07T07:06:28.1817572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1817651Z return func(*args, **kwargs) 2025-09-07T07:06:28.1817976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1818056Z self_outputs = self.self( 2025-09-07T07:06:28.1818329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1818402Z return func(*args, **kwargs) 2025-09-07T07:06:28.1818670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1818741Z return func(*args, **kwargs) 2025-09-07T07:06:28.1819016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1819087Z return func(*args, **kwargs) 2025-09-07T07:06:28.1819402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1819751Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1819760Z 2025-09-07T07:06:28.1819880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1820115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1820188Z return mod(**inputs) 2025-09-07T07:06:28.1820482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1820565Z return func(*args, **kwargs) 2025-09-07T07:06:28.1820834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1820916Z return func(*args, **kwargs) 2025-09-07T07:06:28.1821156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1821238Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1821548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1821673Z outputs = self.layoutlm( 2025-09-07T07:06:28.1821965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1822038Z return func(*args, **kwargs) 2025-09-07T07:06:28.1822323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1822396Z return func(*args, **kwargs) 2025-09-07T07:06:28.1822642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1822731Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1823030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1823117Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1823381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1823455Z return func(*args, **kwargs) 2025-09-07T07:06:28.1823754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1823827Z return func(*args, **kwargs) 2025-09-07T07:06:28.1824098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1824169Z return func(*args, **kwargs) 2025-09-07T07:06:28.1824252Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1824498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1824578Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1824903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1824986Z layer_outputs = layer_module( 2025-09-07T07:06:28.1825237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1825326Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1825591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1825726Z return func(*args, **kwargs) 2025-09-07T07:06:28.1825998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1826080Z return func(*args, **kwargs) 2025-09-07T07:06:28.1826378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1826454Z return func(*args, **kwargs) 2025-09-07T07:06:28.1826765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1826861Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1827134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1827209Z return func(*args, **kwargs) 2025-09-07T07:06:28.1827473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1827556Z return func(*args, **kwargs) 2025-09-07T07:06:28.1827820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1827903Z return func(*args, **kwargs) 2025-09-07T07:06:28.1828205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1828285Z self_outputs = self.self( 2025-09-07T07:06:28.1828555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1828651Z return func(*args, **kwargs) 2025-09-07T07:06:28.1828921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1828994Z return func(*args, **kwargs) 2025-09-07T07:06:28.1829263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1829335Z return func(*args, **kwargs) 2025-09-07T07:06:28.1829645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1829806Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1829810Z 2025-09-07T07:06:28.1829927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1830156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1830228Z return mod(**inputs) 2025-09-07T07:06:28.1830508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1830590Z return func(*args, **kwargs) 2025-09-07T07:06:28.1830849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1830930Z return func(*args, **kwargs) 2025-09-07T07:06:28.1831169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1831258Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1831555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1831646Z outputs = self.layoutlm( 2025-09-07T07:06:28.1831921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1831991Z return func(*args, **kwargs) 2025-09-07T07:06:28.1832240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1832307Z return func(*args, **kwargs) 2025-09-07T07:06:28.1832527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1832612Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1832901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1832984Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1833232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1833302Z return func(*args, **kwargs) 2025-09-07T07:06:28.1833550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1833618Z return func(*args, **kwargs) 2025-09-07T07:06:28.1833872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1833938Z return func(*args, **kwargs) 2025-09-07T07:06:28.1834016Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1834241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1834313Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1834590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1834662Z layer_outputs = layer_module( 2025-09-07T07:06:28.1834890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1834990Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1835231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1835305Z return func(*args, **kwargs) 2025-09-07T07:06:28.1835542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1835614Z return func(*args, **kwargs) 2025-09-07T07:06:28.1835853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1835921Z return func(*args, **kwargs) 2025-09-07T07:06:28.1836199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1836285Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1836531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1836618Z return func(*args, **kwargs) 2025-09-07T07:06:28.1836861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1836935Z return func(*args, **kwargs) 2025-09-07T07:06:28.1837180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1837255Z return func(*args, **kwargs) 2025-09-07T07:06:28.1837532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1837612Z self_outputs = self.self( 2025-09-07T07:06:28.1837878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1837946Z return func(*args, **kwargs) 2025-09-07T07:06:28.1838198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1838266Z return func(*args, **kwargs) 2025-09-07T07:06:28.1838515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1838581Z return func(*args, **kwargs) 2025-09-07T07:06:28.1838855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1839012Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1839016Z 2025-09-07T07:06:28.1839119Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1839210Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1839318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1839526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1839604Z return mod(**inputs) 2025-09-07T07:06:28.1839855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1839930Z return func(*args, **kwargs) 2025-09-07T07:06:28.1840171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1840237Z return func(*args, **kwargs) 2025-09-07T07:06:28.1840464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1840538Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1840823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1840892Z outputs = self.layoutlm( 2025-09-07T07:06:28.1841141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1841227Z return func(*args, **kwargs) 2025-09-07T07:06:28.1841467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1841541Z return func(*args, **kwargs) 2025-09-07T07:06:28.1841760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1841840Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1842115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1842190Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1842439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1842506Z return func(*args, **kwargs) 2025-09-07T07:06:28.1842762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1842852Z return func(*args, **kwargs) 2025-09-07T07:06:28.1843086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1843161Z return func(*args, **kwargs) 2025-09-07T07:06:28.1843236Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1843466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1843538Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1843813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1843907Z layer_outputs = layer_module( 2025-09-07T07:06:28.1844151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1844243Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1844488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1844556Z return func(*args, **kwargs) 2025-09-07T07:06:28.1844802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1844868Z return func(*args, **kwargs) 2025-09-07T07:06:28.1845124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1845189Z return func(*args, **kwargs) 2025-09-07T07:06:28.1845470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1845566Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1845807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1845887Z return func(*args, **kwargs) 2025-09-07T07:06:28.1846129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1846196Z return func(*args, **kwargs) 2025-09-07T07:06:28.1846444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1846513Z return func(*args, **kwargs) 2025-09-07T07:06:28.1846796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1846929Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1847212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1847316Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1847320Z 2025-09-07T07:06:28.1847428Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1847640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1847706Z return mod(**inputs) 2025-09-07T07:06:28.1847960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1848027Z return func(*args, **kwargs) 2025-09-07T07:06:28.1848269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1848346Z return func(*args, **kwargs) 2025-09-07T07:06:28.1848567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1848654Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1848926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1849017Z outputs = self.layoutlm( 2025-09-07T07:06:28.1849271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1849337Z return func(*args, **kwargs) 2025-09-07T07:06:28.1849591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1849657Z return func(*args, **kwargs) 2025-09-07T07:06:28.1849890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1849967Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1850264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1850347Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1850595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1850670Z return func(*args, **kwargs) 2025-09-07T07:06:28.1850927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1850997Z return func(*args, **kwargs) 2025-09-07T07:06:28.1851262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1851333Z return func(*args, **kwargs) 2025-09-07T07:06:28.1851420Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1851671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1851751Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1852049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1852133Z layer_outputs = layer_module( 2025-09-07T07:06:28.1852360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1852438Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1852674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1852746Z return func(*args, **kwargs) 2025-09-07T07:06:28.1852982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1853056Z return func(*args, **kwargs) 2025-09-07T07:06:28.1853295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1853373Z return func(*args, **kwargs) 2025-09-07T07:06:28.1853648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1853755Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1854028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1854106Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1854419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1854543Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1854818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1854909Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1854915Z 2025-09-07T07:06:28.1855021Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1855232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1855320Z return mod(**inputs) 2025-09-07T07:06:28.1855572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1855640Z return func(*args, **kwargs) 2025-09-07T07:06:28.1855881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1855957Z return func(*args, **kwargs) 2025-09-07T07:06:28.1856178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1856262Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1856554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1856630Z outputs = self.layoutlm( 2025-09-07T07:06:28.1856898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1856973Z return func(*args, **kwargs) 2025-09-07T07:06:28.1857246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1857312Z return func(*args, **kwargs) 2025-09-07T07:06:28.1857530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1857613Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1857900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1857983Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1858227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1858302Z return func(*args, **kwargs) 2025-09-07T07:06:28.1858552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1858620Z return func(*args, **kwargs) 2025-09-07T07:06:28.1858866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1858933Z return func(*args, **kwargs) 2025-09-07T07:06:28.1859021Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1859239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1859314Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1859597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1859670Z layer_outputs = layer_module( 2025-09-07T07:06:28.1859899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1859999Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1860241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1860319Z return func(*args, **kwargs) 2025-09-07T07:06:28.1860574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1860652Z return func(*args, **kwargs) 2025-09-07T07:06:28.1860913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1860984Z return func(*args, **kwargs) 2025-09-07T07:06:28.1861283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1861374Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1861667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1861769Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1862099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1862231Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1862520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1862651Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1862881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1862981Z return self.act(input) 2025-09-07T07:06:28.1862985Z 2025-09-07T07:06:28.1863100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1863318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1863397Z return mod(**inputs) 2025-09-07T07:06:28.1863655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1863732Z return func(*args, **kwargs) 2025-09-07T07:06:28.1863993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1864070Z return func(*args, **kwargs) 2025-09-07T07:06:28.1864325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1864407Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1864705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1864782Z outputs = self.layoutlm( 2025-09-07T07:06:28.1865047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1865120Z return func(*args, **kwargs) 2025-09-07T07:06:28.1865374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1865450Z return func(*args, **kwargs) 2025-09-07T07:06:28.1865758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1865854Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1866147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1866226Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1866494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1866604Z return func(*args, **kwargs) 2025-09-07T07:06:28.1866882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1866956Z return func(*args, **kwargs) 2025-09-07T07:06:28.1867224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1867298Z return func(*args, **kwargs) 2025-09-07T07:06:28.1867377Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1867612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1867688Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1867980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1868052Z layer_outputs = layer_module( 2025-09-07T07:06:28.1868286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1868395Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1868636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1868710Z return func(*args, **kwargs) 2025-09-07T07:06:28.1868951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1869018Z return func(*args, **kwargs) 2025-09-07T07:06:28.1869266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1869333Z return func(*args, **kwargs) 2025-09-07T07:06:28.1869627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1869717Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1869981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1870071Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1870375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1870521Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1870817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1870911Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1870914Z 2025-09-07T07:06:28.1871022Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1871226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1871304Z return mod(**inputs) 2025-09-07T07:06:28.1871551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1871627Z return func(*args, **kwargs) 2025-09-07T07:06:28.1871870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1871938Z return func(*args, **kwargs) 2025-09-07T07:06:28.1872166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1872242Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1872522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1872592Z outputs = self.layoutlm( 2025-09-07T07:06:28.1872843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1872932Z return func(*args, **kwargs) 2025-09-07T07:06:28.1873179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1873256Z return func(*args, **kwargs) 2025-09-07T07:06:28.1873481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1873565Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1873842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1873918Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1874172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1874241Z return func(*args, **kwargs) 2025-09-07T07:06:28.1874494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1874581Z return func(*args, **kwargs) 2025-09-07T07:06:28.1874822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1874898Z return func(*args, **kwargs) 2025-09-07T07:06:28.1874977Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1875212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1875285Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1875555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1875633Z layer_outputs = layer_module( 2025-09-07T07:06:28.1875866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1875959Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1876204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1876280Z return func(*args, **kwargs) 2025-09-07T07:06:28.1876531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1876600Z return func(*args, **kwargs) 2025-09-07T07:06:28.1876847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1876914Z return func(*args, **kwargs) 2025-09-07T07:06:28.1877218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1877306Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1877539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1877614Z return func(*args, **kwargs) 2025-09-07T07:06:28.1877849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1877921Z return func(*args, **kwargs) 2025-09-07T07:06:28.1878155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1878223Z return func(*args, **kwargs) 2025-09-07T07:06:28.1878502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1878573Z self_outputs = self.self( 2025-09-07T07:06:28.1878813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1878879Z return func(*args, **kwargs) 2025-09-07T07:06:28.1879138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1879206Z return func(*args, **kwargs) 2025-09-07T07:06:28.1879438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1879511Z return func(*args, **kwargs) 2025-09-07T07:06:28.1879774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:28.1879926Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1879930Z 2025-09-07T07:06:28.1880036Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1880238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1880312Z return mod(**inputs) 2025-09-07T07:06:28.1880555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1880629Z return func(*args, **kwargs) 2025-09-07T07:06:28.1880884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1880949Z return func(*args, **kwargs) 2025-09-07T07:06:28.1881175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1881251Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1881548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1881622Z outputs = self.layoutlm( 2025-09-07T07:06:28.1881926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1882005Z return func(*args, **kwargs) 2025-09-07T07:06:28.1882253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1882329Z return func(*args, **kwargs) 2025-09-07T07:06:28.1882553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1882634Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1882905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1882979Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1883249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1883318Z return func(*args, **kwargs) 2025-09-07T07:06:28.1883573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1883641Z return func(*args, **kwargs) 2025-09-07T07:06:28.1883882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1883957Z return func(*args, **kwargs) 2025-09-07T07:06:28.1884037Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1884265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1884339Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1884616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1884696Z layer_outputs = layer_module( 2025-09-07T07:06:28.1884923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1885015Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1885259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1885352Z return func(*args, **kwargs) 2025-09-07T07:06:28.1885603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1885674Z return func(*args, **kwargs) 2025-09-07T07:06:28.1885943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1886014Z return func(*args, **kwargs) 2025-09-07T07:06:28.1886318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1886410Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1886672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1886754Z return func(*args, **kwargs) 2025-09-07T07:06:28.1887020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1887116Z return func(*args, **kwargs) 2025-09-07T07:06:28.1887370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1887440Z return func(*args, **kwargs) 2025-09-07T07:06:28.1887734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1887807Z self_outputs = self.self( 2025-09-07T07:06:28.1888073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1888145Z return func(*args, **kwargs) 2025-09-07T07:06:28.1888430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1888504Z return func(*args, **kwargs) 2025-09-07T07:06:28.1888760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1888841Z return func(*args, **kwargs) 2025-09-07T07:06:28.1889131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:28.1889287Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1889291Z 2025-09-07T07:06:28.1889403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1889644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1889724Z return mod(**inputs) 2025-09-07T07:06:28.1889982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1890060Z return func(*args, **kwargs) 2025-09-07T07:06:28.1890319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1890392Z return func(*args, **kwargs) 2025-09-07T07:06:28.1890632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1890712Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1891007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1891081Z outputs = self.layoutlm( 2025-09-07T07:06:28.1891356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1891429Z return func(*args, **kwargs) 2025-09-07T07:06:28.1891693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1891793Z return func(*args, **kwargs) 2025-09-07T07:06:28.1892032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1892125Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1892423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1892502Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1892795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1892869Z return func(*args, **kwargs) 2025-09-07T07:06:28.1893151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1893224Z return func(*args, **kwargs) 2025-09-07T07:06:28.1893479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1893559Z return func(*args, **kwargs) 2025-09-07T07:06:28.1893661Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1893901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1893980Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1894276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1894351Z layer_outputs = layer_module( 2025-09-07T07:06:28.1894588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1894684Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1895779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1895867Z return func(*args, **kwargs) 2025-09-07T07:06:28.1896128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1896204Z return func(*args, **kwargs) 2025-09-07T07:06:28.1896475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1896547Z return func(*args, **kwargs) 2025-09-07T07:06:28.1896845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1896934Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1897212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1897296Z return func(*args, **kwargs) 2025-09-07T07:06:28.1897551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1897630Z return func(*args, **kwargs) 2025-09-07T07:06:28.1897885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1897959Z return func(*args, **kwargs) 2025-09-07T07:06:28.1898257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:28.1898332Z self_outputs = self.self( 2025-09-07T07:06:28.1898595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1898666Z return func(*args, **kwargs) 2025-09-07T07:06:28.1898927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1898998Z return func(*args, **kwargs) 2025-09-07T07:06:28.1899253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1899353Z return func(*args, **kwargs) 2025-09-07T07:06:28.1899652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:28.1899816Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:28.1899821Z 2025-09-07T07:06:28.1899908Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1899992Z cudagraph partition due to non gpu ops 2025-09-07T07:06:28.1900112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1900334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1900412Z return mod(**inputs) 2025-09-07T07:06:28.1900678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1900751Z return func(*args, **kwargs) 2025-09-07T07:06:28.1901025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1901118Z return func(*args, **kwargs) 2025-09-07T07:06:28.1901361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1901442Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1901749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1901826Z outputs = self.layoutlm( 2025-09-07T07:06:28.1902092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1902174Z return func(*args, **kwargs) 2025-09-07T07:06:28.1902454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1902536Z return func(*args, **kwargs) 2025-09-07T07:06:28.1902786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1902872Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1903179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1903260Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1903534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1903607Z return func(*args, **kwargs) 2025-09-07T07:06:28.1903890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1903975Z return func(*args, **kwargs) 2025-09-07T07:06:28.1904237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1904319Z return func(*args, **kwargs) 2025-09-07T07:06:28.1904401Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1904642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1904731Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1905028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1905117Z layer_outputs = layer_module( 2025-09-07T07:06:28.1905369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1905468Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1905812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1905892Z return func(*args, **kwargs) 2025-09-07T07:06:28.1906188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1906264Z return func(*args, **kwargs) 2025-09-07T07:06:28.1906541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1906613Z return func(*args, **kwargs) 2025-09-07T07:06:28.1906922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:28.1907023Z self_attention_outputs = self.attention( 2025-09-07T07:06:28.1907282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1907361Z return func(*args, **kwargs) 2025-09-07T07:06:28.1907620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1907696Z return func(*args, **kwargs) 2025-09-07T07:06:28.1907959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1908051Z return func(*args, **kwargs) 2025-09-07T07:06:28.1908355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:28.1908499Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:28.1908816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:28.1908925Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1908929Z 2025-09-07T07:06:28.1909044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1909292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1909366Z return mod(**inputs) 2025-09-07T07:06:28.1909635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1909710Z return func(*args, **kwargs) 2025-09-07T07:06:28.1909972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1910052Z return func(*args, **kwargs) 2025-09-07T07:06:28.1910286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1910374Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1910699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1910778Z outputs = self.layoutlm( 2025-09-07T07:06:28.1911042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1911115Z return func(*args, **kwargs) 2025-09-07T07:06:28.1911379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1911453Z return func(*args, **kwargs) 2025-09-07T07:06:28.1911699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1911780Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1912087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1912171Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1912427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1912503Z return func(*args, **kwargs) 2025-09-07T07:06:28.1912759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1912848Z return func(*args, **kwargs) 2025-09-07T07:06:28.1913113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1913185Z return func(*args, **kwargs) 2025-09-07T07:06:28.1913276Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1913515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1913595Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1913900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1913977Z layer_outputs = layer_module( 2025-09-07T07:06:28.1914228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1914317Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1914586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1914679Z return func(*args, **kwargs) 2025-09-07T07:06:28.1914948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1915029Z return func(*args, **kwargs) 2025-09-07T07:06:28.1915295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1915374Z return func(*args, **kwargs) 2025-09-07T07:06:28.1915676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1915770Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1916096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1916183Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1916534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1916669Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1916985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:28.1917083Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1917087Z 2025-09-07T07:06:28.1917200Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1917443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1917519Z return mod(**inputs) 2025-09-07T07:06:28.1917807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1917883Z return func(*args, **kwargs) 2025-09-07T07:06:28.1918160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1918243Z return func(*args, **kwargs) 2025-09-07T07:06:28.1918489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1918577Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1918881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1918957Z outputs = self.layoutlm( 2025-09-07T07:06:28.1919239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1919313Z return func(*args, **kwargs) 2025-09-07T07:06:28.1919750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1919877Z return func(*args, **kwargs) 2025-09-07T07:06:28.1920121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1920211Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1920510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1920602Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1920876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1920959Z return func(*args, **kwargs) 2025-09-07T07:06:28.1921243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1921321Z return func(*args, **kwargs) 2025-09-07T07:06:28.1921595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1921702Z return func(*args, **kwargs) 2025-09-07T07:06:28.1921797Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1922042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1922124Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1922429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1922506Z layer_outputs = layer_module( 2025-09-07T07:06:28.1922765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1922855Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1923160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1923245Z return func(*args, **kwargs) 2025-09-07T07:06:28.1923520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1923604Z return func(*args, **kwargs) 2025-09-07T07:06:28.1923877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1923950Z return func(*args, **kwargs) 2025-09-07T07:06:28.1924304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1924398Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1924732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1924822Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1925161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:28.1925298Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:28.1925600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:28.1925735Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:28.1925972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:28.1926058Z return self.act(input) 2025-09-07T07:06:28.1926062Z 2025-09-07T07:06:28.1926179Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1926405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1926486Z return mod(**inputs) 2025-09-07T07:06:28.1926764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1926866Z return func(*args, **kwargs) 2025-09-07T07:06:28.1927150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1927230Z return func(*args, **kwargs) 2025-09-07T07:06:28.1927473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1927554Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1927860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-09-07T07:06:28.1927940Z outputs = self.layoutlm( 2025-09-07T07:06:28.1928218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1928291Z return func(*args, **kwargs) 2025-09-07T07:06:28.1928564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1928663Z return func(*args, **kwargs) 2025-09-07T07:06:28.1928895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1928980Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1929278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:28.1929357Z encoder_outputs = self.encoder( 2025-09-07T07:06:28.1929626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1929697Z return func(*args, **kwargs) 2025-09-07T07:06:28.1929984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1930057Z return func(*args, **kwargs) 2025-09-07T07:06:28.1930327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1930401Z return func(*args, **kwargs) 2025-09-07T07:06:28.1930483Z [Previous line repeated 1 more time] 2025-09-07T07:06:28.1930723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1930800Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1931098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:28.1931174Z layer_outputs = layer_module( 2025-09-07T07:06:28.1931444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:28.1931539Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:28.1931798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1931881Z return func(*args, **kwargs) 2025-09-07T07:06:28.1932146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1932216Z return func(*args, **kwargs) 2025-09-07T07:06:28.1932486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1932559Z return func(*args, **kwargs) 2025-09-07T07:06:28.1932870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:28.1932964Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:28.1933253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:28.1933341Z return forward_fn(*input_tensors) 2025-09-07T07:06:28.1933683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:28.1933838Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:28.1934134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:28.1934230Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1934234Z 2025-09-07T07:06:28.1934346Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1934566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1934650Z return mod(**inputs) 2025-09-07T07:06:28.1934915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1934997Z return func(*args, **kwargs) 2025-09-07T07:06:28.1935260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1935350Z return func(*args, **kwargs) 2025-09-07T07:06:28.1935592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1935673Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1935973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-09-07T07:06:28.1936080Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:06:28.1936388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-09-07T07:06:28.1936517Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:06:28.1936844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 472, in forward 2025-09-07T07:06:28.1936954Z hidden_states = self.transform(hidden_states) 2025-09-07T07:06:28.1937241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 447, in forward 2025-09-07T07:06:28.1937338Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:28.1937342Z 2025-09-07T07:06:28.1937452Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1937668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1937748Z return mod(**inputs) 2025-09-07T07:06:28.1938022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1938103Z return func(*args, **kwargs) 2025-09-07T07:06:28.1938361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1938441Z return func(*args, **kwargs) 2025-09-07T07:06:28.1938673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1938755Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1939051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-09-07T07:06:28.1939149Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:06:28.1939441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-09-07T07:06:28.1939561Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:06:28.1939849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 473, in forward 2025-09-07T07:06:28.1939958Z hidden_states = self.decoder(hidden_states) 2025-09-07T07:06:28.1939961Z 2025-09-07T07:06:28.1940071Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:28.1940314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:28.1940387Z return mod(**inputs) 2025-09-07T07:06:28.1940652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1940723Z return func(*args, **kwargs) 2025-09-07T07:06:28.1940976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:28.1941055Z return func(*args, **kwargs) 2025-09-07T07:06:28.1941295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:28.1941386Z output = func(self, *args, **kwargs) 2025-09-07T07:06:28.1941682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 776, in forward 2025-09-07T07:06:28.1941762Z masked_lm_loss = loss_fct( 2025-09-07T07:06:28.1941766Z 2025-09-07T07:06:39.1147450Z Compilation time (from dynamo_timed): 18.306699287 2025-09-07T07:06:39.1231694Z pass 2025-09-07T07:06:39.1236604Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:39.1239333Z TIMING: _recursive_pre_grad_passes:0.0084 _recursive_joint_graph_passes:0.4522 _recursive_post_grad_passes:0.07791 async_compile.wait:0.68455 code_gen:10.12698 inductor_compile:11.4366 backend_compile:15.11173 gc:0.00059 entire_frame_compile:18.3067 total_wall_time:18.3067 2025-09-07T07:06:39.1240497Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:15436 | FakeTensor.__torch_dispatch__:4457 | ProxyTorchDispatchMode.__torch_dispatch__:5848 2025-09-07T07:06:39.1245297Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-09-07T07:06:41.8205184Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:06:41.8206303Z import pynvml # type: ignore[import] 2025-09-07T07:06:44.6294168Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:06:44.6295376Z from pkg_resources import resource_filename 2025-09-07T07:06:45.3272214Z 2025-09-07T07:06:46.3855452Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:06:46.3856024Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:06:46.3864544Z cpu eval LayoutLMForSequenceClassification 2025-09-07T07:06:46.9106833Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:47.1092755Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:47.3100203Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:06:55.7429828Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7430473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7430858Z return mod(**inputs) 2025-09-07T07:06:55.7431257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7431706Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7432140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7432576Z outputs = self.layoutlm( 2025-09-07T07:06:55.7433302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7433705Z return func(*args, **kwargs) 2025-09-07T07:06:55.7434086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7434463Z return func(*args, **kwargs) 2025-09-07T07:06:55.7434847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7435213Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7435643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7436056Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7436583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7437050Z return func(*args, **kwargs) 2025-09-07T07:06:55.7437635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7438186Z return func(*args, **kwargs) 2025-09-07T07:06:55.7438560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7438947Z return func(*args, **kwargs) 2025-09-07T07:06:55.7439150Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7439517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7439888Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7440309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7440853Z layer_outputs = layer_module( 2025-09-07T07:06:55.7441390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7441965Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7442471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7442958Z return func(*args, **kwargs) 2025-09-07T07:06:55.7443408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7443855Z return func(*args, **kwargs) 2025-09-07T07:06:55.7444423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7445718Z return func(*args, **kwargs) 2025-09-07T07:06:55.7446240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7446755Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7447250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7447777Z return func(*args, **kwargs) 2025-09-07T07:06:55.7448164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7448563Z return func(*args, **kwargs) 2025-09-07T07:06:55.7448971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7449376Z return func(*args, **kwargs) 2025-09-07T07:06:55.7449795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7450235Z self_outputs = self.self( 2025-09-07T07:06:55.7450633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7451072Z return func(*args, **kwargs) 2025-09-07T07:06:55.7451470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7451876Z return func(*args, **kwargs) 2025-09-07T07:06:55.7452275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7452716Z return func(*args, **kwargs) 2025-09-07T07:06:55.7453158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.7453777Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7454006Z 2025-09-07T07:06:55.7454133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7454632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7455011Z return mod(**inputs) 2025-09-07T07:06:55.7455431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7455992Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7456534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7457266Z outputs = self.layoutlm( 2025-09-07T07:06:55.7457815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7458317Z return func(*args, **kwargs) 2025-09-07T07:06:55.7458721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7459134Z return func(*args, **kwargs) 2025-09-07T07:06:55.7459599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7460064Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7460627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7461281Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7461814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7462235Z return func(*args, **kwargs) 2025-09-07T07:06:55.7462640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7463058Z return func(*args, **kwargs) 2025-09-07T07:06:55.7463483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7463897Z return func(*args, **kwargs) 2025-09-07T07:06:55.7464116Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7464509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7464908Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7465629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7466294Z layer_outputs = layer_module( 2025-09-07T07:06:55.7466694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7467118Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7467534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7467927Z return func(*args, **kwargs) 2025-09-07T07:06:55.7468328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7468746Z return func(*args, **kwargs) 2025-09-07T07:06:55.7469174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7469699Z return func(*args, **kwargs) 2025-09-07T07:06:55.7470355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7471068Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7471504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7471912Z return func(*args, **kwargs) 2025-09-07T07:06:55.7472304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7472712Z return func(*args, **kwargs) 2025-09-07T07:06:55.7473110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7473520Z return func(*args, **kwargs) 2025-09-07T07:06:55.7473950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7474425Z self_outputs = self.self( 2025-09-07T07:06:55.7474830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7475240Z return func(*args, **kwargs) 2025-09-07T07:06:55.7475644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7476042Z return func(*args, **kwargs) 2025-09-07T07:06:55.7476426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7476823Z return func(*args, **kwargs) 2025-09-07T07:06:55.7477269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.7477854Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7478175Z 2025-09-07T07:06:55.7478337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7478836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7479186Z return mod(**inputs) 2025-09-07T07:06:55.7479542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7479925Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7480378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7480814Z outputs = self.layoutlm( 2025-09-07T07:06:55.7481203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7481603Z return func(*args, **kwargs) 2025-09-07T07:06:55.7482157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7482744Z return func(*args, **kwargs) 2025-09-07T07:06:55.7483291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7483838Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7484272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7484698Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7485101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7485500Z return func(*args, **kwargs) 2025-09-07T07:06:55.7485886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7486309Z return func(*args, **kwargs) 2025-09-07T07:06:55.7486689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7487086Z return func(*args, **kwargs) 2025-09-07T07:06:55.7487299Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7487678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7488053Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7488493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7488933Z layer_outputs = layer_module( 2025-09-07T07:06:55.7489313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7489712Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7490118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7490647Z return func(*args, **kwargs) 2025-09-07T07:06:55.7491036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7491435Z return func(*args, **kwargs) 2025-09-07T07:06:55.7491814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7492214Z return func(*args, **kwargs) 2025-09-07T07:06:55.7492619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7493048Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7493464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7493834Z return func(*args, **kwargs) 2025-09-07T07:06:55.7494200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7494645Z return func(*args, **kwargs) 2025-09-07T07:06:55.7495216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7495724Z return func(*args, **kwargs) 2025-09-07T07:06:55.7496146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7496592Z self_outputs = self.self( 2025-09-07T07:06:55.7497036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7497447Z return func(*args, **kwargs) 2025-09-07T07:06:55.7497840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7498302Z return func(*args, **kwargs) 2025-09-07T07:06:55.7498938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7499557Z return func(*args, **kwargs) 2025-09-07T07:06:55.7500025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.7500547Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7500788Z 2025-09-07T07:06:55.7500931Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7501276Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7501666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7502290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7502841Z return mod(**inputs) 2025-09-07T07:06:55.7503428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7504015Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7504701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7505309Z outputs = self.layoutlm( 2025-09-07T07:06:55.7505824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7506251Z return func(*args, **kwargs) 2025-09-07T07:06:55.7506659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7507069Z return func(*args, **kwargs) 2025-09-07T07:06:55.7507446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7507812Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7508259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7508739Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7509146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7509563Z return func(*args, **kwargs) 2025-09-07T07:06:55.7509960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7510379Z return func(*args, **kwargs) 2025-09-07T07:06:55.7510776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7511174Z return func(*args, **kwargs) 2025-09-07T07:06:55.7511411Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7511807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7512202Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7512644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7513093Z layer_outputs = layer_module( 2025-09-07T07:06:55.7513479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7513882Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7514322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7514723Z return func(*args, **kwargs) 2025-09-07T07:06:55.7515123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7515533Z return func(*args, **kwargs) 2025-09-07T07:06:55.7515938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7516342Z return func(*args, **kwargs) 2025-09-07T07:06:55.7516760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7517179Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7517568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7517941Z return func(*args, **kwargs) 2025-09-07T07:06:55.7518294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7518671Z return func(*args, **kwargs) 2025-09-07T07:06:55.7519038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7519456Z return func(*args, **kwargs) 2025-09-07T07:06:55.7520140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.7520643Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.7521142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.7521568Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7521716Z 2025-09-07T07:06:55.7521835Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7522216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7522576Z return mod(**inputs) 2025-09-07T07:06:55.7522940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7523333Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7523773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7524280Z outputs = self.layoutlm( 2025-09-07T07:06:55.7524646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7525021Z return func(*args, **kwargs) 2025-09-07T07:06:55.7525386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7525762Z return func(*args, **kwargs) 2025-09-07T07:06:55.7526102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7526469Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7526916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7527327Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7527700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7528079Z return func(*args, **kwargs) 2025-09-07T07:06:55.7528442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7528813Z return func(*args, **kwargs) 2025-09-07T07:06:55.7529177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7529538Z return func(*args, **kwargs) 2025-09-07T07:06:55.7529775Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7530167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7530555Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7530987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7531431Z layer_outputs = layer_module( 2025-09-07T07:06:55.7531823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7532238Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7532652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7533049Z return func(*args, **kwargs) 2025-09-07T07:06:55.7533442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7533848Z return func(*args, **kwargs) 2025-09-07T07:06:55.7534245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7534758Z return func(*args, **kwargs) 2025-09-07T07:06:55.7535212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7535688Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7536149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7536603Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7537086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7537652Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7538176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.7538656Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7538814Z 2025-09-07T07:06:55.7538943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7539368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7539789Z return mod(**inputs) 2025-09-07T07:06:55.7540155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7540548Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7540996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7541444Z outputs = self.layoutlm( 2025-09-07T07:06:55.7541857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7542283Z return func(*args, **kwargs) 2025-09-07T07:06:55.7542713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7543120Z return func(*args, **kwargs) 2025-09-07T07:06:55.7543501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7543884Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7544315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7544752Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7545142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7545557Z return func(*args, **kwargs) 2025-09-07T07:06:55.7546020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7546425Z return func(*args, **kwargs) 2025-09-07T07:06:55.7546815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7547239Z return func(*args, **kwargs) 2025-09-07T07:06:55.7547452Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7547833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7548216Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7548644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7549075Z layer_outputs = layer_module( 2025-09-07T07:06:55.7549455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7549846Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7550253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7550684Z return func(*args, **kwargs) 2025-09-07T07:06:55.7551067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7551458Z return func(*args, **kwargs) 2025-09-07T07:06:55.7551838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7552227Z return func(*args, **kwargs) 2025-09-07T07:06:55.7552645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7553093Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7553533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7553962Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7554421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7554966Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7555452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.7555929Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.7556346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.7556695Z return self.act(input) 2025-09-07T07:06:55.7556821Z 2025-09-07T07:06:55.7556931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7557305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7557647Z return mod(**inputs) 2025-09-07T07:06:55.7558048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7558410Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7558826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7559242Z outputs = self.layoutlm( 2025-09-07T07:06:55.7559604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7559989Z return func(*args, **kwargs) 2025-09-07T07:06:55.7560376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7560855Z return func(*args, **kwargs) 2025-09-07T07:06:55.7561227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7561610Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7562053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7562501Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7562913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7563318Z return func(*args, **kwargs) 2025-09-07T07:06:55.7563702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7564110Z return func(*args, **kwargs) 2025-09-07T07:06:55.7564483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7564865Z return func(*args, **kwargs) 2025-09-07T07:06:55.7565066Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7565440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7565816Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7566234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7566652Z layer_outputs = layer_module( 2025-09-07T07:06:55.7567008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7567386Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7567778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7568162Z return func(*args, **kwargs) 2025-09-07T07:06:55.7568528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7568911Z return func(*args, **kwargs) 2025-09-07T07:06:55.7569287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7569671Z return func(*args, **kwargs) 2025-09-07T07:06:55.7570096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7570510Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7570928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7571335Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7571786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.7572294Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.7572787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.7573214Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7573370Z 2025-09-07T07:06:55.7573481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7573857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7574190Z return mod(**inputs) 2025-09-07T07:06:55.7574528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7574892Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7575321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7575754Z outputs = self.layoutlm( 2025-09-07T07:06:55.7576120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7576502Z return func(*args, **kwargs) 2025-09-07T07:06:55.7576872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7577255Z return func(*args, **kwargs) 2025-09-07T07:06:55.7577600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7577956Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7578364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7578777Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7579158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7579527Z return func(*args, **kwargs) 2025-09-07T07:06:55.7579897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7580285Z return func(*args, **kwargs) 2025-09-07T07:06:55.7580697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7581109Z return func(*args, **kwargs) 2025-09-07T07:06:55.7581313Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7581693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7582072Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7582505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7582928Z layer_outputs = layer_module( 2025-09-07T07:06:55.7583307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7583703Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7584110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7584507Z return func(*args, **kwargs) 2025-09-07T07:06:55.7584904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7585301Z return func(*args, **kwargs) 2025-09-07T07:06:55.7585755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7586161Z return func(*args, **kwargs) 2025-09-07T07:06:55.7586577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7587032Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7587474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7587873Z return func(*args, **kwargs) 2025-09-07T07:06:55.7588262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7588633Z return func(*args, **kwargs) 2025-09-07T07:06:55.7589021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7589414Z return func(*args, **kwargs) 2025-09-07T07:06:55.7589833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7590278Z self_outputs = self.self( 2025-09-07T07:06:55.7590677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7591074Z return func(*args, **kwargs) 2025-09-07T07:06:55.7591462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7591860Z return func(*args, **kwargs) 2025-09-07T07:06:55.7592244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7592644Z return func(*args, **kwargs) 2025-09-07T07:06:55.7593060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.7593595Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7593813Z 2025-09-07T07:06:55.7593936Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7594329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7594684Z return mod(**inputs) 2025-09-07T07:06:55.7595041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7595429Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7595919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7596354Z outputs = self.layoutlm( 2025-09-07T07:06:55.7596754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7597165Z return func(*args, **kwargs) 2025-09-07T07:06:55.7597552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7597953Z return func(*args, **kwargs) 2025-09-07T07:06:55.7598318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7598699Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7599138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7599587Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7599984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7600398Z return func(*args, **kwargs) 2025-09-07T07:06:55.7600795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7601199Z return func(*args, **kwargs) 2025-09-07T07:06:55.7601589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7601982Z return func(*args, **kwargs) 2025-09-07T07:06:55.7602195Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7602574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7602970Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7603405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7603820Z layer_outputs = layer_module( 2025-09-07T07:06:55.7604177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7604549Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7604931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7605311Z return func(*args, **kwargs) 2025-09-07T07:06:55.7605704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7606082Z return func(*args, **kwargs) 2025-09-07T07:06:55.7606444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7606811Z return func(*args, **kwargs) 2025-09-07T07:06:55.7607207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7607629Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7608017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7608384Z return func(*args, **kwargs) 2025-09-07T07:06:55.7608752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7609126Z return func(*args, **kwargs) 2025-09-07T07:06:55.7609492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7609864Z return func(*args, **kwargs) 2025-09-07T07:06:55.7610251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7610690Z self_outputs = self.self( 2025-09-07T07:06:55.7611083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7611484Z return func(*args, **kwargs) 2025-09-07T07:06:55.7611868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7612263Z return func(*args, **kwargs) 2025-09-07T07:06:55.7612649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7613049Z return func(*args, **kwargs) 2025-09-07T07:06:55.7613461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.7613936Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7614144Z 2025-09-07T07:06:55.7614254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7614626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7614981Z return mod(**inputs) 2025-09-07T07:06:55.7615319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7615677Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7616090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7616498Z outputs = self.layoutlm( 2025-09-07T07:06:55.7616868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7617238Z return func(*args, **kwargs) 2025-09-07T07:06:55.7617624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7618005Z return func(*args, **kwargs) 2025-09-07T07:06:55.7618349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7618712Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7619115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7619528Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7620074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7620491Z return func(*args, **kwargs) 2025-09-07T07:06:55.7620943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7621357Z return func(*args, **kwargs) 2025-09-07T07:06:55.7621770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7622189Z return func(*args, **kwargs) 2025-09-07T07:06:55.7622412Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7622800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7623182Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7623622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7624068Z layer_outputs = layer_module( 2025-09-07T07:06:55.7624463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7624862Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7625296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7625754Z return func(*args, **kwargs) 2025-09-07T07:06:55.7626200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7626604Z return func(*args, **kwargs) 2025-09-07T07:06:55.7626998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7627403Z return func(*args, **kwargs) 2025-09-07T07:06:55.7627831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7628291Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7628711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7629121Z return func(*args, **kwargs) 2025-09-07T07:06:55.7629518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7629928Z return func(*args, **kwargs) 2025-09-07T07:06:55.7630318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7630763Z return func(*args, **kwargs) 2025-09-07T07:06:55.7631197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7631657Z self_outputs = self.self( 2025-09-07T07:06:55.7632061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7632452Z return func(*args, **kwargs) 2025-09-07T07:06:55.7632838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7633222Z return func(*args, **kwargs) 2025-09-07T07:06:55.7633615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7633985Z return func(*args, **kwargs) 2025-09-07T07:06:55.7634364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.7634853Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7635068Z 2025-09-07T07:06:55.7635151Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7635370Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7635608Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7635994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7636324Z return mod(**inputs) 2025-09-07T07:06:55.7636654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7637006Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7637406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7637809Z outputs = self.layoutlm( 2025-09-07T07:06:55.7638169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7638535Z return func(*args, **kwargs) 2025-09-07T07:06:55.7638885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7639253Z return func(*args, **kwargs) 2025-09-07T07:06:55.7639592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7639945Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7640349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7640764Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7641146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7641529Z return func(*args, **kwargs) 2025-09-07T07:06:55.7641918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7642317Z return func(*args, **kwargs) 2025-09-07T07:06:55.7642709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7643109Z return func(*args, **kwargs) 2025-09-07T07:06:55.7643322Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7643688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7644080Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7644494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7644926Z layer_outputs = layer_module( 2025-09-07T07:06:55.7645281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7645646Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7646033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7646408Z return func(*args, **kwargs) 2025-09-07T07:06:55.7646782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7647151Z return func(*args, **kwargs) 2025-09-07T07:06:55.7647518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7647885Z return func(*args, **kwargs) 2025-09-07T07:06:55.7648284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7648701Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7649088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7649449Z return func(*args, **kwargs) 2025-09-07T07:06:55.7649805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7650175Z return func(*args, **kwargs) 2025-09-07T07:06:55.7650553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7650911Z return func(*args, **kwargs) 2025-09-07T07:06:55.7651314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.7651807Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.7652309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.7652769Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7652921Z 2025-09-07T07:06:55.7653039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7653428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7653781Z return mod(**inputs) 2025-09-07T07:06:55.7654139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7654521Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7654952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7655415Z outputs = self.layoutlm( 2025-09-07T07:06:55.7655800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7656199Z return func(*args, **kwargs) 2025-09-07T07:06:55.7656589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7656992Z return func(*args, **kwargs) 2025-09-07T07:06:55.7657356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7657746Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7658181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7658608Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7659020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7659466Z return func(*args, **kwargs) 2025-09-07T07:06:55.7659880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7660283Z return func(*args, **kwargs) 2025-09-07T07:06:55.7660681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7661079Z return func(*args, **kwargs) 2025-09-07T07:06:55.7661293Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7661670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7662056Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7662525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7662962Z layer_outputs = layer_module( 2025-09-07T07:06:55.7663342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7663737Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7664154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7664551Z return func(*args, **kwargs) 2025-09-07T07:06:55.7664946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7665349Z return func(*args, **kwargs) 2025-09-07T07:06:55.7665896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7666323Z return func(*args, **kwargs) 2025-09-07T07:06:55.7666762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7667231Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7667675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7668101Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7668573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7669097Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7669575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.7669984Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7670131Z 2025-09-07T07:06:55.7670238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7670603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7670969Z return mod(**inputs) 2025-09-07T07:06:55.7671307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7671666Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7672076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7672496Z outputs = self.layoutlm( 2025-09-07T07:06:55.7672861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7673228Z return func(*args, **kwargs) 2025-09-07T07:06:55.7673581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7673947Z return func(*args, **kwargs) 2025-09-07T07:06:55.7674291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7674676Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7675135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7675573Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7675973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7676371Z return func(*args, **kwargs) 2025-09-07T07:06:55.7676760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7677144Z return func(*args, **kwargs) 2025-09-07T07:06:55.7677508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7677900Z return func(*args, **kwargs) 2025-09-07T07:06:55.7678101Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7678455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7678820Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7679231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7679643Z layer_outputs = layer_module( 2025-09-07T07:06:55.7680002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7680366Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7680783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7681171Z return func(*args, **kwargs) 2025-09-07T07:06:55.7681560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7681950Z return func(*args, **kwargs) 2025-09-07T07:06:55.7682337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7682734Z return func(*args, **kwargs) 2025-09-07T07:06:55.7683161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7683588Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7683996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7684424Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7684908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7685409Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7685868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.7686368Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.7686785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.7687164Z return self.act(input) 2025-09-07T07:06:55.7687291Z 2025-09-07T07:06:55.7687418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7687821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7688176Z return mod(**inputs) 2025-09-07T07:06:55.7688540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7688933Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7689378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7689810Z outputs = self.layoutlm( 2025-09-07T07:06:55.7690218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7690616Z return func(*args, **kwargs) 2025-09-07T07:06:55.7691003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7691397Z return func(*args, **kwargs) 2025-09-07T07:06:55.7691750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7692133Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7692568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7693077Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7693663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7694184Z return func(*args, **kwargs) 2025-09-07T07:06:55.7694571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7694969Z return func(*args, **kwargs) 2025-09-07T07:06:55.7695353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7695739Z return func(*args, **kwargs) 2025-09-07T07:06:55.7695950Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7696356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7696742Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7697173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7697624Z layer_outputs = layer_module( 2025-09-07T07:06:55.7698000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7698405Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7698826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7699223Z return func(*args, **kwargs) 2025-09-07T07:06:55.7699624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7700034Z return func(*args, **kwargs) 2025-09-07T07:06:55.7700467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7700869Z return func(*args, **kwargs) 2025-09-07T07:06:55.7701316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7701807Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7702249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7702696Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7703153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.7703688Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.7704267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.7704742Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7704896Z 2025-09-07T07:06:55.7705027Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7705428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7705879Z return mod(**inputs) 2025-09-07T07:06:55.7706275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7706676Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7707122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7707558Z outputs = self.layoutlm( 2025-09-07T07:06:55.7707961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7708386Z return func(*args, **kwargs) 2025-09-07T07:06:55.7708787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7709220Z return func(*args, **kwargs) 2025-09-07T07:06:55.7709603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7710003Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7710451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7710905Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7711309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7711719Z return func(*args, **kwargs) 2025-09-07T07:06:55.7712142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7712554Z return func(*args, **kwargs) 2025-09-07T07:06:55.7712953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7713369Z return func(*args, **kwargs) 2025-09-07T07:06:55.7713593Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7713993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7714389Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7714836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7715287Z layer_outputs = layer_module( 2025-09-07T07:06:55.7715685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7716083Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7716489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7716892Z return func(*args, **kwargs) 2025-09-07T07:06:55.7717285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7717708Z return func(*args, **kwargs) 2025-09-07T07:06:55.7718096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7718495Z return func(*args, **kwargs) 2025-09-07T07:06:55.7718926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7719391Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7719973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7720379Z return func(*args, **kwargs) 2025-09-07T07:06:55.7720762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7721165Z return func(*args, **kwargs) 2025-09-07T07:06:55.7721568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7722025Z return func(*args, **kwargs) 2025-09-07T07:06:55.7722437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7722872Z self_outputs = self.self( 2025-09-07T07:06:55.7723265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7723662Z return func(*args, **kwargs) 2025-09-07T07:06:55.7724054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7724446Z return func(*args, **kwargs) 2025-09-07T07:06:55.7724865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7725263Z return func(*args, **kwargs) 2025-09-07T07:06:55.7725692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.7726209Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7726439Z 2025-09-07T07:06:55.7726560Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7726963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7727323Z return mod(**inputs) 2025-09-07T07:06:55.7727684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7728095Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7728528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7728936Z outputs = self.layoutlm( 2025-09-07T07:06:55.7729305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7729680Z return func(*args, **kwargs) 2025-09-07T07:06:55.7730042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7730419Z return func(*args, **kwargs) 2025-09-07T07:06:55.7730760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7731126Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7731563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7732001Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7732403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7732831Z return func(*args, **kwargs) 2025-09-07T07:06:55.7733219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7733590Z return func(*args, **kwargs) 2025-09-07T07:06:55.7733955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7734332Z return func(*args, **kwargs) 2025-09-07T07:06:55.7734533Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7734889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7735467Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7736070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7736665Z layer_outputs = layer_module( 2025-09-07T07:06:55.7737047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7737437Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7737882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7738303Z return func(*args, **kwargs) 2025-09-07T07:06:55.7738688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7739076Z return func(*args, **kwargs) 2025-09-07T07:06:55.7739464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7739873Z return func(*args, **kwargs) 2025-09-07T07:06:55.7740322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7740785Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7741206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7741619Z return func(*args, **kwargs) 2025-09-07T07:06:55.7742014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7742422Z return func(*args, **kwargs) 2025-09-07T07:06:55.7742823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7743270Z return func(*args, **kwargs) 2025-09-07T07:06:55.7743814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7744265Z self_outputs = self.self( 2025-09-07T07:06:55.7744792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7745199Z return func(*args, **kwargs) 2025-09-07T07:06:55.7745596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7746152Z return func(*args, **kwargs) 2025-09-07T07:06:55.7746560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7746969Z return func(*args, **kwargs) 2025-09-07T07:06:55.7747398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.7747928Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7748156Z 2025-09-07T07:06:55.7748277Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7748693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7749066Z return mod(**inputs) 2025-09-07T07:06:55.7749455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7749853Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7750303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7750746Z outputs = self.layoutlm( 2025-09-07T07:06:55.7751139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7751547Z return func(*args, **kwargs) 2025-09-07T07:06:55.7751943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7752353Z return func(*args, **kwargs) 2025-09-07T07:06:55.7752728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7753116Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7753566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7754042Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7754457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7754866Z return func(*args, **kwargs) 2025-09-07T07:06:55.7755247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7755635Z return func(*args, **kwargs) 2025-09-07T07:06:55.7756000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7756377Z return func(*args, **kwargs) 2025-09-07T07:06:55.7756592Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7756962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7757333Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7757746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7758151Z layer_outputs = layer_module( 2025-09-07T07:06:55.7758510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7758880Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7759286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7759663Z return func(*args, **kwargs) 2025-09-07T07:06:55.7760023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7760403Z return func(*args, **kwargs) 2025-09-07T07:06:55.7760773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7761159Z return func(*args, **kwargs) 2025-09-07T07:06:55.7761549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7761979Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7762370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7762744Z return func(*args, **kwargs) 2025-09-07T07:06:55.7763111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7763549Z return func(*args, **kwargs) 2025-09-07T07:06:55.7763917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7764315Z return func(*args, **kwargs) 2025-09-07T07:06:55.7764712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7765124Z self_outputs = self.self( 2025-09-07T07:06:55.7765487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7765864Z return func(*args, **kwargs) 2025-09-07T07:06:55.7766228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7766602Z return func(*args, **kwargs) 2025-09-07T07:06:55.7766962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7767343Z return func(*args, **kwargs) 2025-09-07T07:06:55.7767742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.7768234Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7768459Z 2025-09-07T07:06:55.7768550Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7768768Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7769014Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7769386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7769718Z return mod(**inputs) 2025-09-07T07:06:55.7770046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7770414Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7770840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7771252Z outputs = self.layoutlm( 2025-09-07T07:06:55.7771620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7771993Z return func(*args, **kwargs) 2025-09-07T07:06:55.7772359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7772737Z return func(*args, **kwargs) 2025-09-07T07:06:55.7773081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7773433Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7773861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7774280Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7774666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7775043Z return func(*args, **kwargs) 2025-09-07T07:06:55.7775412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7775811Z return func(*args, **kwargs) 2025-09-07T07:06:55.7776193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7776590Z return func(*args, **kwargs) 2025-09-07T07:06:55.7776793Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7777176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7777558Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7777996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7778432Z layer_outputs = layer_module( 2025-09-07T07:06:55.7778805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7779234Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7779642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7780043Z return func(*args, **kwargs) 2025-09-07T07:06:55.7780431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7780824Z return func(*args, **kwargs) 2025-09-07T07:06:55.7781216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7781615Z return func(*args, **kwargs) 2025-09-07T07:06:55.7782044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7782494Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7782916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7783376Z return func(*args, **kwargs) 2025-09-07T07:06:55.7783761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7784169Z return func(*args, **kwargs) 2025-09-07T07:06:55.7784557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7784962Z return func(*args, **kwargs) 2025-09-07T07:06:55.7785381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.7786030Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.7786576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.7787051Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7787230Z 2025-09-07T07:06:55.7787347Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7787750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7788126Z return mod(**inputs) 2025-09-07T07:06:55.7788489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7788887Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7789355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7789807Z outputs = self.layoutlm( 2025-09-07T07:06:55.7790211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7790624Z return func(*args, **kwargs) 2025-09-07T07:06:55.7791031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7791449Z return func(*args, **kwargs) 2025-09-07T07:06:55.7791825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7792217Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7792677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7793133Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7793551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7793964Z return func(*args, **kwargs) 2025-09-07T07:06:55.7794360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7794789Z return func(*args, **kwargs) 2025-09-07T07:06:55.7795201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7795611Z return func(*args, **kwargs) 2025-09-07T07:06:55.7795822Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7796213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7796604Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7797056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7797503Z layer_outputs = layer_module( 2025-09-07T07:06:55.7797882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7798286Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7798707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7799150Z return func(*args, **kwargs) 2025-09-07T07:06:55.7799518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7799886Z return func(*args, **kwargs) 2025-09-07T07:06:55.7800250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7800628Z return func(*args, **kwargs) 2025-09-07T07:06:55.7801033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7801453Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7801909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7802355Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7802832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7803368Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7803859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.7804291Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7804443Z 2025-09-07T07:06:55.7804552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7804969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7805335Z return mod(**inputs) 2025-09-07T07:06:55.7805684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7806070Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7806508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7806942Z outputs = self.layoutlm( 2025-09-07T07:06:55.7807335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7807747Z return func(*args, **kwargs) 2025-09-07T07:06:55.7808144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7808552Z return func(*args, **kwargs) 2025-09-07T07:06:55.7808921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7809297Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7809733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7810192Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7810600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7810996Z return func(*args, **kwargs) 2025-09-07T07:06:55.7811386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7811790Z return func(*args, **kwargs) 2025-09-07T07:06:55.7812180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7812580Z return func(*args, **kwargs) 2025-09-07T07:06:55.7812787Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7813174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7813555Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7813992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7814444Z layer_outputs = layer_module( 2025-09-07T07:06:55.7814823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7815213Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7815624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7816023Z return func(*args, **kwargs) 2025-09-07T07:06:55.7816402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7816801Z return func(*args, **kwargs) 2025-09-07T07:06:55.7817204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7817606Z return func(*args, **kwargs) 2025-09-07T07:06:55.7818030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7818472Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7818913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7819339Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7820214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7820816Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7821330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.7821834Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.7822266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.7822652Z return self.act(input) 2025-09-07T07:06:55.7822775Z 2025-09-07T07:06:55.7822890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7823285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7823647Z return mod(**inputs) 2025-09-07T07:06:55.7824004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7824392Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7824826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7825260Z outputs = self.layoutlm( 2025-09-07T07:06:55.7825706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7826168Z return func(*args, **kwargs) 2025-09-07T07:06:55.7826564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7826965Z return func(*args, **kwargs) 2025-09-07T07:06:55.7827329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7827727Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7828165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7828599Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7829016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7829489Z return func(*args, **kwargs) 2025-09-07T07:06:55.7829885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7830281Z return func(*args, **kwargs) 2025-09-07T07:06:55.7830709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7831116Z return func(*args, **kwargs) 2025-09-07T07:06:55.7831339Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7831727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7832108Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7832556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7832998Z layer_outputs = layer_module( 2025-09-07T07:06:55.7833400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7833771Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7834161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7834547Z return func(*args, **kwargs) 2025-09-07T07:06:55.7834926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7835312Z return func(*args, **kwargs) 2025-09-07T07:06:55.7835681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7836073Z return func(*args, **kwargs) 2025-09-07T07:06:55.7836546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7836988Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7837415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7837826Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7838282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.7838801Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.7839286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.7839725Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7839874Z 2025-09-07T07:06:55.7839988Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7840371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7840721Z return mod(**inputs) 2025-09-07T07:06:55.7841071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7841463Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7841894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7842322Z outputs = self.layoutlm( 2025-09-07T07:06:55.7842708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7843105Z return func(*args, **kwargs) 2025-09-07T07:06:55.7843480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7843876Z return func(*args, **kwargs) 2025-09-07T07:06:55.7844240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7844620Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7845042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7845487Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7845884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7846262Z return func(*args, **kwargs) 2025-09-07T07:06:55.7846630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7847002Z return func(*args, **kwargs) 2025-09-07T07:06:55.7847369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7847748Z return func(*args, **kwargs) 2025-09-07T07:06:55.7847948Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7848319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7848684Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7849093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7849512Z layer_outputs = layer_module( 2025-09-07T07:06:55.7849867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7850232Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7850621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7850698Z return func(*args, **kwargs) 2025-09-07T07:06:55.7850955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7851026Z return func(*args, **kwargs) 2025-09-07T07:06:55.7851279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7851351Z return func(*args, **kwargs) 2025-09-07T07:06:55.7851647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7851737Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7851991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7852071Z return func(*args, **kwargs) 2025-09-07T07:06:55.7852327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7852407Z return func(*args, **kwargs) 2025-09-07T07:06:55.7852659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7852740Z return func(*args, **kwargs) 2025-09-07T07:06:55.7853030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7853129Z self_outputs = self.self( 2025-09-07T07:06:55.7853395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7853463Z return func(*args, **kwargs) 2025-09-07T07:06:55.7853710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7853779Z return func(*args, **kwargs) 2025-09-07T07:06:55.7854022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7854099Z return func(*args, **kwargs) 2025-09-07T07:06:55.7854373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.7854533Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7854539Z 2025-09-07T07:06:55.7854665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7854873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7854949Z return mod(**inputs) 2025-09-07T07:06:55.7855175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7855258Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7855539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7855618Z outputs = self.layoutlm( 2025-09-07T07:06:55.7855867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7855951Z return func(*args, **kwargs) 2025-09-07T07:06:55.7856203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7856274Z return func(*args, **kwargs) 2025-09-07T07:06:55.7856500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7856575Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7856847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7856929Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7857185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7857263Z return func(*args, **kwargs) 2025-09-07T07:06:55.7857505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7857573Z return func(*args, **kwargs) 2025-09-07T07:06:55.7857821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7857890Z return func(*args, **kwargs) 2025-09-07T07:06:55.7857977Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7858195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7858268Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7858555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7858630Z layer_outputs = layer_module( 2025-09-07T07:06:55.7858862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7858944Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7859190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7859279Z return func(*args, **kwargs) 2025-09-07T07:06:55.7859524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7859600Z return func(*args, **kwargs) 2025-09-07T07:06:55.7859843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7859918Z return func(*args, **kwargs) 2025-09-07T07:06:55.7860194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7860282Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7860537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7860606Z return func(*args, **kwargs) 2025-09-07T07:06:55.7860861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7860952Z return func(*args, **kwargs) 2025-09-07T07:06:55.7861199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7861278Z return func(*args, **kwargs) 2025-09-07T07:06:55.7861569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7861653Z self_outputs = self.self( 2025-09-07T07:06:55.7861916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7861996Z return func(*args, **kwargs) 2025-09-07T07:06:55.7862278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7862353Z return func(*args, **kwargs) 2025-09-07T07:06:55.7862619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7862693Z return func(*args, **kwargs) 2025-09-07T07:06:55.7862987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.7863136Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7863143Z 2025-09-07T07:06:55.7863258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7863504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7863579Z return mod(**inputs) 2025-09-07T07:06:55.7863823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7863904Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7864191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7864276Z outputs = self.layoutlm( 2025-09-07T07:06:55.7864543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7864623Z return func(*args, **kwargs) 2025-09-07T07:06:55.7864879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7864957Z return func(*args, **kwargs) 2025-09-07T07:06:55.7865190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7865271Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7865571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7865743Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7866041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7866121Z return func(*args, **kwargs) 2025-09-07T07:06:55.7866394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7866479Z return func(*args, **kwargs) 2025-09-07T07:06:55.7866798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7866879Z return func(*args, **kwargs) 2025-09-07T07:06:55.7866968Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7867204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7867294Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7867586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7867719Z layer_outputs = layer_module( 2025-09-07T07:06:55.7867961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7868054Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7868323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7868395Z return func(*args, **kwargs) 2025-09-07T07:06:55.7868665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7868736Z return func(*args, **kwargs) 2025-09-07T07:06:55.7869037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7869114Z return func(*args, **kwargs) 2025-09-07T07:06:55.7869416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7869520Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7869794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7869875Z return func(*args, **kwargs) 2025-09-07T07:06:55.7870137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7870220Z return func(*args, **kwargs) 2025-09-07T07:06:55.7870528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7870601Z return func(*args, **kwargs) 2025-09-07T07:06:55.7870917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7870996Z self_outputs = self.self( 2025-09-07T07:06:55.7871278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7871359Z return func(*args, **kwargs) 2025-09-07T07:06:55.7871639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7871718Z return func(*args, **kwargs) 2025-09-07T07:06:55.7871979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7872059Z return func(*args, **kwargs) 2025-09-07T07:06:55.7872368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.7872533Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7872538Z 2025-09-07T07:06:55.7872657Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7872744Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7872869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7873097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7873169Z return mod(**inputs) 2025-09-07T07:06:55.7873420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7873500Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7873796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7873871Z outputs = self.layoutlm( 2025-09-07T07:06:55.7874128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7874208Z return func(*args, **kwargs) 2025-09-07T07:06:55.7874463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7874565Z return func(*args, **kwargs) 2025-09-07T07:06:55.7874798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7874884Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7875172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7875250Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7875516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7875589Z return func(*args, **kwargs) 2025-09-07T07:06:55.7875872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7875949Z return func(*args, **kwargs) 2025-09-07T07:06:55.7876204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7876285Z return func(*args, **kwargs) 2025-09-07T07:06:55.7876368Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7876609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7876689Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7876976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7877078Z layer_outputs = layer_module( 2025-09-07T07:06:55.7877319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7877412Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7877671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7877747Z return func(*args, **kwargs) 2025-09-07T07:06:55.7878011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7878084Z return func(*args, **kwargs) 2025-09-07T07:06:55.7878348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7878419Z return func(*args, **kwargs) 2025-09-07T07:06:55.7878720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7878810Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7879068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7879149Z return func(*args, **kwargs) 2025-09-07T07:06:55.7879426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7879510Z return func(*args, **kwargs) 2025-09-07T07:06:55.7879768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7879839Z return func(*args, **kwargs) 2025-09-07T07:06:55.7880135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.7880277Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.7880576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.7880668Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7880674Z 2025-09-07T07:06:55.7880795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7881011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7881107Z return mod(**inputs) 2025-09-07T07:06:55.7881348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7881429Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7881730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7881806Z outputs = self.layoutlm( 2025-09-07T07:06:55.7882067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7882148Z return func(*args, **kwargs) 2025-09-07T07:06:55.7882430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7882513Z return func(*args, **kwargs) 2025-09-07T07:06:55.7882751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7882832Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7883128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7883208Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7883470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7883543Z return func(*args, **kwargs) 2025-09-07T07:06:55.7883819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7883901Z return func(*args, **kwargs) 2025-09-07T07:06:55.7884181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7884262Z return func(*args, **kwargs) 2025-09-07T07:06:55.7884346Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7884589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7884669Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7884957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7885041Z layer_outputs = layer_module( 2025-09-07T07:06:55.7885277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7885371Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7885631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7885703Z return func(*args, **kwargs) 2025-09-07T07:06:55.7886001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7886076Z return func(*args, **kwargs) 2025-09-07T07:06:55.7886347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7886419Z return func(*args, **kwargs) 2025-09-07T07:06:55.7886720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7886822Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7887105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7887198Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7887524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7887665Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7887983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.7888075Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7888079Z 2025-09-07T07:06:55.7888202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7888419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7888498Z return mod(**inputs) 2025-09-07T07:06:55.7888735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7888814Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7889144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7889223Z outputs = self.layoutlm( 2025-09-07T07:06:55.7889489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7889563Z return func(*args, **kwargs) 2025-09-07T07:06:55.7889837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7889908Z return func(*args, **kwargs) 2025-09-07T07:06:55.7890140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7890227Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7890547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7890633Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7890908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7890981Z return func(*args, **kwargs) 2025-09-07T07:06:55.7891247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7891330Z return func(*args, **kwargs) 2025-09-07T07:06:55.7891603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7891674Z return func(*args, **kwargs) 2025-09-07T07:06:55.7891758Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7891998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7892080Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7892376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7892454Z layer_outputs = layer_module( 2025-09-07T07:06:55.7892715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7892811Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7893075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7893156Z return func(*args, **kwargs) 2025-09-07T07:06:55.7893419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7893500Z return func(*args, **kwargs) 2025-09-07T07:06:55.7893768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7893840Z return func(*args, **kwargs) 2025-09-07T07:06:55.7894137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7894229Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7894518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7894630Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7894958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7895101Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7895393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.7895525Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.7895755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.7895854Z return self.act(input) 2025-09-07T07:06:55.7895859Z 2025-09-07T07:06:55.7895977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7896199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7896281Z return mod(**inputs) 2025-09-07T07:06:55.7896515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7896604Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7896895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7896971Z outputs = self.layoutlm( 2025-09-07T07:06:55.7897257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7897334Z return func(*args, **kwargs) 2025-09-07T07:06:55.7897598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7897674Z return func(*args, **kwargs) 2025-09-07T07:06:55.7897909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7898001Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7898305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7898392Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7898659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7898732Z return func(*args, **kwargs) 2025-09-07T07:06:55.7899004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7899079Z return func(*args, **kwargs) 2025-09-07T07:06:55.7899351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7899446Z return func(*args, **kwargs) 2025-09-07T07:06:55.7899542Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7899782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7899863Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7900172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7900250Z layer_outputs = layer_module( 2025-09-07T07:06:55.7900504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7900595Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7900861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7900945Z return func(*args, **kwargs) 2025-09-07T07:06:55.7901212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7901314Z return func(*args, **kwargs) 2025-09-07T07:06:55.7901584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7901663Z return func(*args, **kwargs) 2025-09-07T07:06:55.7901973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7902069Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7902375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7902464Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7902824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.7902978Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.7903274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.7903375Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7903379Z 2025-09-07T07:06:55.7903496Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7903724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7903797Z return mod(**inputs) 2025-09-07T07:06:55.7904056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7904148Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7904441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7904532Z outputs = self.layoutlm( 2025-09-07T07:06:55.7904799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7904881Z return func(*args, **kwargs) 2025-09-07T07:06:55.7905152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7905227Z return func(*args, **kwargs) 2025-09-07T07:06:55.7905472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7905555Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7906067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7906157Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7906438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7906547Z return func(*args, **kwargs) 2025-09-07T07:06:55.7906820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7906902Z return func(*args, **kwargs) 2025-09-07T07:06:55.7907177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7907252Z return func(*args, **kwargs) 2025-09-07T07:06:55.7907348Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7907590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7907679Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7907980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7908061Z layer_outputs = layer_module( 2025-09-07T07:06:55.7908316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7908427Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7908705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7908780Z return func(*args, **kwargs) 2025-09-07T07:06:55.7909062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7909137Z return func(*args, **kwargs) 2025-09-07T07:06:55.7909410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7909492Z return func(*args, **kwargs) 2025-09-07T07:06:55.7909817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7909922Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7910207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7910281Z return func(*args, **kwargs) 2025-09-07T07:06:55.7910566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7910640Z return func(*args, **kwargs) 2025-09-07T07:06:55.7910922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7911014Z return func(*args, **kwargs) 2025-09-07T07:06:55.7911313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7911402Z self_outputs = self.self( 2025-09-07T07:06:55.7911676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7911761Z return func(*args, **kwargs) 2025-09-07T07:06:55.7912033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7912114Z return func(*args, **kwargs) 2025-09-07T07:06:55.7912386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7912462Z return func(*args, **kwargs) 2025-09-07T07:06:55.7912773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.7912938Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7912942Z 2025-09-07T07:06:55.7913068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7913289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7913384Z return mod(**inputs) 2025-09-07T07:06:55.7913636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7913719Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7914025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7914103Z outputs = self.layoutlm( 2025-09-07T07:06:55.7914378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7914462Z return func(*args, **kwargs) 2025-09-07T07:06:55.7914732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7914814Z return func(*args, **kwargs) 2025-09-07T07:06:55.7915055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7915168Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7915471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7915551Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7915831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7915907Z return func(*args, **kwargs) 2025-09-07T07:06:55.7916179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7916253Z return func(*args, **kwargs) 2025-09-07T07:06:55.7916532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7916616Z return func(*args, **kwargs) 2025-09-07T07:06:55.7916704Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7916950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7917034Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7917328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7917413Z layer_outputs = layer_module( 2025-09-07T07:06:55.7917658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7917755Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7918039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7918115Z return func(*args, **kwargs) 2025-09-07T07:06:55.7918393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7918468Z return func(*args, **kwargs) 2025-09-07T07:06:55.7918749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7918823Z return func(*args, **kwargs) 2025-09-07T07:06:55.7919129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7919222Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7919488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7919750Z return func(*args, **kwargs) 2025-09-07T07:06:55.7920182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7920266Z return func(*args, **kwargs) 2025-09-07T07:06:55.7920596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7920675Z return func(*args, **kwargs) 2025-09-07T07:06:55.7920978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7921057Z self_outputs = self.self( 2025-09-07T07:06:55.7921328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7921402Z return func(*args, **kwargs) 2025-09-07T07:06:55.7921666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7921749Z return func(*args, **kwargs) 2025-09-07T07:06:55.7922011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7922094Z return func(*args, **kwargs) 2025-09-07T07:06:55.7922392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.7922582Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7922586Z 2025-09-07T07:06:55.7922704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7922925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7923009Z return mod(**inputs) 2025-09-07T07:06:55.7923246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7923339Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7923667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7923744Z outputs = self.layoutlm( 2025-09-07T07:06:55.7924019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7924093Z return func(*args, **kwargs) 2025-09-07T07:06:55.7924362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7924433Z return func(*args, **kwargs) 2025-09-07T07:06:55.7924682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7924765Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7925068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7925153Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7925397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7925475Z return func(*args, **kwargs) 2025-09-07T07:06:55.7925716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7925787Z return func(*args, **kwargs) 2025-09-07T07:06:55.7926035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7926104Z return func(*args, **kwargs) 2025-09-07T07:06:55.7926191Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7926409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7926485Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7926766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7926842Z layer_outputs = layer_module( 2025-09-07T07:06:55.7927079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7927179Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7927428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7927503Z return func(*args, **kwargs) 2025-09-07T07:06:55.7927753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7927830Z return func(*args, **kwargs) 2025-09-07T07:06:55.7928080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7928158Z return func(*args, **kwargs) 2025-09-07T07:06:55.7928463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7928553Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7928823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7928918Z return func(*args, **kwargs) 2025-09-07T07:06:55.7929188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7929260Z return func(*args, **kwargs) 2025-09-07T07:06:55.7929523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7929613Z return func(*args, **kwargs) 2025-09-07T07:06:55.7929887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7929967Z self_outputs = self.self( 2025-09-07T07:06:55.7930235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7930321Z return func(*args, **kwargs) 2025-09-07T07:06:55.7930601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7930675Z return func(*args, **kwargs) 2025-09-07T07:06:55.7930942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7931013Z return func(*args, **kwargs) 2025-09-07T07:06:55.7931314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.7931502Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7931507Z 2025-09-07T07:06:55.7931594Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7931688Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7931803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7932038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7932109Z return mod(**inputs) 2025-09-07T07:06:55.7932329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7932413Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7932686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7932763Z outputs = self.layoutlm( 2025-09-07T07:06:55.7933009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7933080Z return func(*args, **kwargs) 2025-09-07T07:06:55.7933332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7933400Z return func(*args, **kwargs) 2025-09-07T07:06:55.7933664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7933742Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7934013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7934094Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7934337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7934414Z return func(*args, **kwargs) 2025-09-07T07:06:55.7934657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7934731Z return func(*args, **kwargs) 2025-09-07T07:06:55.7934974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7935045Z return func(*args, **kwargs) 2025-09-07T07:06:55.7935135Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7935384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7935469Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7935763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7935836Z layer_outputs = layer_module( 2025-09-07T07:06:55.7936068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7936151Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7936400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7937612Z return func(*args, **kwargs) 2025-09-07T07:06:55.7937871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7937952Z return func(*args, **kwargs) 2025-09-07T07:06:55.7938204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7938283Z return func(*args, **kwargs) 2025-09-07T07:06:55.7938570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7938666Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7938944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7939018Z return func(*args, **kwargs) 2025-09-07T07:06:55.7939291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7939363Z return func(*args, **kwargs) 2025-09-07T07:06:55.7939634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7939709Z return func(*args, **kwargs) 2025-09-07T07:06:55.7940001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.7940151Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.7940450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.7940549Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7940556Z 2025-09-07T07:06:55.7940671Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7940891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7940969Z return mod(**inputs) 2025-09-07T07:06:55.7941202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7941312Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7941599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7941680Z outputs = self.layoutlm( 2025-09-07T07:06:55.7941938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7942008Z return func(*args, **kwargs) 2025-09-07T07:06:55.7942276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7942348Z return func(*args, **kwargs) 2025-09-07T07:06:55.7942588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7942669Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7942956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7943067Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7943330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7943414Z return func(*args, **kwargs) 2025-09-07T07:06:55.7943675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7943752Z return func(*args, **kwargs) 2025-09-07T07:06:55.7944026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7944103Z return func(*args, **kwargs) 2025-09-07T07:06:55.7944213Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7944448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7944530Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7944833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7944913Z layer_outputs = layer_module( 2025-09-07T07:06:55.7945162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7945252Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7945522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7945619Z return func(*args, **kwargs) 2025-09-07T07:06:55.7945972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7946064Z return func(*args, **kwargs) 2025-09-07T07:06:55.7946328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7946413Z return func(*args, **kwargs) 2025-09-07T07:06:55.7946710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7946807Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7947114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7947198Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7947535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7947667Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7947966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.7948081Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7948086Z 2025-09-07T07:06:55.7948202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7948429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7948501Z return mod(**inputs) 2025-09-07T07:06:55.7948743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7948823Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7949115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7949199Z outputs = self.layoutlm( 2025-09-07T07:06:55.7949459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7949540Z return func(*args, **kwargs) 2025-09-07T07:06:55.7949798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7949893Z return func(*args, **kwargs) 2025-09-07T07:06:55.7950136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7950216Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7950517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7950597Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7950868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7950942Z return func(*args, **kwargs) 2025-09-07T07:06:55.7951254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7951337Z return func(*args, **kwargs) 2025-09-07T07:06:55.7951591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7951671Z return func(*args, **kwargs) 2025-09-07T07:06:55.7951755Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7951989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7952077Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7952366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7952464Z layer_outputs = layer_module( 2025-09-07T07:06:55.7952703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7952790Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7953056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7953130Z return func(*args, **kwargs) 2025-09-07T07:06:55.7953392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7953465Z return func(*args, **kwargs) 2025-09-07T07:06:55.7953719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7953799Z return func(*args, **kwargs) 2025-09-07T07:06:55.7954089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7954186Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7954470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7954579Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7954902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.7955034Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.7955330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.7955452Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.7955686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.7955765Z return self.act(input) 2025-09-07T07:06:55.7955769Z 2025-09-07T07:06:55.7955881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7956107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7956180Z return mod(**inputs) 2025-09-07T07:06:55.7956418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7956518Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7956818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7956893Z outputs = self.layoutlm( 2025-09-07T07:06:55.7957150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7957232Z return func(*args, **kwargs) 2025-09-07T07:06:55.7957495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7957572Z return func(*args, **kwargs) 2025-09-07T07:06:55.7957824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7957909Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7958210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7958291Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7958558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7958630Z return func(*args, **kwargs) 2025-09-07T07:06:55.7958892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7958972Z return func(*args, **kwargs) 2025-09-07T07:06:55.7959243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7959324Z return func(*args, **kwargs) 2025-09-07T07:06:55.7959410Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7959645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7959734Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7960023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7960108Z layer_outputs = layer_module( 2025-09-07T07:06:55.7960347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7960438Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7960698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7960771Z return func(*args, **kwargs) 2025-09-07T07:06:55.7961039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7961131Z return func(*args, **kwargs) 2025-09-07T07:06:55.7961394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7961468Z return func(*args, **kwargs) 2025-09-07T07:06:55.7961760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.7961859Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.7962142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.7962231Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.7962557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.7962710Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.7963001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.7963113Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7963116Z 2025-09-07T07:06:55.7963700Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7963922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7964004Z return mod(**inputs) 2025-09-07T07:06:55.7964243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7964326Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7964635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7964714Z outputs = self.layoutlm( 2025-09-07T07:06:55.7965015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7965092Z return func(*args, **kwargs) 2025-09-07T07:06:55.7965362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7965434Z return func(*args, **kwargs) 2025-09-07T07:06:55.7965668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7965757Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7966049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7966152Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7966413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7966483Z return func(*args, **kwargs) 2025-09-07T07:06:55.7966736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7966809Z return func(*args, **kwargs) 2025-09-07T07:06:55.7967058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7967125Z return func(*args, **kwargs) 2025-09-07T07:06:55.7967203Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7967434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7967509Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7967792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7967864Z layer_outputs = layer_module( 2025-09-07T07:06:55.7968092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7968202Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7968455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7968530Z return func(*args, **kwargs) 2025-09-07T07:06:55.7968774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7968848Z return func(*args, **kwargs) 2025-09-07T07:06:55.7969093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7969161Z return func(*args, **kwargs) 2025-09-07T07:06:55.7969442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7969530Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7969782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7969855Z return func(*args, **kwargs) 2025-09-07T07:06:55.7970139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7970219Z return func(*args, **kwargs) 2025-09-07T07:06:55.7970482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7970559Z return func(*args, **kwargs) 2025-09-07T07:06:55.7970859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7970936Z self_outputs = self.self( 2025-09-07T07:06:55.7971218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7971316Z return func(*args, **kwargs) 2025-09-07T07:06:55.7971585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7971662Z return func(*args, **kwargs) 2025-09-07T07:06:55.7971940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7972013Z return func(*args, **kwargs) 2025-09-07T07:06:55.7972314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.7972483Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7972487Z 2025-09-07T07:06:55.7972618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7972843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7972915Z return mod(**inputs) 2025-09-07T07:06:55.7973149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7973242Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7973531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7973616Z outputs = self.layoutlm( 2025-09-07T07:06:55.7973885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7973957Z return func(*args, **kwargs) 2025-09-07T07:06:55.7974228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7974304Z return func(*args, **kwargs) 2025-09-07T07:06:55.7974553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7974640Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7974946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7975051Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7975327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7975412Z return func(*args, **kwargs) 2025-09-07T07:06:55.7975693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7975775Z return func(*args, **kwargs) 2025-09-07T07:06:55.7976044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7976116Z return func(*args, **kwargs) 2025-09-07T07:06:55.7976209Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7976450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7976539Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7976833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7976931Z layer_outputs = layer_module( 2025-09-07T07:06:55.7977177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7977263Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7977531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7977603Z return func(*args, **kwargs) 2025-09-07T07:06:55.7977863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7977944Z return func(*args, **kwargs) 2025-09-07T07:06:55.7978219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7978303Z return func(*args, **kwargs) 2025-09-07T07:06:55.7978597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7978695Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7978952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7979023Z return func(*args, **kwargs) 2025-09-07T07:06:55.7979289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7979379Z return func(*args, **kwargs) 2025-09-07T07:06:55.7979644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7979719Z return func(*args, **kwargs) 2025-09-07T07:06:55.7980008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7980093Z self_outputs = self.self( 2025-09-07T07:06:55.7980350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7980430Z return func(*args, **kwargs) 2025-09-07T07:06:55.7980694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7980767Z return func(*args, **kwargs) 2025-09-07T07:06:55.7981039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7981113Z return func(*args, **kwargs) 2025-09-07T07:06:55.7981424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.7981580Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7981603Z 2025-09-07T07:06:55.7981729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7981952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7982027Z return mod(**inputs) 2025-09-07T07:06:55.7982279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7982363Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7982665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7982744Z outputs = self.layoutlm( 2025-09-07T07:06:55.7983014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7983099Z return func(*args, **kwargs) 2025-09-07T07:06:55.7983366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7983477Z return func(*args, **kwargs) 2025-09-07T07:06:55.7983721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7983803Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7984118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7984200Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7984478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7984552Z return func(*args, **kwargs) 2025-09-07T07:06:55.7984848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7984924Z return func(*args, **kwargs) 2025-09-07T07:06:55.7985199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7985284Z return func(*args, **kwargs) 2025-09-07T07:06:55.7985370Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7985617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7985783Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7986088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7986202Z layer_outputs = layer_module( 2025-09-07T07:06:55.7986447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7986544Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7986821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7986896Z return func(*args, **kwargs) 2025-09-07T07:06:55.7987160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7987234Z return func(*args, **kwargs) 2025-09-07T07:06:55.7987498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7987570Z return func(*args, **kwargs) 2025-09-07T07:06:55.7987878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7987972Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7988231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7988311Z return func(*args, **kwargs) 2025-09-07T07:06:55.7988601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7988683Z return func(*args, **kwargs) 2025-09-07T07:06:55.7988948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7989021Z return func(*args, **kwargs) 2025-09-07T07:06:55.7989335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.7989411Z self_outputs = self.self( 2025-09-07T07:06:55.7989683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7989755Z return func(*args, **kwargs) 2025-09-07T07:06:55.7990023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7990105Z return func(*args, **kwargs) 2025-09-07T07:06:55.7990370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7990473Z return func(*args, **kwargs) 2025-09-07T07:06:55.7990772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.7990945Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.7990950Z 2025-09-07T07:06:55.7991041Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7991130Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.7991258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7991486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7991587Z return mod(**inputs) 2025-09-07T07:06:55.7991830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7991916Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7992223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.7992300Z outputs = self.layoutlm( 2025-09-07T07:06:55.7992583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7992659Z return func(*args, **kwargs) 2025-09-07T07:06:55.7992933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7993038Z return func(*args, **kwargs) 2025-09-07T07:06:55.7993281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7993372Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7993670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.7993753Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.7994044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7994118Z return func(*args, **kwargs) 2025-09-07T07:06:55.7994393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7994466Z return func(*args, **kwargs) 2025-09-07T07:06:55.7994747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7994821Z return func(*args, **kwargs) 2025-09-07T07:06:55.7994905Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.7995155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.7995260Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.7995570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.7995653Z layer_outputs = layer_module( 2025-09-07T07:06:55.7995896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.7995994Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.7996268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7996352Z return func(*args, **kwargs) 2025-09-07T07:06:55.7996635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7996711Z return func(*args, **kwargs) 2025-09-07T07:06:55.7996996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7997072Z return func(*args, **kwargs) 2025-09-07T07:06:55.7997417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.7997510Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.7997784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7997859Z return func(*args, **kwargs) 2025-09-07T07:06:55.7998121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7998204Z return func(*args, **kwargs) 2025-09-07T07:06:55.7998488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.7998571Z return func(*args, **kwargs) 2025-09-07T07:06:55.7998879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.7999024Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.7999330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.7999423Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.7999426Z 2025-09-07T07:06:55.7999550Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.7999776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.7999867Z return mod(**inputs) 2025-09-07T07:06:55.8000119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8000201Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8000504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8000583Z outputs = self.layoutlm( 2025-09-07T07:06:55.8000853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8000929Z return func(*args, **kwargs) 2025-09-07T07:06:55.8001193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8001274Z return func(*args, **kwargs) 2025-09-07T07:06:55.8001514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8001602Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8001899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8001979Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8002276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8002354Z return func(*args, **kwargs) 2025-09-07T07:06:55.8002628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8002701Z return func(*args, **kwargs) 2025-09-07T07:06:55.8002965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8003046Z return func(*args, **kwargs) 2025-09-07T07:06:55.8003132Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8003381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8003464Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8003761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8003850Z layer_outputs = layer_module( 2025-09-07T07:06:55.8004115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8004214Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8004478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8004560Z return func(*args, **kwargs) 2025-09-07T07:06:55.8004824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8004903Z return func(*args, **kwargs) 2025-09-07T07:06:55.8005177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8005268Z return func(*args, **kwargs) 2025-09-07T07:06:55.8005570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8005667Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8005965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8006056Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8006384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8006523Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8006831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.8006932Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8006938Z 2025-09-07T07:06:55.8007051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8007268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8007351Z return mod(**inputs) 2025-09-07T07:06:55.8007587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8007675Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8007966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8008042Z outputs = self.layoutlm( 2025-09-07T07:06:55.8008312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8008385Z return func(*args, **kwargs) 2025-09-07T07:06:55.8008652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8008725Z return func(*args, **kwargs) 2025-09-07T07:06:55.8008979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8009069Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8009358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8009442Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8009699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8009780Z return func(*args, **kwargs) 2025-09-07T07:06:55.8010037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8010111Z return func(*args, **kwargs) 2025-09-07T07:06:55.8010374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8010448Z return func(*args, **kwargs) 2025-09-07T07:06:55.8010537Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8010790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8010869Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8011163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8011240Z layer_outputs = layer_module( 2025-09-07T07:06:55.8011482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8011571Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8011827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8011925Z return func(*args, **kwargs) 2025-09-07T07:06:55.8012188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8012270Z return func(*args, **kwargs) 2025-09-07T07:06:55.8012529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8012603Z return func(*args, **kwargs) 2025-09-07T07:06:55.8012898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8012990Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8013296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8013382Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8013717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8013853Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8014154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.8014290Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.8014533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.8014621Z return self.act(input) 2025-09-07T07:06:55.8014625Z 2025-09-07T07:06:55.8014750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8014971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8015053Z return mod(**inputs) 2025-09-07T07:06:55.8015295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8015381Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8015690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8015775Z outputs = self.layoutlm( 2025-09-07T07:06:55.8016035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8016110Z return func(*args, **kwargs) 2025-09-07T07:06:55.8016374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8016447Z return func(*args, **kwargs) 2025-09-07T07:06:55.8016687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8016768Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8017061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8017150Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8017408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8017507Z return func(*args, **kwargs) 2025-09-07T07:06:55.8017763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8017836Z return func(*args, **kwargs) 2025-09-07T07:06:55.8018098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8018171Z return func(*args, **kwargs) 2025-09-07T07:06:55.8018262Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8018497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8018593Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8018892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8018972Z layer_outputs = layer_module( 2025-09-07T07:06:55.8019224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8019314Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8019760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8019877Z return func(*args, **kwargs) 2025-09-07T07:06:55.8020317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8020406Z return func(*args, **kwargs) 2025-09-07T07:06:55.8020680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8020763Z return func(*args, **kwargs) 2025-09-07T07:06:55.8021063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8021160Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8021459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8021545Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8021890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.8022043Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.8022353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.8022451Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8022455Z 2025-09-07T07:06:55.8022618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8022854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8022931Z return mod(**inputs) 2025-09-07T07:06:55.8023180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8023264Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8023563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8023650Z outputs = self.layoutlm( 2025-09-07T07:06:55.8023919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8024005Z return func(*args, **kwargs) 2025-09-07T07:06:55.8024274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8024350Z return func(*args, **kwargs) 2025-09-07T07:06:55.8024627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8024709Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8025012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8025094Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8025365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8025441Z return func(*args, **kwargs) 2025-09-07T07:06:55.8025751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8025876Z return func(*args, **kwargs) 2025-09-07T07:06:55.8026143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8026228Z return func(*args, **kwargs) 2025-09-07T07:06:55.8026316Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8026558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8026648Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8026947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8027032Z layer_outputs = layer_module( 2025-09-07T07:06:55.8027299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8027389Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8027665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8027741Z return func(*args, **kwargs) 2025-09-07T07:06:55.8028014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8028091Z return func(*args, **kwargs) 2025-09-07T07:06:55.8028355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8028437Z return func(*args, **kwargs) 2025-09-07T07:06:55.8028736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8028838Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8029116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8029195Z return func(*args, **kwargs) 2025-09-07T07:06:55.8029463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8029556Z return func(*args, **kwargs) 2025-09-07T07:06:55.8029836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8029911Z return func(*args, **kwargs) 2025-09-07T07:06:55.8030234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8030314Z self_outputs = self.self( 2025-09-07T07:06:55.8030595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8030680Z return func(*args, **kwargs) 2025-09-07T07:06:55.8030966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8031049Z return func(*args, **kwargs) 2025-09-07T07:06:55.8031318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8031416Z return func(*args, **kwargs) 2025-09-07T07:06:55.8031740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.8031904Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8031911Z 2025-09-07T07:06:55.8032035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8032274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8032356Z return mod(**inputs) 2025-09-07T07:06:55.8032605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8032691Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8033027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8033108Z outputs = self.layoutlm( 2025-09-07T07:06:55.8033383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8033457Z return func(*args, **kwargs) 2025-09-07T07:06:55.8033732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8033815Z return func(*args, **kwargs) 2025-09-07T07:06:55.8034072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8034179Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8034469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8034548Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8034821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8034893Z return func(*args, **kwargs) 2025-09-07T07:06:55.8035143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8035211Z return func(*args, **kwargs) 2025-09-07T07:06:55.8035461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8035529Z return func(*args, **kwargs) 2025-09-07T07:06:55.8035608Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8035836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8035912Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8036221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8036324Z layer_outputs = layer_module( 2025-09-07T07:06:55.8036567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8036661Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8036927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8037006Z return func(*args, **kwargs) 2025-09-07T07:06:55.8037272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8037345Z return func(*args, **kwargs) 2025-09-07T07:06:55.8037618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8037690Z return func(*args, **kwargs) 2025-09-07T07:06:55.8037998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8038084Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8038359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8038428Z return func(*args, **kwargs) 2025-09-07T07:06:55.8038672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8038747Z return func(*args, **kwargs) 2025-09-07T07:06:55.8038991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8039069Z return func(*args, **kwargs) 2025-09-07T07:06:55.8039344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8039434Z self_outputs = self.self( 2025-09-07T07:06:55.8039684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8039755Z return func(*args, **kwargs) 2025-09-07T07:06:55.8040005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8040073Z return func(*args, **kwargs) 2025-09-07T07:06:55.8040315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8040390Z return func(*args, **kwargs) 2025-09-07T07:06:55.8040683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.8040835Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8040839Z 2025-09-07T07:06:55.8040947Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8041169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8041243Z return mod(**inputs) 2025-09-07T07:06:55.8041478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8041567Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8041855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8041936Z outputs = self.layoutlm( 2025-09-07T07:06:55.8042194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8042267Z return func(*args, **kwargs) 2025-09-07T07:06:55.8042531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8042604Z return func(*args, **kwargs) 2025-09-07T07:06:55.8042844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8042950Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8043238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8043324Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8043581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8043662Z return func(*args, **kwargs) 2025-09-07T07:06:55.8043917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8043998Z return func(*args, **kwargs) 2025-09-07T07:06:55.8044254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8044327Z return func(*args, **kwargs) 2025-09-07T07:06:55.8044419Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8044649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8044755Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8045044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8045121Z layer_outputs = layer_module( 2025-09-07T07:06:55.8045371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8045454Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8045704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8045793Z return func(*args, **kwargs) 2025-09-07T07:06:55.8046034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8046114Z return func(*args, **kwargs) 2025-09-07T07:06:55.8046360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8046438Z return func(*args, **kwargs) 2025-09-07T07:06:55.8046712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8046799Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8047054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8047151Z return func(*args, **kwargs) 2025-09-07T07:06:55.8047405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8047474Z return func(*args, **kwargs) 2025-09-07T07:06:55.8047725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8047796Z return func(*args, **kwargs) 2025-09-07T07:06:55.8048068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8048147Z self_outputs = self.self( 2025-09-07T07:06:55.8048392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8048469Z return func(*args, **kwargs) 2025-09-07T07:06:55.8048726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8048795Z return func(*args, **kwargs) 2025-09-07T07:06:55.8049063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8049135Z return func(*args, **kwargs) 2025-09-07T07:06:55.8049453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.8049613Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8049618Z 2025-09-07T07:06:55.8049712Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8049795Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8049906Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8050125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8050195Z return mod(**inputs) 2025-09-07T07:06:55.8050434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8050517Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8050803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8050887Z outputs = self.layoutlm( 2025-09-07T07:06:55.8051178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8051257Z return func(*args, **kwargs) 2025-09-07T07:06:55.8051530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8051602Z return func(*args, **kwargs) 2025-09-07T07:06:55.8051841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8051922Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8052227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8052323Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8052584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8052667Z return func(*args, **kwargs) 2025-09-07T07:06:55.8052933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8053011Z return func(*args, **kwargs) 2025-09-07T07:06:55.8053280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8053352Z return func(*args, **kwargs) 2025-09-07T07:06:55.8053445Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8053694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8053782Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8054073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8054160Z layer_outputs = layer_module( 2025-09-07T07:06:55.8054398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8054488Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8054756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8054828Z return func(*args, **kwargs) 2025-09-07T07:06:55.8055100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8055172Z return func(*args, **kwargs) 2025-09-07T07:06:55.8055427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8055508Z return func(*args, **kwargs) 2025-09-07T07:06:55.8055809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8055928Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8056185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8056259Z return func(*args, **kwargs) 2025-09-07T07:06:55.8056547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8056623Z return func(*args, **kwargs) 2025-09-07T07:06:55.8056899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8056976Z return func(*args, **kwargs) 2025-09-07T07:06:55.8057289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.8057434Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.8057751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.8057907Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8057911Z 2025-09-07T07:06:55.8058026Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8058253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8058325Z return mod(**inputs) 2025-09-07T07:06:55.8058562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8058655Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8058960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8059060Z outputs = self.layoutlm( 2025-09-07T07:06:55.8059321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8059400Z return func(*args, **kwargs) 2025-09-07T07:06:55.8059681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8059756Z return func(*args, **kwargs) 2025-09-07T07:06:55.8060001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8060084Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8060420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8060504Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8060779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8060865Z return func(*args, **kwargs) 2025-09-07T07:06:55.8061130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8061216Z return func(*args, **kwargs) 2025-09-07T07:06:55.8061481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8061554Z return func(*args, **kwargs) 2025-09-07T07:06:55.8061646Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8061885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8061975Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8062271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8062348Z layer_outputs = layer_module( 2025-09-07T07:06:55.8062603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8062710Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8062981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8063053Z return func(*args, **kwargs) 2025-09-07T07:06:55.8063322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8063396Z return func(*args, **kwargs) 2025-09-07T07:06:55.8063656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8063741Z return func(*args, **kwargs) 2025-09-07T07:06:55.8064038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8064140Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8064430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8064539Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8064884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8065021Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8065321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.8065412Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8065416Z 2025-09-07T07:06:55.8065539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8065838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8088576Z return mod(**inputs) 2025-09-07T07:06:55.8089038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8089151Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8089485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8089574Z outputs = self.layoutlm( 2025-09-07T07:06:55.8089870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8089964Z return func(*args, **kwargs) 2025-09-07T07:06:55.8090284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8090372Z return func(*args, **kwargs) 2025-09-07T07:06:55.8090618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8090705Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8091013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8091102Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8091378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8091455Z return func(*args, **kwargs) 2025-09-07T07:06:55.8091714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8091796Z return func(*args, **kwargs) 2025-09-07T07:06:55.8092057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8092140Z return func(*args, **kwargs) 2025-09-07T07:06:55.8092233Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8092479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8092604Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8092910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8093003Z layer_outputs = layer_module( 2025-09-07T07:06:55.8093256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8093358Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8093625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8093703Z return func(*args, **kwargs) 2025-09-07T07:06:55.8093979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8094058Z return func(*args, **kwargs) 2025-09-07T07:06:55.8094332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8094441Z return func(*args, **kwargs) 2025-09-07T07:06:55.8094749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8094856Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8095144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8095239Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8095572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8095716Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8096029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.8096162Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.8096397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.8096483Z return self.act(input) 2025-09-07T07:06:55.8096491Z 2025-09-07T07:06:55.8096612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8096844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8096919Z return mod(**inputs) 2025-09-07T07:06:55.8097185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8097271Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8097564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8097651Z outputs = self.layoutlm( 2025-09-07T07:06:55.8097918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8098000Z return func(*args, **kwargs) 2025-09-07T07:06:55.8098260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8098333Z return func(*args, **kwargs) 2025-09-07T07:06:55.8098577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8098658Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8098958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8099044Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8099310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8099411Z return func(*args, **kwargs) 2025-09-07T07:06:55.8099682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8099762Z return func(*args, **kwargs) 2025-09-07T07:06:55.8100019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8100099Z return func(*args, **kwargs) 2025-09-07T07:06:55.8100186Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8100427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8100516Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8100816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8100916Z layer_outputs = layer_module( 2025-09-07T07:06:55.8101164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8101271Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8101536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8101608Z return func(*args, **kwargs) 2025-09-07T07:06:55.8101872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8101945Z return func(*args, **kwargs) 2025-09-07T07:06:55.8102202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8102284Z return func(*args, **kwargs) 2025-09-07T07:06:55.8102600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8102707Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8103004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8103101Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8103442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.8103596Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.8103908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.8104023Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8104027Z 2025-09-07T07:06:55.8104156Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8104388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8104467Z return mod(**inputs) 2025-09-07T07:06:55.8104718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8104803Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8105111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8105193Z outputs = self.layoutlm( 2025-09-07T07:06:55.8105462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8105544Z return func(*args, **kwargs) 2025-09-07T07:06:55.8105920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8106012Z return func(*args, **kwargs) 2025-09-07T07:06:55.8106256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8106373Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8106672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8106758Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8107033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8107110Z return func(*args, **kwargs) 2025-09-07T07:06:55.8107381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8107458Z return func(*args, **kwargs) 2025-09-07T07:06:55.8107726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8107815Z return func(*args, **kwargs) 2025-09-07T07:06:55.8107903Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8108158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8108263Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8108562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8108652Z layer_outputs = layer_module( 2025-09-07T07:06:55.8108899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8108999Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8109267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8109348Z return func(*args, **kwargs) 2025-09-07T07:06:55.8109631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8109710Z return func(*args, **kwargs) 2025-09-07T07:06:55.8109982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8110058Z return func(*args, **kwargs) 2025-09-07T07:06:55.8110363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8110459Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8110724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8110807Z return func(*args, **kwargs) 2025-09-07T07:06:55.8111107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8111190Z return func(*args, **kwargs) 2025-09-07T07:06:55.8111459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8111535Z return func(*args, **kwargs) 2025-09-07T07:06:55.8111841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8111924Z self_outputs = self.self( 2025-09-07T07:06:55.8112195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8112269Z return func(*args, **kwargs) 2025-09-07T07:06:55.8112533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8112618Z return func(*args, **kwargs) 2025-09-07T07:06:55.8112884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8112966Z return func(*args, **kwargs) 2025-09-07T07:06:55.8113275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.8113482Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8113487Z 2025-09-07T07:06:55.8113608Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8113833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8113919Z return mod(**inputs) 2025-09-07T07:06:55.8114160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8114252Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8114563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8114640Z outputs = self.layoutlm( 2025-09-07T07:06:55.8114924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8115000Z return func(*args, **kwargs) 2025-09-07T07:06:55.8115309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8115383Z return func(*args, **kwargs) 2025-09-07T07:06:55.8115632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8115717Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8116036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8116127Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8116397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8116499Z return func(*args, **kwargs) 2025-09-07T07:06:55.8116776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8116855Z return func(*args, **kwargs) 2025-09-07T07:06:55.8117125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8117199Z return func(*args, **kwargs) 2025-09-07T07:06:55.8117292Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8117534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8117615Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8117957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8118039Z layer_outputs = layer_module( 2025-09-07T07:06:55.8118294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8118385Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8118656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8118741Z return func(*args, **kwargs) 2025-09-07T07:06:55.8119006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8119089Z return func(*args, **kwargs) 2025-09-07T07:06:55.8119363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8119446Z return func(*args, **kwargs) 2025-09-07T07:06:55.8120048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8120188Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8120585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8120759Z return func(*args, **kwargs) 2025-09-07T07:06:55.8121033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8121108Z return func(*args, **kwargs) 2025-09-07T07:06:55.8121385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8121469Z return func(*args, **kwargs) 2025-09-07T07:06:55.8121781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8121874Z self_outputs = self.self( 2025-09-07T07:06:55.8122138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8122215Z return func(*args, **kwargs) 2025-09-07T07:06:55.8122500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8122610Z return func(*args, **kwargs) 2025-09-07T07:06:55.8122889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8122963Z return func(*args, **kwargs) 2025-09-07T07:06:55.8123267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.8123436Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8123441Z 2025-09-07T07:06:55.8123560Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8123790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8123888Z return mod(**inputs) 2025-09-07T07:06:55.8124134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8124218Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8124510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8124593Z outputs = self.layoutlm( 2025-09-07T07:06:55.8124850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8124930Z return func(*args, **kwargs) 2025-09-07T07:06:55.8125186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8125284Z return func(*args, **kwargs) 2025-09-07T07:06:55.8125528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8125611Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8125905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8125991Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8126255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8126328Z return func(*args, **kwargs) 2025-09-07T07:06:55.8126585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8126665Z return func(*args, **kwargs) 2025-09-07T07:06:55.8126923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8127002Z return func(*args, **kwargs) 2025-09-07T07:06:55.8127085Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8127319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8127427Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8127722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8127807Z layer_outputs = layer_module( 2025-09-07T07:06:55.8128046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8128131Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8128396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8128470Z return func(*args, **kwargs) 2025-09-07T07:06:55.8128732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8128807Z return func(*args, **kwargs) 2025-09-07T07:06:55.8129063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8129147Z return func(*args, **kwargs) 2025-09-07T07:06:55.8129455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8129553Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8129815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8129893Z return func(*args, **kwargs) 2025-09-07T07:06:55.8130155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8130229Z return func(*args, **kwargs) 2025-09-07T07:06:55.8130499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8130590Z return func(*args, **kwargs) 2025-09-07T07:06:55.8130891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8130972Z self_outputs = self.self( 2025-09-07T07:06:55.8131225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8131305Z return func(*args, **kwargs) 2025-09-07T07:06:55.8131559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8131637Z return func(*args, **kwargs) 2025-09-07T07:06:55.8131913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8131985Z return func(*args, **kwargs) 2025-09-07T07:06:55.8132289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.8132453Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8132458Z 2025-09-07T07:06:55.8132559Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8132645Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8132765Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8132983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8133055Z return mod(**inputs) 2025-09-07T07:06:55.8133300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8133380Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8133684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8133761Z outputs = self.layoutlm( 2025-09-07T07:06:55.8134023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8134120Z return func(*args, **kwargs) 2025-09-07T07:06:55.8134377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8134456Z return func(*args, **kwargs) 2025-09-07T07:06:55.8134688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8134768Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8135069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8135150Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8135418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8135492Z return func(*args, **kwargs) 2025-09-07T07:06:55.8135756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8135849Z return func(*args, **kwargs) 2025-09-07T07:06:55.8136111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8136191Z return func(*args, **kwargs) 2025-09-07T07:06:55.8136274Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8136519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8136600Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8136901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8136987Z layer_outputs = layer_module( 2025-09-07T07:06:55.8137245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8137343Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8137602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8137676Z return func(*args, **kwargs) 2025-09-07T07:06:55.8137942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8138013Z return func(*args, **kwargs) 2025-09-07T07:06:55.8138276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8138347Z return func(*args, **kwargs) 2025-09-07T07:06:55.8138667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8138773Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8139031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8139112Z return func(*args, **kwargs) 2025-09-07T07:06:55.8139370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8139449Z return func(*args, **kwargs) 2025-09-07T07:06:55.8139703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8139775Z return func(*args, **kwargs) 2025-09-07T07:06:55.8140070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.8140215Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.8140516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.8140607Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8140629Z 2025-09-07T07:06:55.8140748Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8140980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8141053Z return mod(**inputs) 2025-09-07T07:06:55.8141304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8141388Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8141692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8141773Z outputs = self.layoutlm( 2025-09-07T07:06:55.8142040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8142124Z return func(*args, **kwargs) 2025-09-07T07:06:55.8142391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8142494Z return func(*args, **kwargs) 2025-09-07T07:06:55.8142742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8142826Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8143132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8143214Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8143486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8143563Z return func(*args, **kwargs) 2025-09-07T07:06:55.8143847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8143931Z return func(*args, **kwargs) 2025-09-07T07:06:55.8144198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8144282Z return func(*args, **kwargs) 2025-09-07T07:06:55.8144369Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8144608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8144696Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8144994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8145082Z layer_outputs = layer_module( 2025-09-07T07:06:55.8145348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8145445Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8145799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8145885Z return func(*args, **kwargs) 2025-09-07T07:06:55.8146163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8146239Z return func(*args, **kwargs) 2025-09-07T07:06:55.8146513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8146588Z return func(*args, **kwargs) 2025-09-07T07:06:55.8146890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8147005Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8147273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8147362Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8147696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8147826Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8148108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.8148194Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8148198Z 2025-09-07T07:06:55.8148315Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8148520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8148596Z return mod(**inputs) 2025-09-07T07:06:55.8148818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8148897Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8149181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8149272Z outputs = self.layoutlm( 2025-09-07T07:06:55.8149529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8149614Z return func(*args, **kwargs) 2025-09-07T07:06:55.8149884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8149966Z return func(*args, **kwargs) 2025-09-07T07:06:55.8150212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8150296Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8150625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8150709Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8150985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8151062Z return func(*args, **kwargs) 2025-09-07T07:06:55.8151335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8151409Z return func(*args, **kwargs) 2025-09-07T07:06:55.8151675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8151758Z return func(*args, **kwargs) 2025-09-07T07:06:55.8151844Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8152112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8152201Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8152514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8152610Z layer_outputs = layer_module( 2025-09-07T07:06:55.8152867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8152969Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8153246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8153328Z return func(*args, **kwargs) 2025-09-07T07:06:55.8153614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8153696Z return func(*args, **kwargs) 2025-09-07T07:06:55.8153981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8154063Z return func(*args, **kwargs) 2025-09-07T07:06:55.8154382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8154511Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8154808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8154907Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8155249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8155395Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8155714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.8155832Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.8156063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.8156138Z return self.act(input) 2025-09-07T07:06:55.8156160Z 2025-09-07T07:06:55.8156275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8156491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8156562Z return mod(**inputs) 2025-09-07T07:06:55.8156803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8156884Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8157189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8157265Z outputs = self.layoutlm( 2025-09-07T07:06:55.8157559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8157634Z return func(*args, **kwargs) 2025-09-07T07:06:55.8157897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8157980Z return func(*args, **kwargs) 2025-09-07T07:06:55.8158213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8158309Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8158583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8158658Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8158932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8159004Z return func(*args, **kwargs) 2025-09-07T07:06:55.8159257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8159328Z return func(*args, **kwargs) 2025-09-07T07:06:55.8159585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8159669Z return func(*args, **kwargs) 2025-09-07T07:06:55.8159753Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8159993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8160073Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8160369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8160449Z layer_outputs = layer_module( 2025-09-07T07:06:55.8160686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8160784Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8161035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8161137Z return func(*args, **kwargs) 2025-09-07T07:06:55.8161401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8161473Z return func(*args, **kwargs) 2025-09-07T07:06:55.8161738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8161809Z return func(*args, **kwargs) 2025-09-07T07:06:55.8162118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8162212Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8162495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8162591Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8162915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.8163088Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.8163380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.8163479Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8163483Z 2025-09-07T07:06:55.8163598Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8163829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8163909Z return mod(**inputs) 2025-09-07T07:06:55.8164172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8164259Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8164538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8164613Z outputs = self.layoutlm( 2025-09-07T07:06:55.8164883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8164955Z return func(*args, **kwargs) 2025-09-07T07:06:55.8165221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8165294Z return func(*args, **kwargs) 2025-09-07T07:06:55.8165547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8165635Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8165935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8166026Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8166282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8166362Z return func(*args, **kwargs) 2025-09-07T07:06:55.8166617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8166689Z return func(*args, **kwargs) 2025-09-07T07:06:55.8166953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8167027Z return func(*args, **kwargs) 2025-09-07T07:06:55.8167117Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8167350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8167430Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8167745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8167825Z layer_outputs = layer_module( 2025-09-07T07:06:55.8168070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8168155Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8168413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8168494Z return func(*args, **kwargs) 2025-09-07T07:06:55.8168754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8168834Z return func(*args, **kwargs) 2025-09-07T07:06:55.8169090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8169164Z return func(*args, **kwargs) 2025-09-07T07:06:55.8169459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8169571Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8169837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8169908Z return func(*args, **kwargs) 2025-09-07T07:06:55.8170169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8170243Z return func(*args, **kwargs) 2025-09-07T07:06:55.8170499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8170578Z return func(*args, **kwargs) 2025-09-07T07:06:55.8170885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8170973Z self_outputs = self.self( 2025-09-07T07:06:55.8171235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8171306Z return func(*args, **kwargs) 2025-09-07T07:06:55.8171571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8171641Z return func(*args, **kwargs) 2025-09-07T07:06:55.8171903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8171975Z return func(*args, **kwargs) 2025-09-07T07:06:55.8172289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.8172461Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8172467Z 2025-09-07T07:06:55.8172582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8172810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8172883Z return mod(**inputs) 2025-09-07T07:06:55.8173132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8173215Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8173524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8173607Z outputs = self.layoutlm( 2025-09-07T07:06:55.8173865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8173944Z return func(*args, **kwargs) 2025-09-07T07:06:55.8174204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8174295Z return func(*args, **kwargs) 2025-09-07T07:06:55.8174537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8174618Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8174916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8174995Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8175262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8175337Z return func(*args, **kwargs) 2025-09-07T07:06:55.8175596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8175679Z return func(*args, **kwargs) 2025-09-07T07:06:55.8175938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8176021Z return func(*args, **kwargs) 2025-09-07T07:06:55.8176124Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8176356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8176441Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8176731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8176815Z layer_outputs = layer_module( 2025-09-07T07:06:55.8177056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8177141Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8177497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8177572Z return func(*args, **kwargs) 2025-09-07T07:06:55.8177845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8177919Z return func(*args, **kwargs) 2025-09-07T07:06:55.8178185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8178266Z return func(*args, **kwargs) 2025-09-07T07:06:55.8178575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8178675Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8178957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8179038Z return func(*args, **kwargs) 2025-09-07T07:06:55.8179306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8179381Z return func(*args, **kwargs) 2025-09-07T07:06:55.8179645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8179717Z return func(*args, **kwargs) 2025-09-07T07:06:55.8180028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8180107Z self_outputs = self.self( 2025-09-07T07:06:55.8180368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8180454Z return func(*args, **kwargs) 2025-09-07T07:06:55.8180738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8180823Z return func(*args, **kwargs) 2025-09-07T07:06:55.8181106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8181202Z return func(*args, **kwargs) 2025-09-07T07:06:55.8181527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.8181684Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8181688Z 2025-09-07T07:06:55.8181813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8182051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8182131Z return mod(**inputs) 2025-09-07T07:06:55.8182379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8182463Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8182778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8182856Z outputs = self.layoutlm( 2025-09-07T07:06:55.8183156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8183232Z return func(*args, **kwargs) 2025-09-07T07:06:55.8183503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8183595Z return func(*args, **kwargs) 2025-09-07T07:06:55.8183826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8183916Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8184202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8184301Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8184576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8184652Z return func(*args, **kwargs) 2025-09-07T07:06:55.8184922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8184999Z return func(*args, **kwargs) 2025-09-07T07:06:55.8185282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8185356Z return func(*args, **kwargs) 2025-09-07T07:06:55.8185442Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8185797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8185890Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8186200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8186282Z layer_outputs = layer_module( 2025-09-07T07:06:55.8186528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8186628Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8186902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8186983Z return func(*args, **kwargs) 2025-09-07T07:06:55.8187240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8187314Z return func(*args, **kwargs) 2025-09-07T07:06:55.8187582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8187655Z return func(*args, **kwargs) 2025-09-07T07:06:55.8187952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8188062Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8188322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8188402Z return func(*args, **kwargs) 2025-09-07T07:06:55.8188657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8188737Z return func(*args, **kwargs) 2025-09-07T07:06:55.8188991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8189070Z return func(*args, **kwargs) 2025-09-07T07:06:55.8189360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8189438Z self_outputs = self.self( 2025-09-07T07:06:55.8189704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8189805Z return func(*args, **kwargs) 2025-09-07T07:06:55.8190069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8190141Z return func(*args, **kwargs) 2025-09-07T07:06:55.8190398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8190478Z return func(*args, **kwargs) 2025-09-07T07:06:55.8190770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.8190936Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8190940Z 2025-09-07T07:06:55.8191045Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8191133Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8191258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8191482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8191563Z return mod(**inputs) 2025-09-07T07:06:55.8191804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8191893Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8192190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8192263Z outputs = self.layoutlm( 2025-09-07T07:06:55.8192553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8192627Z return func(*args, **kwargs) 2025-09-07T07:06:55.8192892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8192967Z return func(*args, **kwargs) 2025-09-07T07:06:55.8193201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8193293Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8193593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8193681Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8193956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8194030Z return func(*args, **kwargs) 2025-09-07T07:06:55.8194309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8194421Z return func(*args, **kwargs) 2025-09-07T07:06:55.8194792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8194923Z return func(*args, **kwargs) 2025-09-07T07:06:55.8195027Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8195378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8195487Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8195793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8195870Z layer_outputs = layer_module( 2025-09-07T07:06:55.8196117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8196202Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8196463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8196547Z return func(*args, **kwargs) 2025-09-07T07:06:55.8196807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8196906Z return func(*args, **kwargs) 2025-09-07T07:06:55.8197161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8197235Z return func(*args, **kwargs) 2025-09-07T07:06:55.8197531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8197621Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8197887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8197976Z return func(*args, **kwargs) 2025-09-07T07:06:55.8198233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8198314Z return func(*args, **kwargs) 2025-09-07T07:06:55.8198572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8198650Z return func(*args, **kwargs) 2025-09-07T07:06:55.8198938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.8199087Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.8199396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.8199493Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8199497Z 2025-09-07T07:06:55.8199621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8199838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8199919Z return mod(**inputs) 2025-09-07T07:06:55.8200156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8200237Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8200533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8200606Z outputs = self.layoutlm( 2025-09-07T07:06:55.8200871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8200945Z return func(*args, **kwargs) 2025-09-07T07:06:55.8201201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8201284Z return func(*args, **kwargs) 2025-09-07T07:06:55.8201518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8201625Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8201917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8202005Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8202261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8202333Z return func(*args, **kwargs) 2025-09-07T07:06:55.8202595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8202667Z return func(*args, **kwargs) 2025-09-07T07:06:55.8202928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8202999Z return func(*args, **kwargs) 2025-09-07T07:06:55.8203083Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8203322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8203419Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8203715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8203792Z layer_outputs = layer_module( 2025-09-07T07:06:55.8204034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8204126Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8204385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8204463Z return func(*args, **kwargs) 2025-09-07T07:06:55.8204741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8204822Z return func(*args, **kwargs) 2025-09-07T07:06:55.8205083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8205157Z return func(*args, **kwargs) 2025-09-07T07:06:55.8205454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8205545Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8205836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8205941Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8206268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8206423Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8206715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.8206807Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8206818Z 2025-09-07T07:06:55.8206931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8207147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8207227Z return mod(**inputs) 2025-09-07T07:06:55.8207465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8207554Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8207846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8207924Z outputs = self.layoutlm( 2025-09-07T07:06:55.8208189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8208283Z return func(*args, **kwargs) 2025-09-07T07:06:55.8208548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8208621Z return func(*args, **kwargs) 2025-09-07T07:06:55.8208854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8208944Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8209238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8209325Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8209585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8209658Z return func(*args, **kwargs) 2025-09-07T07:06:55.8209926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8210020Z return func(*args, **kwargs) 2025-09-07T07:06:55.8210282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8210354Z return func(*args, **kwargs) 2025-09-07T07:06:55.8210437Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8210676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8210754Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8211050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8211126Z layer_outputs = layer_module( 2025-09-07T07:06:55.8211388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8211476Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8211735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8211817Z return func(*args, **kwargs) 2025-09-07T07:06:55.8212075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8212154Z return func(*args, **kwargs) 2025-09-07T07:06:55.8212409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8212499Z return func(*args, **kwargs) 2025-09-07T07:06:55.8212797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8212890Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8213174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8213262Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8213586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8213723Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8214013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.8214143Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.8214372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.8214455Z return self.act(input) 2025-09-07T07:06:55.8214459Z 2025-09-07T07:06:55.8214573Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8214826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8214907Z return mod(**inputs) 2025-09-07T07:06:55.8215139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8215225Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8215512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8215588Z outputs = self.layoutlm( 2025-09-07T07:06:55.8215851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8215923Z return func(*args, **kwargs) 2025-09-07T07:06:55.8216188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8216261Z return func(*args, **kwargs) 2025-09-07T07:06:55.8216506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8216607Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8216901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8216987Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8217244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8217323Z return func(*args, **kwargs) 2025-09-07T07:06:55.8217584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8217655Z return func(*args, **kwargs) 2025-09-07T07:06:55.8217936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8218012Z return func(*args, **kwargs) 2025-09-07T07:06:55.8218103Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8218344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8218422Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8218721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8218800Z layer_outputs = layer_module( 2025-09-07T07:06:55.8219044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8219149Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8219419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8219503Z return func(*args, **kwargs) 2025-09-07T07:06:55.8220049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8220149Z return func(*args, **kwargs) 2025-09-07T07:06:55.8220424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8220506Z return func(*args, **kwargs) 2025-09-07T07:06:55.8220804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8220900Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8221199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8221286Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8221632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.8221842Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.8222142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.8222242Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8222248Z 2025-09-07T07:06:55.8222362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8222594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8222669Z return mod(**inputs) 2025-09-07T07:06:55.8222917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8222998Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8223295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8223382Z outputs = self.layoutlm( 2025-09-07T07:06:55.8223670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8224373Z return func(*args, **kwargs) 2025-09-07T07:06:55.8224649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8224723Z return func(*args, **kwargs) 2025-09-07T07:06:55.8224978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8225060Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8225383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8225468Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8225835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8225932Z return func(*args, **kwargs) 2025-09-07T07:06:55.8226210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8226295Z return func(*args, **kwargs) 2025-09-07T07:06:55.8226570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8226651Z return func(*args, **kwargs) 2025-09-07T07:06:55.8226738Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8226981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8227105Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8227410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8227497Z layer_outputs = layer_module( 2025-09-07T07:06:55.8227745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8227836Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8228112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8228189Z return func(*args, **kwargs) 2025-09-07T07:06:55.8228466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8228538Z return func(*args, **kwargs) 2025-09-07T07:06:55.8228801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8228883Z return func(*args, **kwargs) 2025-09-07T07:06:55.8229185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8229286Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8229572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8229648Z return func(*args, **kwargs) 2025-09-07T07:06:55.8229920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8229992Z return func(*args, **kwargs) 2025-09-07T07:06:55.8230260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8230334Z return func(*args, **kwargs) 2025-09-07T07:06:55.8230637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8230717Z self_outputs = self.self( 2025-09-07T07:06:55.8230983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8231068Z return func(*args, **kwargs) 2025-09-07T07:06:55.8231330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8231429Z return func(*args, **kwargs) 2025-09-07T07:06:55.8231702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8231777Z return func(*args, **kwargs) 2025-09-07T07:06:55.8232090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.8232244Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8232248Z 2025-09-07T07:06:55.8232365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8232588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8232660Z return mod(**inputs) 2025-09-07T07:06:55.8232887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8232966Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8233244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8233314Z outputs = self.layoutlm( 2025-09-07T07:06:55.8233561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8233629Z return func(*args, **kwargs) 2025-09-07T07:06:55.8233889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8233967Z return func(*args, **kwargs) 2025-09-07T07:06:55.8234194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8234278Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8234552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8234626Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8234878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8234946Z return func(*args, **kwargs) 2025-09-07T07:06:55.8235196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8235266Z return func(*args, **kwargs) 2025-09-07T07:06:55.8235508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8235585Z return func(*args, **kwargs) 2025-09-07T07:06:55.8235663Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8235913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8235990Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8236268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8236341Z layer_outputs = layer_module( 2025-09-07T07:06:55.8236564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8236654Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8236899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8236976Z return func(*args, **kwargs) 2025-09-07T07:06:55.8237221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8237291Z return func(*args, **kwargs) 2025-09-07T07:06:55.8237539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8237625Z return func(*args, **kwargs) 2025-09-07T07:06:55.8237904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8237991Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8238270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8238346Z return func(*args, **kwargs) 2025-09-07T07:06:55.8238594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8238672Z return func(*args, **kwargs) 2025-09-07T07:06:55.8238931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8239002Z return func(*args, **kwargs) 2025-09-07T07:06:55.8239288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8239363Z self_outputs = self.self( 2025-09-07T07:06:55.8239612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8239680Z return func(*args, **kwargs) 2025-09-07T07:06:55.8239929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8240023Z return func(*args, **kwargs) 2025-09-07T07:06:55.8240269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8240348Z return func(*args, **kwargs) 2025-09-07T07:06:55.8240624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.8240780Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8240783Z 2025-09-07T07:06:55.8240891Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8241094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8241170Z return mod(**inputs) 2025-09-07T07:06:55.8241392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8241476Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8241751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8241830Z outputs = self.layoutlm( 2025-09-07T07:06:55.8242079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8242167Z return func(*args, **kwargs) 2025-09-07T07:06:55.8242427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8242496Z return func(*args, **kwargs) 2025-09-07T07:06:55.8242731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8242808Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8243090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8243174Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8243423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8243501Z return func(*args, **kwargs) 2025-09-07T07:06:55.8243754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8243843Z return func(*args, **kwargs) 2025-09-07T07:06:55.8244100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8244169Z return func(*args, **kwargs) 2025-09-07T07:06:55.8244256Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8244482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8244557Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8244843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8244916Z layer_outputs = layer_module( 2025-09-07T07:06:55.8245176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8245257Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8245503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8245570Z return func(*args, **kwargs) 2025-09-07T07:06:55.8245814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8245889Z return func(*args, **kwargs) 2025-09-07T07:06:55.8246132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8246207Z return func(*args, **kwargs) 2025-09-07T07:06:55.8246499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8246587Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8246838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8246909Z return func(*args, **kwargs) 2025-09-07T07:06:55.8247160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8247229Z return func(*args, **kwargs) 2025-09-07T07:06:55.8247472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8247546Z return func(*args, **kwargs) 2025-09-07T07:06:55.8247819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8247900Z self_outputs = self.self( 2025-09-07T07:06:55.8248141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8248220Z return func(*args, **kwargs) 2025-09-07T07:06:55.8248460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8248547Z return func(*args, **kwargs) 2025-09-07T07:06:55.8248801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8248869Z return func(*args, **kwargs) 2025-09-07T07:06:55.8249156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.8249307Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8249311Z 2025-09-07T07:06:55.8249396Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8249487Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8249595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8249816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8249886Z return mod(**inputs) 2025-09-07T07:06:55.8250112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8250217Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8250492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8250574Z outputs = self.layoutlm( 2025-09-07T07:06:55.8250835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8250907Z return func(*args, **kwargs) 2025-09-07T07:06:55.8251173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8251245Z return func(*args, **kwargs) 2025-09-07T07:06:55.8251507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8251592Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8251892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8251972Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8252236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8252318Z return func(*args, **kwargs) 2025-09-07T07:06:55.8252579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8252677Z return func(*args, **kwargs) 2025-09-07T07:06:55.8252938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8253012Z return func(*args, **kwargs) 2025-09-07T07:06:55.8253102Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8253338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8253426Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8253715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8253793Z layer_outputs = layer_module( 2025-09-07T07:06:55.8254040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8254125Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8254394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8254466Z return func(*args, **kwargs) 2025-09-07T07:06:55.8254731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8254822Z return func(*args, **kwargs) 2025-09-07T07:06:55.8255079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8255161Z return func(*args, **kwargs) 2025-09-07T07:06:55.8255449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8255546Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8255805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8255873Z return func(*args, **kwargs) 2025-09-07T07:06:55.8256124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8256192Z return func(*args, **kwargs) 2025-09-07T07:06:55.8256444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8256514Z return func(*args, **kwargs) 2025-09-07T07:06:55.8256806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.8256949Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.8257222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.8257317Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8257320Z 2025-09-07T07:06:55.8257427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8257642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8257711Z return mod(**inputs) 2025-09-07T07:06:55.8257954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8258042Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8258341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8258423Z outputs = self.layoutlm( 2025-09-07T07:06:55.8258679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8258758Z return func(*args, **kwargs) 2025-09-07T07:06:55.8259012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8259098Z return func(*args, **kwargs) 2025-09-07T07:06:55.8259325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8259402Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8259676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8259760Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8260003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8260076Z return func(*args, **kwargs) 2025-09-07T07:06:55.8260320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8260396Z return func(*args, **kwargs) 2025-09-07T07:06:55.8260660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8260732Z return func(*args, **kwargs) 2025-09-07T07:06:55.8260824Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8261057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8261160Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8261450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8261530Z layer_outputs = layer_module( 2025-09-07T07:06:55.8261774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8261859Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8262131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8262203Z return func(*args, **kwargs) 2025-09-07T07:06:55.8262469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8262550Z return func(*args, **kwargs) 2025-09-07T07:06:55.8262818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8262899Z return func(*args, **kwargs) 2025-09-07T07:06:55.8263216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8263308Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8263598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8263682Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8264011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8264144Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8264480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.8264570Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8264576Z 2025-09-07T07:06:55.8264689Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8264915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8264987Z return mod(**inputs) 2025-09-07T07:06:55.8265230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8265310Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8265613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8265799Z outputs = self.layoutlm( 2025-09-07T07:06:55.8266087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8266174Z return func(*args, **kwargs) 2025-09-07T07:06:55.8266460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8266549Z return func(*args, **kwargs) 2025-09-07T07:06:55.8266791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8266873Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8267179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8267262Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8267547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8267622Z return func(*args, **kwargs) 2025-09-07T07:06:55.8267898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8267983Z return func(*args, **kwargs) 2025-09-07T07:06:55.8268281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8268365Z return func(*args, **kwargs) 2025-09-07T07:06:55.8268449Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8268684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8268772Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8269061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8269144Z layer_outputs = layer_module( 2025-09-07T07:06:55.8269397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8269482Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8269750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8269825Z return func(*args, **kwargs) 2025-09-07T07:06:55.8270109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8270183Z return func(*args, **kwargs) 2025-09-07T07:06:55.8270444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8270518Z return func(*args, **kwargs) 2025-09-07T07:06:55.8270805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8270905Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8271189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8271297Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8271625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8271757Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8272054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.8272180Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.8272416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.8272492Z return self.act(input) 2025-09-07T07:06:55.8272496Z 2025-09-07T07:06:55.8272632Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8272848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8272921Z return mod(**inputs) 2025-09-07T07:06:55.8273165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8273249Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8273548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8273623Z outputs = self.layoutlm( 2025-09-07T07:06:55.8273880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8273960Z return func(*args, **kwargs) 2025-09-07T07:06:55.8274218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8274299Z return func(*args, **kwargs) 2025-09-07T07:06:55.8274532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8274612Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8274926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8275007Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8275271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8275344Z return func(*args, **kwargs) 2025-09-07T07:06:55.8275607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8275679Z return func(*args, **kwargs) 2025-09-07T07:06:55.8275938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8276017Z return func(*args, **kwargs) 2025-09-07T07:06:55.8276103Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8276350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8276431Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8276739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8276819Z layer_outputs = layer_module( 2025-09-07T07:06:55.8277040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8277127Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8277387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8277460Z return func(*args, **kwargs) 2025-09-07T07:06:55.8277726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8277814Z return func(*args, **kwargs) 2025-09-07T07:06:55.8278078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8278155Z return func(*args, **kwargs) 2025-09-07T07:06:55.8278442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8278542Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8278823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8278912Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8279256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.8279401Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.8279674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.8279761Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8279766Z 2025-09-07T07:06:55.8279880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8280083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8280159Z return mod(**inputs) 2025-09-07T07:06:55.8280379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8280455Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8280737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8280810Z outputs = self.layoutlm( 2025-09-07T07:06:55.8281063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8281133Z return func(*args, **kwargs) 2025-09-07T07:06:55.8281400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8281472Z return func(*args, **kwargs) 2025-09-07T07:06:55.8281695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8281779Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8282057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8282143Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8282403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8282475Z return func(*args, **kwargs) 2025-09-07T07:06:55.8282744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8282821Z return func(*args, **kwargs) 2025-09-07T07:06:55.8283087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8283205Z return func(*args, **kwargs) 2025-09-07T07:06:55.8283290Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8283533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8283613Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8283909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8283987Z layer_outputs = layer_module( 2025-09-07T07:06:55.8284232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8284335Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8284597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8284677Z return func(*args, **kwargs) 2025-09-07T07:06:55.8284934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8285014Z return func(*args, **kwargs) 2025-09-07T07:06:55.8285276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8285349Z return func(*args, **kwargs) 2025-09-07T07:06:55.8285659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8285752Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8286017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8286091Z return func(*args, **kwargs) 2025-09-07T07:06:55.8286347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8286431Z return func(*args, **kwargs) 2025-09-07T07:06:55.8286691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8286770Z return func(*args, **kwargs) 2025-09-07T07:06:55.8287059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8287136Z self_outputs = self.self( 2025-09-07T07:06:55.8287403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8287475Z return func(*args, **kwargs) 2025-09-07T07:06:55.8287741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8287832Z return func(*args, **kwargs) 2025-09-07T07:06:55.8288099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8288173Z return func(*args, **kwargs) 2025-09-07T07:06:55.8288463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.8288633Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8288637Z 2025-09-07T07:06:55.8288749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8288975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8289048Z return mod(**inputs) 2025-09-07T07:06:55.8289288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8289378Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8289672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8289784Z outputs = self.layoutlm( 2025-09-07T07:06:55.8290044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8290123Z return func(*args, **kwargs) 2025-09-07T07:06:55.8290380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8290453Z return func(*args, **kwargs) 2025-09-07T07:06:55.8290704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8290785Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8291099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8291182Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8291439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8291517Z return func(*args, **kwargs) 2025-09-07T07:06:55.8291775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8291854Z return func(*args, **kwargs) 2025-09-07T07:06:55.8292107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8292178Z return func(*args, **kwargs) 2025-09-07T07:06:55.8292285Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8292518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8292605Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8292896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8292974Z layer_outputs = layer_module( 2025-09-07T07:06:55.8293221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8293306Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8293570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8293645Z return func(*args, **kwargs) 2025-09-07T07:06:55.8293909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8293981Z return func(*args, **kwargs) 2025-09-07T07:06:55.8294241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8294341Z return func(*args, **kwargs) 2025-09-07T07:06:55.8294634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8294733Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8294995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8295066Z return func(*args, **kwargs) 2025-09-07T07:06:55.8295338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8295409Z return func(*args, **kwargs) 2025-09-07T07:06:55.8295687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8295760Z return func(*args, **kwargs) 2025-09-07T07:06:55.8296060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8296145Z self_outputs = self.self( 2025-09-07T07:06:55.8296426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8296505Z return func(*args, **kwargs) 2025-09-07T07:06:55.8296762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8296843Z return func(*args, **kwargs) 2025-09-07T07:06:55.8297099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8297171Z return func(*args, **kwargs) 2025-09-07T07:06:55.8297470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.8297638Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8297644Z 2025-09-07T07:06:55.8297768Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8297988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8298061Z return mod(**inputs) 2025-09-07T07:06:55.8298305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8298387Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8298686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8298762Z outputs = self.layoutlm( 2025-09-07T07:06:55.8299037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8299120Z return func(*args, **kwargs) 2025-09-07T07:06:55.8299382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8299462Z return func(*args, **kwargs) 2025-09-07T07:06:55.8299702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8299791Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8300086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8300163Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8300431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8300504Z return func(*args, **kwargs) 2025-09-07T07:06:55.8300768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8300839Z return func(*args, **kwargs) 2025-09-07T07:06:55.8301099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8301199Z return func(*args, **kwargs) 2025-09-07T07:06:55.8301283Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8301532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8301612Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8301921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8302008Z layer_outputs = layer_module( 2025-09-07T07:06:55.8302257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8302352Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8302629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8302703Z return func(*args, **kwargs) 2025-09-07T07:06:55.8302984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8303076Z return func(*args, **kwargs) 2025-09-07T07:06:55.8303349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8303421Z return func(*args, **kwargs) 2025-09-07T07:06:55.8303740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8303832Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8304107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8304188Z return func(*args, **kwargs) 2025-09-07T07:06:55.8304477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8304561Z return func(*args, **kwargs) 2025-09-07T07:06:55.8304825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8304898Z return func(*args, **kwargs) 2025-09-07T07:06:55.8305212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8305289Z self_outputs = self.self( 2025-09-07T07:06:55.8305570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8305737Z return func(*args, **kwargs) 2025-09-07T07:06:55.8306023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8306108Z return func(*args, **kwargs) 2025-09-07T07:06:55.8306369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8306454Z return func(*args, **kwargs) 2025-09-07T07:06:55.8306762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.8306933Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8306938Z 2025-09-07T07:06:55.8307030Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8307118Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8307245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8307475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8307555Z return mod(**inputs) 2025-09-07T07:06:55.8307791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8307897Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8308206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8308282Z outputs = self.layoutlm( 2025-09-07T07:06:55.8308553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8308625Z return func(*args, **kwargs) 2025-09-07T07:06:55.8308889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8308969Z return func(*args, **kwargs) 2025-09-07T07:06:55.8309210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8309295Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8309596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8309684Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8309970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8310041Z return func(*args, **kwargs) 2025-09-07T07:06:55.8310308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8310378Z return func(*args, **kwargs) 2025-09-07T07:06:55.8310632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8310702Z return func(*args, **kwargs) 2025-09-07T07:06:55.8310780Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8311034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8311115Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8311414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8311493Z layer_outputs = layer_module( 2025-09-07T07:06:55.8311735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8311829Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8312086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8312166Z return func(*args, **kwargs) 2025-09-07T07:06:55.8312442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8312517Z return func(*args, **kwargs) 2025-09-07T07:06:55.8312786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8312861Z return func(*args, **kwargs) 2025-09-07T07:06:55.8313157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8313249Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8313517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8313590Z return func(*args, **kwargs) 2025-09-07T07:06:55.8313846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8313926Z return func(*args, **kwargs) 2025-09-07T07:06:55.8314186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8314265Z return func(*args, **kwargs) 2025-09-07T07:06:55.8314557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.8314729Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.8315030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.8315122Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8315126Z 2025-09-07T07:06:55.8315246Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8315463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8315535Z return mod(**inputs) 2025-09-07T07:06:55.8315779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8315861Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8316158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8316234Z outputs = self.layoutlm( 2025-09-07T07:06:55.8316516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8316589Z return func(*args, **kwargs) 2025-09-07T07:06:55.8316845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8316958Z return func(*args, **kwargs) 2025-09-07T07:06:55.8317299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8317405Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8317698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8317793Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8318063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8318138Z return func(*args, **kwargs) 2025-09-07T07:06:55.8318405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8318481Z return func(*args, **kwargs) 2025-09-07T07:06:55.8318741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8318822Z return func(*args, **kwargs) 2025-09-07T07:06:55.8318909Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8319177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8319257Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8319706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8319826Z layer_outputs = layer_module( 2025-09-07T07:06:55.8320125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8320222Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8320483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8320565Z return func(*args, **kwargs) 2025-09-07T07:06:55.8320821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8320894Z return func(*args, **kwargs) 2025-09-07T07:06:55.8321159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8321232Z return func(*args, **kwargs) 2025-09-07T07:06:55.8321528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8321677Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8321961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8322052Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8322377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8322518Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8322807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.8322905Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8322909Z 2025-09-07T07:06:55.8323022Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8323239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8323324Z return mod(**inputs) 2025-09-07T07:06:55.8323605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8323693Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8323985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8324061Z outputs = self.layoutlm( 2025-09-07T07:06:55.8324329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8324402Z return func(*args, **kwargs) 2025-09-07T07:06:55.8324667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8324764Z return func(*args, **kwargs) 2025-09-07T07:06:55.8324997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8325085Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8325376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8325473Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8325717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8325792Z return func(*args, **kwargs) 2025-09-07T07:06:55.8326033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8326130Z return func(*args, **kwargs) 2025-09-07T07:06:55.8326381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8326449Z return func(*args, **kwargs) 2025-09-07T07:06:55.8326538Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8326770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8326851Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8327147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8327224Z layer_outputs = layer_module( 2025-09-07T07:06:55.8327469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8327554Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8327811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8327889Z return func(*args, **kwargs) 2025-09-07T07:06:55.8328147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8328251Z return func(*args, **kwargs) 2025-09-07T07:06:55.8328512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8328591Z return func(*args, **kwargs) 2025-09-07T07:06:55.8328884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8328975Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8329269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8329355Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8329691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8329823Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8330117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.8330270Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.8330498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.8330582Z return self.act(input) 2025-09-07T07:06:55.8330586Z 2025-09-07T07:06:55.8330699Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8330923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8330997Z return mod(**inputs) 2025-09-07T07:06:55.8331230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8331336Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8331625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8331710Z outputs = self.layoutlm( 2025-09-07T07:06:55.8331970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8332041Z return func(*args, **kwargs) 2025-09-07T07:06:55.8332304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8332376Z return func(*args, **kwargs) 2025-09-07T07:06:55.8332629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8332727Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8333020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8333109Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8333367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8333449Z return func(*args, **kwargs) 2025-09-07T07:06:55.8333703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8333774Z return func(*args, **kwargs) 2025-09-07T07:06:55.8334036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8334106Z return func(*args, **kwargs) 2025-09-07T07:06:55.8334197Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8334432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8334518Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8334807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8334905Z layer_outputs = layer_module( 2025-09-07T07:06:55.8335153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8335240Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8335510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8335584Z return func(*args, **kwargs) 2025-09-07T07:06:55.8335844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8335927Z return func(*args, **kwargs) 2025-09-07T07:06:55.8336190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8336272Z return func(*args, **kwargs) 2025-09-07T07:06:55.8336567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8336678Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8336968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8337051Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8337378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.8337523Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.8337822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.8337905Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8337908Z 2025-09-07T07:06:55.8338034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8338245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8338314Z return mod(**inputs) 2025-09-07T07:06:55.8338535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8338607Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8338870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8338947Z outputs = self.layoutlm( 2025-09-07T07:06:55.8339208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8339285Z return func(*args, **kwargs) 2025-09-07T07:06:55.8339527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8339597Z return func(*args, **kwargs) 2025-09-07T07:06:55.8339823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8339901Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8340177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8340258Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8340522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8340597Z return func(*args, **kwargs) 2025-09-07T07:06:55.8340861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8340944Z return func(*args, **kwargs) 2025-09-07T07:06:55.8341206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8341302Z return func(*args, **kwargs) 2025-09-07T07:06:55.8341385Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8341622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8341711Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8342009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8342095Z layer_outputs = layer_module( 2025-09-07T07:06:55.8342344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8342431Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8342695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8342768Z return func(*args, **kwargs) 2025-09-07T07:06:55.8343054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8343149Z return func(*args, **kwargs) 2025-09-07T07:06:55.8343430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8343514Z return func(*args, **kwargs) 2025-09-07T07:06:55.8343816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8343913Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8344183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8344263Z return func(*args, **kwargs) 2025-09-07T07:06:55.8344548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8344621Z return func(*args, **kwargs) 2025-09-07T07:06:55.8344904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8344979Z return func(*args, **kwargs) 2025-09-07T07:06:55.8345286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8345363Z self_outputs = self.self( 2025-09-07T07:06:55.8345630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8345786Z return func(*args, **kwargs) 2025-09-07T07:06:55.8346083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8346168Z return func(*args, **kwargs) 2025-09-07T07:06:55.8346441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8346517Z return func(*args, **kwargs) 2025-09-07T07:06:55.8346821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-09-07T07:06:55.8346974Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8346978Z 2025-09-07T07:06:55.8347092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8347293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8347369Z return mod(**inputs) 2025-09-07T07:06:55.8347591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8347667Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8347950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8348041Z outputs = self.layoutlm( 2025-09-07T07:06:55.8348294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8348364Z return func(*args, **kwargs) 2025-09-07T07:06:55.8348604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8348682Z return func(*args, **kwargs) 2025-09-07T07:06:55.8348900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8348983Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8349251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8349329Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8349571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8349639Z return func(*args, **kwargs) 2025-09-07T07:06:55.8349884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8349965Z return func(*args, **kwargs) 2025-09-07T07:06:55.8350211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8350278Z return func(*args, **kwargs) 2025-09-07T07:06:55.8350356Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8350577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8350653Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8350942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8351014Z layer_outputs = layer_module( 2025-09-07T07:06:55.8351237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8351326Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8351567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8351641Z return func(*args, **kwargs) 2025-09-07T07:06:55.8351880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8351945Z return func(*args, **kwargs) 2025-09-07T07:06:55.8352260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8352329Z return func(*args, **kwargs) 2025-09-07T07:06:55.8352607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8352695Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8352948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8353017Z return func(*args, **kwargs) 2025-09-07T07:06:55.8353254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8353331Z return func(*args, **kwargs) 2025-09-07T07:06:55.8353579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8353653Z return func(*args, **kwargs) 2025-09-07T07:06:55.8353916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8353987Z self_outputs = self.self( 2025-09-07T07:06:55.8354231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8354314Z return func(*args, **kwargs) 2025-09-07T07:06:55.8354556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8354624Z return func(*args, **kwargs) 2025-09-07T07:06:55.8354867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8354942Z return func(*args, **kwargs) 2025-09-07T07:06:55.8355211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-09-07T07:06:55.8355364Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8355368Z 2025-09-07T07:06:55.8355477Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8355687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8355756Z return mod(**inputs) 2025-09-07T07:06:55.8355974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8356076Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8356353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8356431Z outputs = self.layoutlm( 2025-09-07T07:06:55.8356675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8356743Z return func(*args, **kwargs) 2025-09-07T07:06:55.8356992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8357060Z return func(*args, **kwargs) 2025-09-07T07:06:55.8357302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8357381Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8357664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8357747Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8357999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8358076Z return func(*args, **kwargs) 2025-09-07T07:06:55.8358325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8358420Z return func(*args, **kwargs) 2025-09-07T07:06:55.8358664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8358733Z return func(*args, **kwargs) 2025-09-07T07:06:55.8358821Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8359043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8359129Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8359403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8359475Z layer_outputs = layer_module( 2025-09-07T07:06:55.8359707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8359787Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8360040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8360108Z return func(*args, **kwargs) 2025-09-07T07:06:55.8360352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8360452Z return func(*args, **kwargs) 2025-09-07T07:06:55.8360692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8360771Z return func(*args, **kwargs) 2025-09-07T07:06:55.8361044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8361130Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8361381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8361449Z return func(*args, **kwargs) 2025-09-07T07:06:55.8361699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8361767Z return func(*args, **kwargs) 2025-09-07T07:06:55.8362029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8362103Z return func(*args, **kwargs) 2025-09-07T07:06:55.8362413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-09-07T07:06:55.8362499Z self_outputs = self.self( 2025-09-07T07:06:55.8362757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8362832Z return func(*args, **kwargs) 2025-09-07T07:06:55.8363078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8363148Z return func(*args, **kwargs) 2025-09-07T07:06:55.8363402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8363493Z return func(*args, **kwargs) 2025-09-07T07:06:55.8363774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-09-07T07:06:55.8363929Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-09-07T07:06:55.8363932Z 2025-09-07T07:06:55.8364023Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8364105Z cudagraph partition due to non gpu ops 2025-09-07T07:06:55.8364211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8364423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8364490Z return mod(**inputs) 2025-09-07T07:06:55.8364735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8364814Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8365089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8365170Z outputs = self.layoutlm( 2025-09-07T07:06:55.8365415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8365492Z return func(*args, **kwargs) 2025-09-07T07:06:55.8365735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8365803Z return func(*args, **kwargs) 2025-09-07T07:06:55.8366032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8366108Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8366388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8366462Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8366707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8366800Z return func(*args, **kwargs) 2025-09-07T07:06:55.8367048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8367123Z return func(*args, **kwargs) 2025-09-07T07:06:55.8367367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8367444Z return func(*args, **kwargs) 2025-09-07T07:06:55.8367521Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8367741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8367835Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8368102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8368178Z layer_outputs = layer_module( 2025-09-07T07:06:55.8368397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8368494Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8368744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8368812Z return func(*args, **kwargs) 2025-09-07T07:06:55.8369058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8369126Z return func(*args, **kwargs) 2025-09-07T07:06:55.8369364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8369439Z return func(*args, **kwargs) 2025-09-07T07:06:55.8369722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-09-07T07:06:55.8369816Z self_attention_outputs = self.attention( 2025-09-07T07:06:55.8370057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8370127Z return func(*args, **kwargs) 2025-09-07T07:06:55.8370383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8370449Z return func(*args, **kwargs) 2025-09-07T07:06:55.8370690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8370755Z return func(*args, **kwargs) 2025-09-07T07:06:55.8371044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-09-07T07:06:55.8371177Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:06:55.8371442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-09-07T07:06:55.8371536Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8371540Z 2025-09-07T07:06:55.8371647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8371858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8371926Z return mod(**inputs) 2025-09-07T07:06:55.8372147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8372229Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8372505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8372583Z outputs = self.layoutlm( 2025-09-07T07:06:55.8372827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8372922Z return func(*args, **kwargs) 2025-09-07T07:06:55.8373163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8373233Z return func(*args, **kwargs) 2025-09-07T07:06:55.8373460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8373544Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8373815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8373887Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8374123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8374199Z return func(*args, **kwargs) 2025-09-07T07:06:55.8374434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8374511Z return func(*args, **kwargs) 2025-09-07T07:06:55.8374762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8374828Z return func(*args, **kwargs) 2025-09-07T07:06:55.8374914Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8375128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8375209Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8375478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8375551Z layer_outputs = layer_module( 2025-09-07T07:06:55.8375790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8375874Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8376130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8376201Z return func(*args, **kwargs) 2025-09-07T07:06:55.8376457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8376526Z return func(*args, **kwargs) 2025-09-07T07:06:55.8376770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8376845Z return func(*args, **kwargs) 2025-09-07T07:06:55.8377153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8377246Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8377507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8377586Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8377892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8378013Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8378284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-09-07T07:06:55.8378367Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8378370Z 2025-09-07T07:06:55.8378479Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8378679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8378745Z return mod(**inputs) 2025-09-07T07:06:55.8378965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8379055Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8379325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8379393Z outputs = self.layoutlm( 2025-09-07T07:06:55.8379638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8379713Z return func(*args, **kwargs) 2025-09-07T07:06:55.8379953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8380031Z return func(*args, **kwargs) 2025-09-07T07:06:55.8380251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8380328Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8380609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8380686Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8380955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8381025Z return func(*args, **kwargs) 2025-09-07T07:06:55.8381266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8381342Z return func(*args, **kwargs) 2025-09-07T07:06:55.8381582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8381659Z return func(*args, **kwargs) 2025-09-07T07:06:55.8381738Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8381977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8382053Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8382331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8382415Z layer_outputs = layer_module( 2025-09-07T07:06:55.8382656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8382748Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8383003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8383074Z return func(*args, **kwargs) 2025-09-07T07:06:55.8383353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8383427Z return func(*args, **kwargs) 2025-09-07T07:06:55.8383690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8383763Z return func(*args, **kwargs) 2025-09-07T07:06:55.8384066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8384163Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8384444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8384535Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8384859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-09-07T07:06:55.8385000Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:06:55.8385312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-09-07T07:06:55.8385433Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:06:55.8385828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:06:55.8385914Z return self.act(input) 2025-09-07T07:06:55.8385918Z 2025-09-07T07:06:55.8386039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8386310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8386384Z return mod(**inputs) 2025-09-07T07:06:55.8386635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8386720Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8387036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8387114Z outputs = self.layoutlm( 2025-09-07T07:06:55.8387382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8387457Z return func(*args, **kwargs) 2025-09-07T07:06:55.8387747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8387827Z return func(*args, **kwargs) 2025-09-07T07:06:55.8388060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8388148Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8388438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-09-07T07:06:55.8388519Z encoder_outputs = self.encoder( 2025-09-07T07:06:55.8388820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8388895Z return func(*args, **kwargs) 2025-09-07T07:06:55.8389163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8389237Z return func(*args, **kwargs) 2025-09-07T07:06:55.8389509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8389589Z return func(*args, **kwargs) 2025-09-07T07:06:55.8389674Z [Previous line repeated 1 more time] 2025-09-07T07:06:55.8389914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8389992Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8390298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-09-07T07:06:55.8390384Z layer_outputs = layer_module( 2025-09-07T07:06:55.8390621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:06:55.8390714Z return super().__call__(*args, **kwargs) 2025-09-07T07:06:55.8390972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8391053Z return func(*args, **kwargs) 2025-09-07T07:06:55.8391319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8391391Z return func(*args, **kwargs) 2025-09-07T07:06:55.8391659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8391732Z return func(*args, **kwargs) 2025-09-07T07:06:55.8392042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-09-07T07:06:55.8392136Z layer_output = apply_chunking_to_forward( 2025-09-07T07:06:55.8392415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:06:55.8392525Z return forward_fn(*input_tensors) 2025-09-07T07:06:55.8392847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-09-07T07:06:55.8393001Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:06:55.8393302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-09-07T07:06:55.8393409Z hidden_states = self.dense(hidden_states) 2025-09-07T07:06:55.8393413Z 2025-09-07T07:06:55.8393528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8393746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8393828Z return mod(**inputs) 2025-09-07T07:06:55.8394062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8394168Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8394460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8394535Z outputs = self.layoutlm( 2025-09-07T07:06:55.8394801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8394873Z return func(*args, **kwargs) 2025-09-07T07:06:55.8395140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8395209Z return func(*args, **kwargs) 2025-09-07T07:06:55.8395442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8395528Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8395822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-09-07T07:06:55.8395936Z pooled_output = self.pooler(sequence_output) 2025-09-07T07:06:55.8396238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 430, in forward 2025-09-07T07:06:55.8396345Z pooled_output = self.dense(first_token_tensor) 2025-09-07T07:06:55.8396349Z 2025-09-07T07:06:55.8396456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8396661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8396754Z return mod(**inputs) 2025-09-07T07:06:55.8396979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8397063Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8397338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-09-07T07:06:55.8397412Z outputs = self.layoutlm( 2025-09-07T07:06:55.8397664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8397733Z return func(*args, **kwargs) 2025-09-07T07:06:55.8397983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:06:55.8398051Z return func(*args, **kwargs) 2025-09-07T07:06:55.8398273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8398357Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8398634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-09-07T07:06:55.8398739Z pooled_output = self.pooler(sequence_output) 2025-09-07T07:06:55.8399028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 431, in forward 2025-09-07T07:06:55.8399137Z pooled_output = self.activation(pooled_output) 2025-09-07T07:06:55.8399141Z 2025-09-07T07:06:55.8399245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8399450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8399524Z return mod(**inputs) 2025-09-07T07:06:55.8399743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8399827Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8400098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 891, in forward 2025-09-07T07:06:55.8400185Z logits = self.classifier(pooled_output) 2025-09-07T07:06:55.8400190Z 2025-09-07T07:06:55.8400304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8400527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8400600Z return mod(**inputs) 2025-09-07T07:06:55.8400827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8400908Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8401188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-09-07T07:06:55.8401337Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-09-07T07:06:55.8401341Z 2025-09-07T07:06:55.8401462Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8401697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8401774Z return mod(**inputs) 2025-09-07T07:06:55.8402006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8402088Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8402383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-09-07T07:06:55.8402523Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-09-07T07:06:55.8402526Z 2025-09-07T07:06:55.8402642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:06:55.8402853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:06:55.8402946Z return mod(**inputs) 2025-09-07T07:06:55.8403181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:06:55.8403263Z output = func(self, *args, **kwargs) 2025-09-07T07:06:55.8403558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-09-07T07:06:55.8403694Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-09-07T07:06:55.8403698Z 2025-09-07T07:07:10.6222672Z Compilation time (from dynamo_timed): 22.051301275 2025-09-07T07:07:10.6223161Z pass 2025-09-07T07:07:10.6225903Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:07:10.6226878Z TIMING: _recursive_pre_grad_passes:0.01295 _recursive_joint_graph_passes:0.4521 _recursive_post_grad_passes:0.07399 async_compile.wait:0.67132 code_gen:10.22222 inductor_compile:11.46389 backend_compile:15.88334 gc:0.00328 entire_frame_compile:22.0513 total_wall_time:22.0513 2025-09-07T07:07:10.6227915Z STATS: call_* op count: 860 | FakeTensorMode.__torch_dispatch__:16775 | FakeTensor.__torch_dispatch__:4359 | ProxyTorchDispatchMode.__torch_dispatch__:5774 2025-09-07T07:07:10.6228471Z Dynamo produced 2 graphs covering 860 ops with 0 graph breaks (0 unique) 2025-09-07T07:07:13.4062137Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:07:13.4063091Z import pynvml # type: ignore[import] 2025-09-07T07:07:16.1958120Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:07:16.1959108Z from pkg_resources import resource_filename 2025-09-07T07:07:16.8590245Z 2025-09-07T07:07:23.6872414Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:07:23.6872709Z loading model: 0it [00:06, ?it/s] 2025-09-07T07:07:23.6895952Z cpu eval M2M100ForConditionalGeneration 2025-09-07T07:07:24.5254036Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:07:24.8583714Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:07:25.1884765Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:07:42.3287927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3288404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3288806Z return mod(**inputs) 2025-09-07T07:07:42.3289284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3290086Z outputs = self.model( 2025-09-07T07:07:42.3290512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3290963Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3291410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-09-07T07:07:42.3291893Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-09-07T07:07:42.3292339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-09-07T07:07:42.3292754Z return func(*args, **kwargs) 2025-09-07T07:07:42.3293239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-09-07T07:07:42.3293883Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-09-07T07:07:42.3294568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-09-07T07:07:42.3295072Z mask = input_ids.ne(padding_idx).int() 2025-09-07T07:07:42.3295238Z 2025-09-07T07:07:42.3295359Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3295760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3296119Z return mod(**inputs) 2025-09-07T07:07:42.3296521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3296946Z outputs = self.model( 2025-09-07T07:07:42.3297353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3297788Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3298218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-09-07T07:07:42.3298858Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-09-07T07:07:42.3299357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-09-07T07:07:42.3299746Z return func(*args, **kwargs) 2025-09-07T07:07:42.3300182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-09-07T07:07:42.3300759Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-09-07T07:07:42.3301395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-09-07T07:07:42.3301893Z mask = input_ids.ne(padding_idx).int() 2025-09-07T07:07:42.3302045Z 2025-09-07T07:07:42.3302138Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3302387Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3302615Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3302892Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3303109Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3303338Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3303566Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3303794Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3304014Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3304241Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3304468Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3304694Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3304951Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3305364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3305970Z return mod(**inputs) 2025-09-07T07:07:42.3306399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3306853Z outputs = self.model( 2025-09-07T07:07:42.3307274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3307735Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3308184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-09-07T07:07:42.3308674Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-09-07T07:07:42.3309137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-09-07T07:07:42.3309550Z return func(*args, **kwargs) 2025-09-07T07:07:42.3309981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-09-07T07:07:42.3310579Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-09-07T07:07:42.3311255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-09-07T07:07:42.3311892Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:07:42.3312166Z 2025-09-07T07:07:42.3312288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3312702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3313078Z return mod(**inputs) 2025-09-07T07:07:42.3313489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3313921Z outputs = self.model( 2025-09-07T07:07:42.3314330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3314796Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3315230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-09-07T07:07:42.3315711Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-09-07T07:07:42.3316156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-09-07T07:07:42.3316587Z return func(*args, **kwargs) 2025-09-07T07:07:42.3317003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-09-07T07:07:42.3317582Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-09-07T07:07:42.3318243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-09-07T07:07:42.3318872Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:07:42.3319220Z 2025-09-07T07:07:42.3319346Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3319917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3320308Z return mod(**inputs) 2025-09-07T07:07:42.3320732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3321169Z outputs = self.model( 2025-09-07T07:07:42.3321573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3321988Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3322452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3322889Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3323274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3323674Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3324113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3324562Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3325039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3325555Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3325777Z 2025-09-07T07:07:42.3325894Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3326286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3326638Z return mod(**inputs) 2025-09-07T07:07:42.3327038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3327474Z outputs = self.model( 2025-09-07T07:07:42.3327865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3328296Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3328741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3329184Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3329569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3329973Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3330447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3330911Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3331359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3331786Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3331946Z 2025-09-07T07:07:42.3332063Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3332461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3332823Z return mod(**inputs) 2025-09-07T07:07:42.3333223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3333646Z outputs = self.model( 2025-09-07T07:07:42.3334050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3334511Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3334928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3335350Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3335736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3336133Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3336563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3337003Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3337458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3337900Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3338060Z 2025-09-07T07:07:42.3338148Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3338384Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3338610Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3338826Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3339078Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3339474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3339858Z return mod(**inputs) 2025-09-07T07:07:42.3340576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3341006Z outputs = self.model( 2025-09-07T07:07:42.3341421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3341854Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3342284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3342717Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3343114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3343523Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3343957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3344411Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3344860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3345321Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3345947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3346549Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3346759Z 2025-09-07T07:07:42.3346886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3347286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3347652Z return mod(**inputs) 2025-09-07T07:07:42.3348064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3348498Z outputs = self.model( 2025-09-07T07:07:42.3348902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3349340Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3349765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3350196Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3350617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3351015Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3351456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3351915Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3352377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3352837Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3353346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3353870Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3354064Z 2025-09-07T07:07:42.3354181Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3354588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3354946Z return mod(**inputs) 2025-09-07T07:07:42.3355403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3355836Z outputs = self.model( 2025-09-07T07:07:42.3356236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3356657Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3357067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3357482Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3357862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3358257Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3358693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3359100Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3359511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3359916Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3360056Z 2025-09-07T07:07:42.3360171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3360553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3360924Z return mod(**inputs) 2025-09-07T07:07:42.3361315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3361759Z outputs = self.model( 2025-09-07T07:07:42.3362160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3362578Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3362992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3363388Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3363754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3364134Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3364535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3364985Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3365174Z 2025-09-07T07:07:42.3365281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3365676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3365999Z return mod(**inputs) 2025-09-07T07:07:42.3366373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3366766Z outputs = self.model( 2025-09-07T07:07:42.3367141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3367542Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3367970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3368420Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3368783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3369167Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3369568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3370008Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3370194Z 2025-09-07T07:07:42.3370301Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3370670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3371026Z return mod(**inputs) 2025-09-07T07:07:42.3371391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3371808Z outputs = self.model( 2025-09-07T07:07:42.3372201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3372639Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3373059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3373477Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3373851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3374248Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3374675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3375101Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3375253Z 2025-09-07T07:07:42.3375369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3375761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3376164Z return mod(**inputs) 2025-09-07T07:07:42.3376574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3377004Z outputs = self.model( 2025-09-07T07:07:42.3377416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3377857Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3378283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3378715Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3379107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3379521Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3379959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3380422Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3380861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3381353Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3381582Z 2025-09-07T07:07:42.3381693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3382103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3382456Z return mod(**inputs) 2025-09-07T07:07:42.3382849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3383274Z outputs = self.model( 2025-09-07T07:07:42.3383675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3384098Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3384511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3384931Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3385345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3385845Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3386317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3386788Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3387244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3387679Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3387836Z 2025-09-07T07:07:42.3387954Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3388347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3388695Z return mod(**inputs) 2025-09-07T07:07:42.3389086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3389508Z outputs = self.model( 2025-09-07T07:07:42.3389914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3390337Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3390753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3391179Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3391597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3392002Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3392437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3392880Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3393331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3393777Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3393936Z 2025-09-07T07:07:42.3394038Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3394283Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3394514Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3394748Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3395013Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3395421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3395800Z return mod(**inputs) 2025-09-07T07:07:42.3396210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3396643Z outputs = self.model( 2025-09-07T07:07:42.3397060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3397492Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3397927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3398356Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3398774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3399184Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3399611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3400063Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3400515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3400979Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3401498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3402055Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3402271Z 2025-09-07T07:07:42.3402392Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3402802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3403172Z return mod(**inputs) 2025-09-07T07:07:42.3403596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3404035Z outputs = self.model( 2025-09-07T07:07:42.3404437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3404864Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3405286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3405711Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3406113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3406534Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3406989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3407484Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3407918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3408385Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3408865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3409367Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3409542Z 2025-09-07T07:07:42.3409665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3410049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3410406Z return mod(**inputs) 2025-09-07T07:07:42.3410801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3411253Z outputs = self.model( 2025-09-07T07:07:42.3411650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3412083Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3412507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3412944Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3413340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3413741Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3414206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3414650Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3415082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3415521Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3415671Z 2025-09-07T07:07:42.3415783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3416179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3416545Z return mod(**inputs) 2025-09-07T07:07:42.3416961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3417381Z outputs = self.model( 2025-09-07T07:07:42.3417784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3418225Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3418642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3419060Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3419435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3419977Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3420414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3420898Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3421087Z 2025-09-07T07:07:42.3421213Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3421607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3421995Z return mod(**inputs) 2025-09-07T07:07:42.3422401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3422896Z outputs = self.model( 2025-09-07T07:07:42.3423330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3423835Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3424264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3424695Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3425087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3425487Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3426012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3426507Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3426704Z 2025-09-07T07:07:42.3426877Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3427285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3427645Z return mod(**inputs) 2025-09-07T07:07:42.3428055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3428484Z outputs = self.model( 2025-09-07T07:07:42.3428892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3429334Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3429790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3430224Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3430622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3431032Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3431459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3431900Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3432058Z 2025-09-07T07:07:42.3432174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3432575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3432981Z return mod(**inputs) 2025-09-07T07:07:42.3433382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3433818Z outputs = self.model( 2025-09-07T07:07:42.3434210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3434628Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3435032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3435449Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3435831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3436223Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3436650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-09-07T07:07:42.3437070Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3437224Z 2025-09-07T07:07:42.3437337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3437726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3438097Z return mod(**inputs) 2025-09-07T07:07:42.3438498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3438909Z outputs = self.model( 2025-09-07T07:07:42.3439306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3439743Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3440163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3440591Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3440978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3441378Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3441806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3442270Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3442693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3443203Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3443429Z 2025-09-07T07:07:42.3443542Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3443931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3444285Z return mod(**inputs) 2025-09-07T07:07:42.3444674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3445109Z outputs = self.model( 2025-09-07T07:07:42.3445507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3445932Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3446350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3446768Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3447148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3447540Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3447983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3448418Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3448855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3449282Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3449427Z 2025-09-07T07:07:42.3449551Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3449940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3450293Z return mod(**inputs) 2025-09-07T07:07:42.3450688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3451103Z outputs = self.model( 2025-09-07T07:07:42.3451494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3451925Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3452333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3452753Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3453993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3454371Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3454769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3455176Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3455575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3455974Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3456115Z 2025-09-07T07:07:42.3456207Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3456418Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3456634Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3456844Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3457084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3457445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3457803Z return mod(**inputs) 2025-09-07T07:07:42.3458179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3458575Z outputs = self.model( 2025-09-07T07:07:42.3458948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3459342Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3459736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3460132Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3460515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3460886Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3461287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3461717Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3462130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3462553Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3463010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3463558Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3463770Z 2025-09-07T07:07:42.3463883Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3464278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3464639Z return mod(**inputs) 2025-09-07T07:07:42.3465035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3465460Z outputs = self.model( 2025-09-07T07:07:42.3465981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3466419Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3466843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3467237Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3467594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3467965Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3468365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3468788Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3469193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3469602Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3470048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3470507Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3470669Z 2025-09-07T07:07:42.3470776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3471137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3471470Z return mod(**inputs) 2025-09-07T07:07:42.3471838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3472248Z outputs = self.model( 2025-09-07T07:07:42.3472609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3473000Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3473410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3473797Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3474144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3474512Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3474919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3475326Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3475726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3476116Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3476259Z 2025-09-07T07:07:42.3476363Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3476721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3477047Z return mod(**inputs) 2025-09-07T07:07:42.3477409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3477806Z outputs = self.model( 2025-09-07T07:07:42.3478181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3478554Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3478930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3479308Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3479657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3480016Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3480402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3480849Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3481026Z 2025-09-07T07:07:42.3481134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3481499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3481831Z return mod(**inputs) 2025-09-07T07:07:42.3482201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3482616Z outputs = self.model( 2025-09-07T07:07:42.3482983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3483380Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3483762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3484148Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3484489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3484851Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3485238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3485669Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3485841Z 2025-09-07T07:07:42.3485953Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3486342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3486685Z return mod(**inputs) 2025-09-07T07:07:42.3487066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3487470Z outputs = self.model( 2025-09-07T07:07:42.3487832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3488224Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3488665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3489054Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3489419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3489789Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3490191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3490603Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3490743Z 2025-09-07T07:07:42.3490859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3491222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3491541Z return mod(**inputs) 2025-09-07T07:07:42.3491925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3492310Z outputs = self.model( 2025-09-07T07:07:42.3492670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3493049Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3493434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3493816Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3494196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3494586Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3495002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3495441Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3495935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3496395Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3496619Z 2025-09-07T07:07:42.3496732Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3497084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3497412Z return mod(**inputs) 2025-09-07T07:07:42.3497779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3498162Z outputs = self.model( 2025-09-07T07:07:42.3498530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3498954Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3499375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3499793Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3500173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3500584Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3501009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3501449Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3501889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3502322Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3502470Z 2025-09-07T07:07:42.3502585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3502978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3503351Z return mod(**inputs) 2025-09-07T07:07:42.3503745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3504166Z outputs = self.model( 2025-09-07T07:07:42.3504561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3504985Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3505404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3505954Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3506350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3506760Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3507202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3507660Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3508099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3508529Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3508691Z 2025-09-07T07:07:42.3508778Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3509014Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3509241Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3509459Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3509711Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3510104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3510453Z return mod(**inputs) 2025-09-07T07:07:42.3510844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3511281Z outputs = self.model( 2025-09-07T07:07:42.3511674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3512095Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3512509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3512924Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3513301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3513690Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3514114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3514554Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3514944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3515356Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3515827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3516310Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3516492Z 2025-09-07T07:07:42.3516603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3516957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3517284Z return mod(**inputs) 2025-09-07T07:07:42.3517654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3518044Z outputs = self.model( 2025-09-07T07:07:42.3518422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3518817Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3519205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3519765Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3520131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3520490Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3520950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3521387Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3521796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3522220Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3522680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3523162Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3523340Z 2025-09-07T07:07:42.3523447Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3523828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3524158Z return mod(**inputs) 2025-09-07T07:07:42.3524521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3524911Z outputs = self.model( 2025-09-07T07:07:42.3525289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3525694Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3526123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3526517Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3526882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3527246Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3527635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3528032Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3528446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3528849Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3528991Z 2025-09-07T07:07:42.3529107Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3529476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3529838Z return mod(**inputs) 2025-09-07T07:07:42.3530215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3530615Z outputs = self.model( 2025-09-07T07:07:42.3530978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3531360Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3531749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3532144Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3532539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3532910Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3533301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3533747Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3533933Z 2025-09-07T07:07:42.3534039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3534410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3534742Z return mod(**inputs) 2025-09-07T07:07:42.3535127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3535523Z outputs = self.model( 2025-09-07T07:07:42.3535907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3536307Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3536693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3537090Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3537448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3537819Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3538216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3538651Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3538834Z 2025-09-07T07:07:42.3538943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3539311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3539646Z return mod(**inputs) 2025-09-07T07:07:42.3540012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3540435Z outputs = self.model( 2025-09-07T07:07:42.3540813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3541211Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3541606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3541993Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3542375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3542770Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3543196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3543623Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3543770Z 2025-09-07T07:07:42.3543881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3544304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3544672Z return mod(**inputs) 2025-09-07T07:07:42.3545062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3545470Z outputs = self.model( 2025-09-07T07:07:42.3545988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3546451Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3546885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3547344Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3547704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3548108Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3548566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-09-07T07:07:42.3549022Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3549173Z 2025-09-07T07:07:42.3549295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3549691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3550058Z return mod(**inputs) 2025-09-07T07:07:42.3550484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3550912Z outputs = self.model( 2025-09-07T07:07:42.3551317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3551753Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3552182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3552614Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3553002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3553398Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3553837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3554291Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3554720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3555180Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3555407Z 2025-09-07T07:07:42.3555515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3555876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3556201Z return mod(**inputs) 2025-09-07T07:07:42.3556563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3556942Z outputs = self.model( 2025-09-07T07:07:42.3557308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3557699Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3558082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3558471Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3558816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3559199Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3559586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3559997Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3560407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3560802Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3560943Z 2025-09-07T07:07:42.3561050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3561413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3561762Z return mod(**inputs) 2025-09-07T07:07:42.3562116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3562499Z outputs = self.model( 2025-09-07T07:07:42.3562861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3563250Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3563631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3564010Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3564379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3564739Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3565127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3565527Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3565920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3566321Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3566469Z 2025-09-07T07:07:42.3566550Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3566764Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3566968Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3567179Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3567418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3567792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3568127Z return mod(**inputs) 2025-09-07T07:07:42.3568497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3568920Z outputs = self.model( 2025-09-07T07:07:42.3569294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3569692Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3570086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3570510Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3570894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3571290Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3571715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3572151Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3572583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3573024Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3573541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3574065Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3574265Z 2025-09-07T07:07:42.3574379Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3574767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3575121Z return mod(**inputs) 2025-09-07T07:07:42.3575520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3575930Z outputs = self.model( 2025-09-07T07:07:42.3576346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3576772Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3577189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3577609Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3578027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3578424Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3578846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3579345Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3579835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3580286Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3580774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3581280Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3581460Z 2025-09-07T07:07:42.3581585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3581985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3582341Z return mod(**inputs) 2025-09-07T07:07:42.3582748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3583175Z outputs = self.model( 2025-09-07T07:07:42.3583586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3584008Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3584443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3584860Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3585238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3585629Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3586271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3586713Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3587145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3587550Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3587691Z 2025-09-07T07:07:42.3587810Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3588190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3588585Z return mod(**inputs) 2025-09-07T07:07:42.3588995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3589423Z outputs = self.model( 2025-09-07T07:07:42.3589802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3590208Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3590614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3591018Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3591412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3591767Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3592161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3592604Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3592791Z 2025-09-07T07:07:42.3592911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3593269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3593590Z return mod(**inputs) 2025-09-07T07:07:42.3593962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3594373Z outputs = self.model( 2025-09-07T07:07:42.3594750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3595140Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3595537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3595937Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3596288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3596656Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3597048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3597489Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3597671Z 2025-09-07T07:07:42.3597780Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3598147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3598474Z return mod(**inputs) 2025-09-07T07:07:42.3598847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3599274Z outputs = self.model( 2025-09-07T07:07:42.3599643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3600038Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3600420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3600812Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3601168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3601541Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3601934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3602328Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3602475Z 2025-09-07T07:07:42.3602580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3602984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3603322Z return mod(**inputs) 2025-09-07T07:07:42.3603689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3604077Z outputs = self.model( 2025-09-07T07:07:42.3604454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3604851Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3605245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3605650Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3606017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3606394Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3606795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3607218Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3607627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3608098Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3608317Z 2025-09-07T07:07:42.3608440Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3608816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3609150Z return mod(**inputs) 2025-09-07T07:07:42.3609527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3609925Z outputs = self.model( 2025-09-07T07:07:42.3610303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3610704Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3611094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3611496Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3611855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3612226Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3612631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3613065Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3613480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3613886Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3614023Z 2025-09-07T07:07:42.3614137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3614515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3614832Z return mod(**inputs) 2025-09-07T07:07:42.3615190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3615571Z outputs = self.model( 2025-09-07T07:07:42.3615934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3616316Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3616699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3617104Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3617454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3617825Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3618211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3618620Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3619029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3619433Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3619813Z 2025-09-07T07:07:42.3619910Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3620122Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3620333Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3620549Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3620783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3621142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3621476Z return mod(**inputs) 2025-09-07T07:07:42.3621855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3622255Z outputs = self.model( 2025-09-07T07:07:42.3622656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3623055Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3623453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3623852Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3624218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3624585Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3624993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3625417Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3625940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3626405Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3626887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3627400Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3627621Z 2025-09-07T07:07:42.3627724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3628081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3628397Z return mod(**inputs) 2025-09-07T07:07:42.3628764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3629148Z outputs = self.model( 2025-09-07T07:07:42.3629516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3629918Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3630315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3630741Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3631100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3631509Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3631911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3632320Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3632734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3633153Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3633614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3634078Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3634271Z 2025-09-07T07:07:42.3634378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3634750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3635085Z return mod(**inputs) 2025-09-07T07:07:42.3635457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3635848Z outputs = self.model( 2025-09-07T07:07:42.3636224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3636623Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3637032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3637423Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3637777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3638150Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3638556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3638982Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3639390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3639803Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3639952Z 2025-09-07T07:07:42.3640060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3640434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3640772Z return mod(**inputs) 2025-09-07T07:07:42.3641146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3641538Z outputs = self.model( 2025-09-07T07:07:42.3641939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3642336Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3642727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3643112Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3643474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3643848Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3644252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3644676Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3644857Z 2025-09-07T07:07:42.3644961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3645318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3645661Z return mod(**inputs) 2025-09-07T07:07:42.3646024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3646406Z outputs = self.model( 2025-09-07T07:07:42.3646773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3647173Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3647577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3647964Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3648331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3648706Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3649105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3649548Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3649721Z 2025-09-07T07:07:42.3649834Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3650196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3650528Z return mod(**inputs) 2025-09-07T07:07:42.3650935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3651327Z outputs = self.model( 2025-09-07T07:07:42.3651692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3652092Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3652487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3652880Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3653262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3653675Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3654106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3654561Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3654709Z 2025-09-07T07:07:42.3654831Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3655212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3655546Z return mod(**inputs) 2025-09-07T07:07:42.3655920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3656344Z outputs = self.model( 2025-09-07T07:07:42.3656716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3657105Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3657501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3657903Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3658266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3658640Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3659041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-09-07T07:07:42.3659452Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3659601Z 2025-09-07T07:07:42.3659710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3660104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3660430Z return mod(**inputs) 2025-09-07T07:07:42.3660821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3661238Z outputs = self.model( 2025-09-07T07:07:42.3661636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3662058Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3662483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3662923Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3663308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3663709Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3664134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3664566Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3665008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3665514Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3665854Z 2025-09-07T07:07:42.3666074Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3666701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3667269Z return mod(**inputs) 2025-09-07T07:07:42.3667883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3668549Z outputs = self.model( 2025-09-07T07:07:42.3669003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3669389Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3669780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3670172Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3670536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3670902Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3671298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3671767Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3672176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3672578Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3672718Z 2025-09-07T07:07:42.3672826Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3673198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3673534Z return mod(**inputs) 2025-09-07T07:07:42.3673910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3674299Z outputs = self.model( 2025-09-07T07:07:42.3674669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3675067Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3675461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3675884Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3676242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3676606Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3677009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3677425Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3677845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3678250Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3678403Z 2025-09-07T07:07:42.3678510Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3678740Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3678964Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3679185Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3679427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3679828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3680187Z return mod(**inputs) 2025-09-07T07:07:42.3680591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3681008Z outputs = self.model( 2025-09-07T07:07:42.3681435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3681844Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3682234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3682625Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3682975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3683356Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3683777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3684210Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3684636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3685080Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3685570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3686091Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3686314Z 2025-09-07T07:07:42.3686434Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3686823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3687178Z return mod(**inputs) 2025-09-07T07:07:42.3687578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3687998Z outputs = self.model( 2025-09-07T07:07:42.3688397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3688817Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3689234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3689653Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3690031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3690449Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3690867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3691306Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3691751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3692174Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3692628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3693099Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3693291Z 2025-09-07T07:07:42.3693400Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3693774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3694111Z return mod(**inputs) 2025-09-07T07:07:42.3694478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3694875Z outputs = self.model( 2025-09-07T07:07:42.3695251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3695648Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3696065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3696457Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3696697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3696779Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3697047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3697142Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3697401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3697494Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3697498Z 2025-09-07T07:07:42.3697605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3697816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3697884Z return mod(**inputs) 2025-09-07T07:07:42.3698143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3698221Z outputs = self.model( 2025-09-07T07:07:42.3698502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3698587Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3698846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3698925Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3699159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3699240Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3699509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3699634Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3699638Z 2025-09-07T07:07:42.3699760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3699980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3700068Z return mod(**inputs) 2025-09-07T07:07:42.3700360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3700431Z outputs = self.model( 2025-09-07T07:07:42.3700711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3700789Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3701071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3701147Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3701416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3701509Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3701784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3701921Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3701925Z 2025-09-07T07:07:42.3702036Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3702250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3702327Z return mod(**inputs) 2025-09-07T07:07:42.3702607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3702699Z outputs = self.model( 2025-09-07T07:07:42.3702987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3703066Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3703353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3703430Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3703678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3703763Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3704042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3704133Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3704137Z 2025-09-07T07:07:42.3704249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3704475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3704547Z return mod(**inputs) 2025-09-07T07:07:42.3704830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3704924Z outputs = self.model( 2025-09-07T07:07:42.3705214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3705300Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3705617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3705807Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3706226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3706358Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3706848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3706979Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3707424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3707770Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3707778Z 2025-09-07T07:07:42.3707939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3708151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3708219Z return mod(**inputs) 2025-09-07T07:07:42.3708500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3708568Z outputs = self.model( 2025-09-07T07:07:42.3708877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3708954Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3709222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3709296Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3709528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3709621Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3709894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3710000Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3710292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3710381Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3710387Z 2025-09-07T07:07:42.3710507Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3710732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3710809Z return mod(**inputs) 2025-09-07T07:07:42.3711068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3711136Z outputs = self.model( 2025-09-07T07:07:42.3711415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3711493Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3711773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3711850Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3712099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3712185Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3712475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3712580Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3712854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3712954Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3712958Z 2025-09-07T07:07:42.3713047Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3713133Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3713225Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3713307Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3713426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3713639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3713710Z return mod(**inputs) 2025-09-07T07:07:42.3713991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3714082Z outputs = self.model( 2025-09-07T07:07:42.3714362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3714441Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3714717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3714794Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3715036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3715128Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3715417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3715524Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3715796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3715901Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3716225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3716369Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3716373Z 2025-09-07T07:07:42.3716508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3716727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3716807Z return mod(**inputs) 2025-09-07T07:07:42.3717085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3717159Z outputs = self.model( 2025-09-07T07:07:42.3717442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3717521Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3717799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3717876Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3718113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3718206Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3718478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3718581Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3718868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3718974Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3719300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3719418Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3719421Z 2025-09-07T07:07:42.3719539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3719930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3720016Z return mod(**inputs) 2025-09-07T07:07:42.3720296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3720369Z outputs = self.model( 2025-09-07T07:07:42.3720656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3720788Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3721071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3721149Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3721386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3721481Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3721753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3721855Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3722148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3722247Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3722253Z 2025-09-07T07:07:42.3722365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3722580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3722658Z return mod(**inputs) 2025-09-07T07:07:42.3722931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3723010Z outputs = self.model( 2025-09-07T07:07:42.3723319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3723397Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3723677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3723756Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3724002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3724088Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3724357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3724498Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3724502Z 2025-09-07T07:07:42.3724605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3724817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3724884Z return mod(**inputs) 2025-09-07T07:07:42.3725148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3725216Z outputs = self.model( 2025-09-07T07:07:42.3725504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3725587Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3725843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3725921Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3726146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3726226Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3726491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3726613Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3726618Z 2025-09-07T07:07:42.3726731Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3726932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3727033Z return mod(**inputs) 2025-09-07T07:07:42.3727299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3727369Z outputs = self.model( 2025-09-07T07:07:42.3727654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3727731Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3728015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3728100Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3728344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3728435Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3728695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3728790Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3728793Z 2025-09-07T07:07:42.3728897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3729106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3729174Z return mod(**inputs) 2025-09-07T07:07:42.3729432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3729526Z outputs = self.model( 2025-09-07T07:07:42.3729787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3729868Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3730142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3730221Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3730467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3730551Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3730830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-09-07T07:07:42.3730924Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3730927Z 2025-09-07T07:07:42.3731046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3731276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3731344Z return mod(**inputs) 2025-09-07T07:07:42.3731608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3731693Z outputs = self.model( 2025-09-07T07:07:42.3731964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3732036Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3732289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3732367Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3732591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3732679Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3732940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3733032Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3733301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3733474Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3733478Z 2025-09-07T07:07:42.3733588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3733790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3733865Z return mod(**inputs) 2025-09-07T07:07:42.3734127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3734196Z outputs = self.model( 2025-09-07T07:07:42.3734483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3734558Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3734829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3734904Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3735139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3735231Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3735506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3735610Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3735898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3735995Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3736001Z 2025-09-07T07:07:42.3736110Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3736325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3736413Z return mod(**inputs) 2025-09-07T07:07:42.3736670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3736743Z outputs = self.model( 2025-09-07T07:07:42.3737008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3737086Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3737366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3737444Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3737690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3737773Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3738069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3738173Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3738442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3738541Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3738545Z 2025-09-07T07:07:42.3738631Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3738723Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3738808Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3738889Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3739004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3739217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3739297Z return mod(**inputs) 2025-09-07T07:07:42.3739571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3739663Z outputs = self.model( 2025-09-07T07:07:42.3739956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3740034Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3740309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3740386Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3740624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3740718Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3741003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3741112Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3741387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3741498Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3741816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3741960Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3741964Z 2025-09-07T07:07:42.3742100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3742315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3742395Z return mod(**inputs) 2025-09-07T07:07:42.3742668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3742742Z outputs = self.model( 2025-09-07T07:07:42.3743028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3743107Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3743385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3743462Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3743699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3743792Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3744064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3744168Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3744455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3744567Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3744882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3744999Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3745003Z 2025-09-07T07:07:42.3745120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3745342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3745419Z return mod(**inputs) 2025-09-07T07:07:42.3745783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3745888Z outputs = self.model( 2025-09-07T07:07:42.3746379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3746538Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3747041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3747147Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3747558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3747672Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3748141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3748285Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3748633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3748739Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3748748Z 2025-09-07T07:07:42.3748859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3749074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3749151Z return mod(**inputs) 2025-09-07T07:07:42.3749428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3749506Z outputs = self.model( 2025-09-07T07:07:42.3749783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3749868Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3750127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3750202Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3750438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3750520Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3750783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3750906Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3750910Z 2025-09-07T07:07:42.3751014Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3751227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3751307Z return mod(**inputs) 2025-09-07T07:07:42.3751558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3751625Z outputs = self.model( 2025-09-07T07:07:42.3751907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3751980Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3752222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3752298Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3752509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3752592Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3752833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3752949Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3752953Z 2025-09-07T07:07:42.3753062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3753265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3753355Z return mod(**inputs) 2025-09-07T07:07:42.3753617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3753681Z outputs = self.model( 2025-09-07T07:07:42.3753933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3754004Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3754258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3754328Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3754564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3754641Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3754887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3754977Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3754981Z 2025-09-07T07:07:42.3755083Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3755285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3755350Z return mod(**inputs) 2025-09-07T07:07:42.3755603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3755694Z outputs = self.model( 2025-09-07T07:07:42.3755946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3756029Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3756277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3756356Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3756575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3756655Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3756917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3757011Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3757276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3757429Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3757434Z 2025-09-07T07:07:42.3757539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3757762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3757831Z return mod(**inputs) 2025-09-07T07:07:42.3758098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3758167Z outputs = self.model( 2025-09-07T07:07:42.3758431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3758504Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3758762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3758843Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3759070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3759159Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3759415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3759524Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3759786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3759867Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3759871Z 2025-09-07T07:07:42.3759983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3760188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3760259Z return mod(**inputs) 2025-09-07T07:07:42.3760550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3760619Z outputs = self.model( 2025-09-07T07:07:42.3760881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3760956Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3761211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3761283Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3761504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3761600Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3761858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3761955Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3762199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3762283Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3762295Z 2025-09-07T07:07:42.3762374Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3762452Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3762536Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3762611Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3762711Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3762918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3762983Z return mod(**inputs) 2025-09-07T07:07:42.3763242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3763308Z outputs = self.model( 2025-09-07T07:07:42.3763567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3763660Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3763910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3763989Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3764210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3764296Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3764543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3764634Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3764892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3764991Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3765301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3765449Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3765453Z 2025-09-07T07:07:42.3765558Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3765752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3765817Z return mod(**inputs) 2025-09-07T07:07:42.3766082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3766150Z outputs = self.model( 2025-09-07T07:07:42.3766411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3766500Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3766752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3766832Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3767047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3767131Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3767383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3767473Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3767750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3767846Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3768142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3768252Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3768257Z 2025-09-07T07:07:42.3768365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3768563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3768626Z return mod(**inputs) 2025-09-07T07:07:42.3768893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3768958Z outputs = self.model( 2025-09-07T07:07:42.3769219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3769290Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3769546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3769644Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3769880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3769966Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3770221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3770317Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3770574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3770658Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3770661Z 2025-09-07T07:07:42.3770772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3770977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3771050Z return mod(**inputs) 2025-09-07T07:07:42.3771319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3771403Z outputs = self.model( 2025-09-07T07:07:42.3771664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3771738Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3772016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3772090Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3772321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3772409Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3772689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3772824Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3772829Z 2025-09-07T07:07:42.3772932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3773143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3773208Z return mod(**inputs) 2025-09-07T07:07:42.3773473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3773551Z outputs = self.model( 2025-09-07T07:07:42.3773893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3773986Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3774238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3774312Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3774539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3774620Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3774876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3774991Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3774995Z 2025-09-07T07:07:42.3775103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3775301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3775365Z return mod(**inputs) 2025-09-07T07:07:42.3775631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3775697Z outputs = self.model( 2025-09-07T07:07:42.3775972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3776045Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3776293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3776370Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3776589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3776673Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3776928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3777017Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3777021Z 2025-09-07T07:07:42.3777123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3777319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3777412Z return mod(**inputs) 2025-09-07T07:07:42.3777661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3777734Z outputs = self.model( 2025-09-07T07:07:42.3777985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3778056Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3778315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3778387Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3778628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3778708Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3778959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-09-07T07:07:42.3779047Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3779050Z 2025-09-07T07:07:42.3779151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3779360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3779426Z return mod(**inputs) 2025-09-07T07:07:42.3779692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3779778Z outputs = self.model( 2025-09-07T07:07:42.3780055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3780140Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3780413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3780501Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3780739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3780824Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3781103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3781210Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3781482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3781637Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3781642Z 2025-09-07T07:07:42.3781754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3781985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3782056Z return mod(**inputs) 2025-09-07T07:07:42.3782337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3782409Z outputs = self.model( 2025-09-07T07:07:42.3782689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3782766Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3783038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3783124Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3783366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3783457Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3783747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3783868Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3784143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3784228Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3784232Z 2025-09-07T07:07:42.3784351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3784570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3784648Z return mod(**inputs) 2025-09-07T07:07:42.3784941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3785014Z outputs = self.model( 2025-09-07T07:07:42.3785297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3785376Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3785655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3785839Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3786227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3786350Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3786874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3787027Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3787482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3787624Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3787633Z 2025-09-07T07:07:42.3787750Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3787868Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3787990Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3788097Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3788265Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3788494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3788565Z return mod(**inputs) 2025-09-07T07:07:42.3788852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3788928Z outputs = self.model( 2025-09-07T07:07:42.3789211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3789325Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3789599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3789685Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3789939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3790033Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3790317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3790423Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3790713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3790820Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3791151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3791323Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3791327Z 2025-09-07T07:07:42.3791444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3791660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3791731Z return mod(**inputs) 2025-09-07T07:07:42.3792014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3792089Z outputs = self.model( 2025-09-07T07:07:42.3792387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3792488Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3792776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3792857Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3793109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3793204Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3793475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3793581Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3793872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3793977Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3794300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3794420Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3794425Z 2025-09-07T07:07:42.3794541Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3794754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3794839Z return mod(**inputs) 2025-09-07T07:07:42.3795091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3795159Z outputs = self.model( 2025-09-07T07:07:42.3795420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3795494Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3795753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3795842Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3796063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3796150Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3796398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3796493Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3796742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3796824Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3796836Z 2025-09-07T07:07:42.3796937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3797136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3797209Z return mod(**inputs) 2025-09-07T07:07:42.3797460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3797554Z outputs = self.model( 2025-09-07T07:07:42.3797807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3797878Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3798133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3798202Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3798428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3798507Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3798781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3798908Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3798913Z 2025-09-07T07:07:42.3799015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3799216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3799279Z return mod(**inputs) 2025-09-07T07:07:42.3799537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3799605Z outputs = self.model( 2025-09-07T07:07:42.3799869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3799950Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3800200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3800278Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3800498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3800577Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3800834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3800953Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3800957Z 2025-09-07T07:07:42.3801063Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3801265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3801340Z return mod(**inputs) 2025-09-07T07:07:42.3801594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3801660Z outputs = self.model( 2025-09-07T07:07:42.3801932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3802005Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3802259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3802329Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3802547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3802633Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3802885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3802972Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3802976Z 2025-09-07T07:07:42.3803077Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3803272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3803362Z return mod(**inputs) 2025-09-07T07:07:42.3803611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3803686Z outputs = self.model( 2025-09-07T07:07:42.3803936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3804018Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3804266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3804336Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3804583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3804663Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3804925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3805016Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3805265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3805421Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3805424Z 2025-09-07T07:07:42.3805523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3805744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3805810Z return mod(**inputs) 2025-09-07T07:07:42.3806073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3806139Z outputs = self.model( 2025-09-07T07:07:42.3806391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3806473Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3806719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3806799Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3807021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3807099Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3807365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3807455Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3807725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3807844Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3807849Z 2025-09-07T07:07:42.3807958Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3808156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3808222Z return mod(**inputs) 2025-09-07T07:07:42.3808486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3808553Z outputs = self.model( 2025-09-07T07:07:42.3808826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3808906Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3809182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3809270Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3809506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3809618Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3809897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3809995Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3810275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3831306Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3831336Z 2025-09-07T07:07:42.3831537Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3831633Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3831869Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3831967Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3832094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3832324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3832403Z return mod(**inputs) 2025-09-07T07:07:42.3832703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3832788Z outputs = self.model( 2025-09-07T07:07:42.3833052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3833138Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3833456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3833537Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3833774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3833863Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3834131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3834232Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3834493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3834599Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3834890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3835031Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3835035Z 2025-09-07T07:07:42.3835144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3835389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3835458Z return mod(**inputs) 2025-09-07T07:07:42.3835713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3835786Z outputs = self.model( 2025-09-07T07:07:42.3836043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3836117Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3836373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3836445Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3836667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3836756Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3837002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3837136Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3837382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3837484Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3837769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3837881Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3837885Z 2025-09-07T07:07:42.3837996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3838208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3838284Z return mod(**inputs) 2025-09-07T07:07:42.3838536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3838608Z outputs = self.model( 2025-09-07T07:07:42.3838871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3838946Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3839211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3839282Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3839513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3839603Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3839852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-09-07T07:07:42.3839950Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:07:42.3840200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3840293Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3840297Z 2025-09-07T07:07:42.3840400Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3840601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3840676Z return mod(**inputs) 2025-09-07T07:07:42.3840940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3841015Z outputs = self.model( 2025-09-07T07:07:42.3841264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3841352Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3841601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3841675Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3841899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3841976Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3842219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3842351Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3842354Z 2025-09-07T07:07:42.3842455Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3842653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3842718Z return mod(**inputs) 2025-09-07T07:07:42.3842965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3843053Z outputs = self.model( 2025-09-07T07:07:42.3843304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3843380Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3843625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3843693Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3843921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3843999Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3844266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-09-07T07:07:42.3844386Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3844391Z 2025-09-07T07:07:42.3844498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3844690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3844753Z return mod(**inputs) 2025-09-07T07:07:42.3845005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3845069Z outputs = self.model( 2025-09-07T07:07:42.3845333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3845404Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3845648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3845726Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3845942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3846026Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3846270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-09-07T07:07:42.3846348Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3846359Z 2025-09-07T07:07:42.3846457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3846649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3846719Z return mod(**inputs) 2025-09-07T07:07:42.3846968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3847040Z outputs = self.model( 2025-09-07T07:07:42.3847305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-09-07T07:07:42.3847377Z encoder_outputs = self.encoder( 2025-09-07T07:07:42.3847629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-09-07T07:07:42.3847699Z layer_outputs = encoder_layer( 2025-09-07T07:07:42.3847922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3847998Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3848241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-09-07T07:07:42.3848328Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3848332Z 2025-09-07T07:07:42.3848431Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3848634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3848716Z return mod(**inputs) 2025-09-07T07:07:42.3848972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3849046Z outputs = self.model( 2025-09-07T07:07:42.3849304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3849382Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3849640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-09-07T07:07:42.3849820Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-09-07T07:07:42.3850075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-09-07T07:07:42.3850154Z return func(*args, **kwargs) 2025-09-07T07:07:42.3850428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-09-07T07:07:42.3850643Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-09-07T07:07:42.3850975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-09-07T07:07:42.3851172Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:07:42.3851176Z 2025-09-07T07:07:42.3851315Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3851519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3851588Z return mod(**inputs) 2025-09-07T07:07:42.3851863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3851935Z outputs = self.model( 2025-09-07T07:07:42.3852267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3852341Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3852598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-09-07T07:07:42.3852775Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-09-07T07:07:42.3853010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-09-07T07:07:42.3853088Z return func(*args, **kwargs) 2025-09-07T07:07:42.3853350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-09-07T07:07:42.3853598Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-09-07T07:07:42.3853922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-09-07T07:07:42.3854110Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:07:42.3854114Z 2025-09-07T07:07:42.3854227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3854432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3854505Z return mod(**inputs) 2025-09-07T07:07:42.3854770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3854848Z outputs = self.model( 2025-09-07T07:07:42.3855111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3855201Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3855469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3855543Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3855774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3855854Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3856114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3856228Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3856497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3856666Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3856673Z 2025-09-07T07:07:42.3856778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3856986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3857053Z return mod(**inputs) 2025-09-07T07:07:42.3857311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3857387Z outputs = self.model( 2025-09-07T07:07:42.3857948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3858034Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3858295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3858370Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3858606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3858688Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3858954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3859057Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3859803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3859886Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3859891Z 2025-09-07T07:07:42.3859996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3860207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3860275Z return mod(**inputs) 2025-09-07T07:07:42.3860560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3860630Z outputs = self.model( 2025-09-07T07:07:42.3860888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3860969Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3861226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3861306Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3861532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3861611Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3861873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3861976Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3862237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3862343Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3862346Z 2025-09-07T07:07:42.3862437Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3862521Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3862598Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3862681Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3862786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3863014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3863083Z return mod(**inputs) 2025-09-07T07:07:42.3863375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3863470Z outputs = self.model( 2025-09-07T07:07:42.3863730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3863814Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3864076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3864152Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3864399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3864499Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3864781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3864889Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3865168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3865277Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3865595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3865890Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3865899Z 2025-09-07T07:07:42.3866063Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3866413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3866519Z return mod(**inputs) 2025-09-07T07:07:42.3866975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3867085Z outputs = self.model( 2025-09-07T07:07:42.3867551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3867734Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3868179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3868286Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3868628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3868718Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3868986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3869087Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3869351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3869453Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3869747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3869911Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3869915Z 2025-09-07T07:07:42.3870019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3870229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3870294Z return mod(**inputs) 2025-09-07T07:07:42.3870564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3870633Z outputs = self.model( 2025-09-07T07:07:42.3870911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3870999Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3871258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3871344Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3871567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3871648Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3871909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3872008Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3872290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3872376Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3872380Z 2025-09-07T07:07:42.3872488Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3872691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3872759Z return mod(**inputs) 2025-09-07T07:07:42.3873021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3873090Z outputs = self.model( 2025-09-07T07:07:42.3873348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3873421Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3873678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3873759Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3873981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3874084Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3874335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3874448Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3874712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3874866Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3874870Z 2025-09-07T07:07:42.3874981Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3875181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3875254Z return mod(**inputs) 2025-09-07T07:07:42.3875513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3875584Z outputs = self.model( 2025-09-07T07:07:42.3875870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3875942Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3876206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3876278Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3876501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3876589Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3876846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3876978Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3877235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3877322Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3877326Z 2025-09-07T07:07:42.3877429Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3877639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3877716Z return mod(**inputs) 2025-09-07T07:07:42.3877989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3878074Z outputs = self.model( 2025-09-07T07:07:42.3878360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3878436Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3878700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3878774Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3879005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3879085Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3879340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3879459Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3879716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3879811Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3879815Z 2025-09-07T07:07:42.3879898Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3879987Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3880084Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3880161Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3880274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3880476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3880550Z return mod(**inputs) 2025-09-07T07:07:42.3880811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3880878Z outputs = self.model( 2025-09-07T07:07:42.3881146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3881219Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3881485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3881558Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3881787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3881895Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3882154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3882272Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3882531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3882638Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3882943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3883097Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3883103Z 2025-09-07T07:07:42.3883218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3883418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3883493Z return mod(**inputs) 2025-09-07T07:07:42.3883751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3883820Z outputs = self.model( 2025-09-07T07:07:42.3884089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3884164Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3884447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3884520Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3884757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3884840Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3885102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3885218Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3885473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3885578Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3885875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3885985Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3885989Z 2025-09-07T07:07:42.3886103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3886317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3886415Z return mod(**inputs) 2025-09-07T07:07:42.3886704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3886780Z outputs = self.model( 2025-09-07T07:07:42.3887043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3887117Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3887387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3887460Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3887695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3887775Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3888036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3888171Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3888430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3888520Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3888524Z 2025-09-07T07:07:42.3888626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3888834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3888900Z return mod(**inputs) 2025-09-07T07:07:42.3889158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3889249Z outputs = self.model( 2025-09-07T07:07:42.3889512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3889595Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3889859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3889934Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3890180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3890262Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3890567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.3890699Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3890703Z 2025-09-07T07:07:42.3890813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3891036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3891120Z return mod(**inputs) 2025-09-07T07:07:42.3891385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3891453Z outputs = self.model( 2025-09-07T07:07:42.3891719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3891791Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3892051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3892133Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3892368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3892457Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3892759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.3892888Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3892895Z 2025-09-07T07:07:42.3893014Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3893229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3893305Z return mod(**inputs) 2025-09-07T07:07:42.3893587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3893667Z outputs = self.model( 2025-09-07T07:07:42.3893955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3894032Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3894322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3894419Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3894670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3894754Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3895037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.3895133Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3895137Z 2025-09-07T07:07:42.3895249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3895468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3895537Z return mod(**inputs) 2025-09-07T07:07:42.3895841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3895926Z outputs = self.model( 2025-09-07T07:07:42.3896218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3896305Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3896579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3896663Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3896900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3897001Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3897288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3897395Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3897678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3897842Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3897846Z 2025-09-07T07:07:42.3897959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3898178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3898248Z return mod(**inputs) 2025-09-07T07:07:42.3898530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3898604Z outputs = self.model( 2025-09-07T07:07:42.3898887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3898964Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3899258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3899353Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3899589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3899682Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3899954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3900059Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3900342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3900430Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3900433Z 2025-09-07T07:07:42.3900551Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3900764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3900852Z return mod(**inputs) 2025-09-07T07:07:42.3901132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3901204Z outputs = self.model( 2025-09-07T07:07:42.3901482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3901560Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3901841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3901918Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3902178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3902272Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3902541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3902653Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3902923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3903014Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3903018Z 2025-09-07T07:07:42.3903110Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3903194Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3903299Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3903382Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3903492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3903717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3903788Z return mod(**inputs) 2025-09-07T07:07:42.3904072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3904145Z outputs = self.model( 2025-09-07T07:07:42.3904421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3904505Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3904778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3904862Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3905102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3905191Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3905468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3905590Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3906116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3906281Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3906851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3907083Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3907090Z 2025-09-07T07:07:42.3907285Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3907633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3907729Z return mod(**inputs) 2025-09-07T07:07:42.3908192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3908273Z outputs = self.model( 2025-09-07T07:07:42.3908596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3908671Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3908924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3909007Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3909237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3909328Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3909600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3909702Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3909966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3910063Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3910363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3910471Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3910475Z 2025-09-07T07:07:42.3910583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3910801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3910869Z return mod(**inputs) 2025-09-07T07:07:42.3911133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3911201Z outputs = self.model( 2025-09-07T07:07:42.3911459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3911533Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3911784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3911863Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3912079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3912168Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3912417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3912526Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3912779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3912882Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3912887Z 2025-09-07T07:07:42.3912997Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3913197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3913269Z return mod(**inputs) 2025-09-07T07:07:42.3913519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3913585Z outputs = self.model( 2025-09-07T07:07:42.3913846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3913917Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3914178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3914250Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3914467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3914568Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3914819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-09-07T07:07:42.3914906Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3914909Z 2025-09-07T07:07:42.3915010Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3915212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3915276Z return mod(**inputs) 2025-09-07T07:07:42.3915539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3915615Z outputs = self.model( 2025-09-07T07:07:42.3915871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3915954Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3916200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3916270Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3916496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3916573Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3916845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3916956Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3917212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3917365Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3917370Z 2025-09-07T07:07:42.3917473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3917678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3917742Z return mod(**inputs) 2025-09-07T07:07:42.3918002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3918069Z outputs = self.model( 2025-09-07T07:07:42.3918319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3918398Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3918650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3918743Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3918966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3919051Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3919304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3919411Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3919860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3919950Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3919954Z 2025-09-07T07:07:42.3920064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3920260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3920328Z return mod(**inputs) 2025-09-07T07:07:42.3920588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3920706Z outputs = self.model( 2025-09-07T07:07:42.3920969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3921042Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3921308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3921379Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3921612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3921698Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3921971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3922087Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3922338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3922424Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3922427Z 2025-09-07T07:07:42.3922517Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3922596Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3922680Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3922757Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3922882Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3923086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3923152Z return mod(**inputs) 2025-09-07T07:07:42.3923417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3923487Z outputs = self.model( 2025-09-07T07:07:42.3923745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3923826Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3924082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3924161Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3924387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3924473Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3924734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3924842Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3925132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3925233Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3925544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3925678Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3925682Z 2025-09-07T07:07:42.3925781Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3925986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3926052Z return mod(**inputs) 2025-09-07T07:07:42.3926313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3926381Z outputs = self.model( 2025-09-07T07:07:42.3926647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3926749Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3927004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3927085Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3927309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3927397Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3927652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3927761Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3928040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3928142Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3928448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3928558Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3928562Z 2025-09-07T07:07:42.3928672Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3928875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3928942Z return mod(**inputs) 2025-09-07T07:07:42.3929257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3929328Z outputs = self.model( 2025-09-07T07:07:42.3929596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3929671Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3929935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3930017Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3930240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3930325Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3930579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3930687Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3930951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3931033Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3931099Z 2025-09-07T07:07:42.3931211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3931414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3931489Z return mod(**inputs) 2025-09-07T07:07:42.3931749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3931818Z outputs = self.model( 2025-09-07T07:07:42.3932082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3932157Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3932426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3932501Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3932726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3932832Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3933087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.3933217Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3933221Z 2025-09-07T07:07:42.3933324Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3933534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3933600Z return mod(**inputs) 2025-09-07T07:07:42.3933860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3933937Z outputs = self.model( 2025-09-07T07:07:42.3934212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3934297Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3934555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3934628Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3934862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3934941Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3935206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.3935342Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3935347Z 2025-09-07T07:07:42.3935459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3935661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3935729Z return mod(**inputs) 2025-09-07T07:07:42.3935991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3936060Z outputs = self.model( 2025-09-07T07:07:42.3936319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3936391Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3936647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3936730Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3936953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3937039Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3937295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.3937398Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3937409Z 2025-09-07T07:07:42.3937514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3937722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3937796Z return mod(**inputs) 2025-09-07T07:07:42.3938047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3938119Z outputs = self.model( 2025-09-07T07:07:42.3938371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3938443Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3938700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3938772Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3939011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3939088Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3939339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3939447Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3939700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3939861Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3939865Z 2025-09-07T07:07:42.3939983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3940188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3940255Z return mod(**inputs) 2025-09-07T07:07:42.3940508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3940582Z outputs = self.model( 2025-09-07T07:07:42.3940835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3940916Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3941174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3941261Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3941494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3941575Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3941839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3941942Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3942206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3942287Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3942290Z 2025-09-07T07:07:42.3942393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3942603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3942670Z return mod(**inputs) 2025-09-07T07:07:42.3942935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3943005Z outputs = self.model( 2025-09-07T07:07:42.3943269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3943370Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3943632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3943712Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3943937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3944014Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3944282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3944380Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3944646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3944737Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3944740Z 2025-09-07T07:07:42.3944829Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3944933Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3945011Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3945095Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3945198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3945408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3945472Z return mod(**inputs) 2025-09-07T07:07:42.3945854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3945950Z outputs = self.model( 2025-09-07T07:07:42.3946248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3946336Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3946612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3946691Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3946938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3947022Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3947297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3947394Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3947671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3947772Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3948064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3948208Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3948212Z 2025-09-07T07:07:42.3948313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3948521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3948585Z return mod(**inputs) 2025-09-07T07:07:42.3948837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3948910Z outputs = self.model( 2025-09-07T07:07:42.3949163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3949243Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3949494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3949599Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3949819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3949896Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3950153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3950248Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3950504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3950595Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3950883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3951000Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3951003Z 2025-09-07T07:07:42.3951120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3951325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3951388Z return mod(**inputs) 2025-09-07T07:07:42.3951650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3951715Z outputs = self.model( 2025-09-07T07:07:42.3951970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3952051Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3952324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3952403Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3952624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3952701Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3952967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3953074Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3953326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3953415Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3953419Z 2025-09-07T07:07:42.3953538Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3953742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3953809Z return mod(**inputs) 2025-09-07T07:07:42.3954067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3954135Z outputs = self.model( 2025-09-07T07:07:42.3954387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3954466Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3954715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3954794Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3955025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3957700Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3957959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3958067Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3958307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3958464Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3958469Z 2025-09-07T07:07:42.3958569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3958772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3958835Z return mod(**inputs) 2025-09-07T07:07:42.3959091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3959196Z outputs = self.model( 2025-09-07T07:07:42.3959450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3959529Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3959777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3959877Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3960100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3960178Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3960442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3960548Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3960811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3960911Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3960916Z 2025-09-07T07:07:42.3961019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3961222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3961290Z return mod(**inputs) 2025-09-07T07:07:42.3961555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3961624Z outputs = self.model( 2025-09-07T07:07:42.3961890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3961974Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3962244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3962329Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3962549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3962633Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3962889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3963000Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3963261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3963349Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3963353Z 2025-09-07T07:07:42.3963442Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3963522Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3963602Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3963754Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3963859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3964066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3964132Z return mod(**inputs) 2025-09-07T07:07:42.3964393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3964469Z outputs = self.model( 2025-09-07T07:07:42.3964727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3964807Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3965065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3965146Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3965380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3965458Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3965714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3965839Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3966100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3966199Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3966498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3966653Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3966659Z 2025-09-07T07:07:42.3966760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3966983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3967050Z return mod(**inputs) 2025-09-07T07:07:42.3967312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3967381Z outputs = self.model( 2025-09-07T07:07:42.3967638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3967720Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3967971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3968048Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3968285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3968369Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3968641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3968753Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3969021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3969123Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3969428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3969540Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3969544Z 2025-09-07T07:07:42.3969650Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3969868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3969970Z return mod(**inputs) 2025-09-07T07:07:42.3970244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3970313Z outputs = self.model( 2025-09-07T07:07:42.3970571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3970654Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3970922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3970999Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3971218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3971296Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3971558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.3971669Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.3971932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3972035Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3972038Z 2025-09-07T07:07:42.3972148Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3972350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3972416Z return mod(**inputs) 2025-09-07T07:07:42.3972683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3972753Z outputs = self.model( 2025-09-07T07:07:42.3973018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3973092Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3973364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3973446Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3973674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3973762Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3974025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-09-07T07:07:42.3974114Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.3974117Z 2025-09-07T07:07:42.3974219Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3974447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3974525Z return mod(**inputs) 2025-09-07T07:07:42.3974783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3974860Z outputs = self.model( 2025-09-07T07:07:42.3975118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3975194Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3975461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3975532Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3975768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3975848Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3976111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.3976261Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3976265Z 2025-09-07T07:07:42.3976368Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3976576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3976642Z return mod(**inputs) 2025-09-07T07:07:42.3976904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3976972Z outputs = self.model( 2025-09-07T07:07:42.3977229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3977309Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3977568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3977647Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3977873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3977953Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3978229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.3978346Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.3978350Z 2025-09-07T07:07:42.3978460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3978660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3978733Z return mod(**inputs) 2025-09-07T07:07:42.3978992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3979062Z outputs = self.model( 2025-09-07T07:07:42.3979342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3979419Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3979682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3979754Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3979977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3980065Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3980325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.3980431Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.3980436Z 2025-09-07T07:07:42.3980540Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3980750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3980817Z return mod(**inputs) 2025-09-07T07:07:42.3981080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3981160Z outputs = self.model( 2025-09-07T07:07:42.3981431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3981511Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3981777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3981854Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3982108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3982213Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3982494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3982601Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3982872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.3983043Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.3983047Z 2025-09-07T07:07:42.3983155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3983374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3983443Z return mod(**inputs) 2025-09-07T07:07:42.3983722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3983795Z outputs = self.model( 2025-09-07T07:07:42.3984068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3984153Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3984443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3984526Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3984761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3984846Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3985126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3985232Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3985508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.3985607Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.3985612Z 2025-09-07T07:07:42.3985865Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3986229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3986341Z return mod(**inputs) 2025-09-07T07:07:42.3986811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3986920Z outputs = self.model( 2025-09-07T07:07:42.3987365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3987465Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3987963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3988083Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3988498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3988630Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3988914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3989028Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3989302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.3989396Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.3989400Z 2025-09-07T07:07:42.3989494Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3989580Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3989702Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3989782Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.3989895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3990115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3990188Z return mod(**inputs) 2025-09-07T07:07:42.3990468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3990541Z outputs = self.model( 2025-09-07T07:07:42.3990813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3990898Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3991171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3991255Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3991494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3991579Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3991856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3991988Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3992269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3992372Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3992695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.3992842Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.3992846Z 2025-09-07T07:07:42.3992956Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3993195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3993267Z return mod(**inputs) 2025-09-07T07:07:42.3993548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3993621Z outputs = self.model( 2025-09-07T07:07:42.3993895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3993980Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3994250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3994333Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3994585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3994679Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3994947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3995049Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3995330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.3995431Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.3995749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.3995866Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.3995869Z 2025-09-07T07:07:42.3995979Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3996197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3996284Z return mod(**inputs) 2025-09-07T07:07:42.3996564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3996637Z outputs = self.model( 2025-09-07T07:07:42.3996921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3997006Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3997260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3997337Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3997555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.3997641Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.3997894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.3997991Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.3998252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.3998350Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.3998353Z 2025-09-07T07:07:42.3998461Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.3998656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.3998728Z return mod(**inputs) 2025-09-07T07:07:42.3998982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.3999052Z outputs = self.model( 2025-09-07T07:07:42.3999318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.3999418Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.3999693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.3999768Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.3999993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4000083Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4000340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4000463Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4000749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4000903Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4000913Z 2025-09-07T07:07:42.4001014Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4001222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4001293Z return mod(**inputs) 2025-09-07T07:07:42.4001538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4001610Z outputs = self.model( 2025-09-07T07:07:42.4001861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4001932Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4002192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4002282Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4002512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4002588Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4002840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4002953Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4003204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4003291Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4003294Z 2025-09-07T07:07:42.4003393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4003597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4003660Z return mod(**inputs) 2025-09-07T07:07:42.4003914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4003987Z outputs = self.model( 2025-09-07T07:07:42.4004234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4004338Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4004581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4004650Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4004874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4004951Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4005199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4005304Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4005574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4005661Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4005666Z 2025-09-07T07:07:42.4005746Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4005830Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4005905Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4005984Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4006084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4006277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4006347Z return mod(**inputs) 2025-09-07T07:07:42.4006611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4006687Z outputs = self.model( 2025-09-07T07:07:42.4006942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4007013Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4007277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4007345Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4007558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4007634Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4007872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4007982Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4008244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4008345Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4008629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4008762Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4008766Z 2025-09-07T07:07:42.4008864Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4009054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4009122Z return mod(**inputs) 2025-09-07T07:07:42.4009367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4009439Z outputs = self.model( 2025-09-07T07:07:42.4009689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4009760Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4010010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4010094Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4010317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4010394Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4010649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4010755Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4011005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4011109Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4011414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4011527Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4011532Z 2025-09-07T07:07:42.4011633Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4011828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4011900Z return mod(**inputs) 2025-09-07T07:07:42.4012160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4012233Z outputs = self.model( 2025-09-07T07:07:42.4012496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4012578Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4012826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4012897Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4013119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4013196Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4013446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4013548Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4013793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4013881Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4013901Z 2025-09-07T07:07:42.4014004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4014210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4014275Z return mod(**inputs) 2025-09-07T07:07:42.4014540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4014608Z outputs = self.model( 2025-09-07T07:07:42.4014860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4014939Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4015189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4015267Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4015489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4015576Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4015838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4015958Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4015978Z 2025-09-07T07:07:42.4016086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4016280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4016350Z return mod(**inputs) 2025-09-07T07:07:42.4016601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4016668Z outputs = self.model( 2025-09-07T07:07:42.4016928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4017002Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4017274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4017345Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4017565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4017652Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4017901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4018024Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4018028Z 2025-09-07T07:07:42.4018126Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4018336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4018409Z return mod(**inputs) 2025-09-07T07:07:42.4018667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4018744Z outputs = self.model( 2025-09-07T07:07:42.4019003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4019083Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4019342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4019414Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4019814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4019902Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4020173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4020309Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4020316Z 2025-09-07T07:07:42.4020419Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4020627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4020695Z return mod(**inputs) 2025-09-07T07:07:42.4020959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4021027Z outputs = self.model( 2025-09-07T07:07:42.4021294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4021366Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4021626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4021710Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4021938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4022025Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4022283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-09-07T07:07:42.4022400Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.4022405Z 2025-09-07T07:07:42.4022514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4022714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4022785Z return mod(**inputs) 2025-09-07T07:07:42.4023043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4023110Z outputs = self.model( 2025-09-07T07:07:42.4023397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4023473Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4023736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4023811Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4024041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4024120Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4024388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4024505Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4024800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4024975Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4024979Z 2025-09-07T07:07:42.4025089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4025307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4025386Z return mod(**inputs) 2025-09-07T07:07:42.4025731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4025840Z outputs = self.model( 2025-09-07T07:07:42.4026292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4026408Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4026863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4027013Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4027429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4027548Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4027967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4028099Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4028385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4028479Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4028483Z 2025-09-07T07:07:42.4028584Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4028792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4028860Z return mod(**inputs) 2025-09-07T07:07:42.4029122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4029192Z outputs = self.model( 2025-09-07T07:07:42.4029451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4029560Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4029818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4029898Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4030124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4030212Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4030473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4030604Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4030869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4030956Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4030961Z 2025-09-07T07:07:42.4031050Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4031132Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4031209Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4031294Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4031398Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4031599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4031672Z return mod(**inputs) 2025-09-07T07:07:42.4031950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4032029Z outputs = self.model( 2025-09-07T07:07:42.4032296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4032380Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4032648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4032723Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4032968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4033050Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4033327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4033430Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4033703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4033809Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4034105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4034249Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4034253Z 2025-09-07T07:07:42.4034357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4034565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4034632Z return mod(**inputs) 2025-09-07T07:07:42.4034892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4034970Z outputs = self.model( 2025-09-07T07:07:42.4035233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4035314Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4035574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4035664Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4035898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4035977Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4036245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4036344Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4036618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4036732Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4037029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4037145Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4037150Z 2025-09-07T07:07:42.4037253Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4037460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4037526Z return mod(**inputs) 2025-09-07T07:07:42.4037788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4037863Z outputs = self.model( 2025-09-07T07:07:42.4038140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4038224Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4038484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4038556Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4038791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4038870Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4039132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4039229Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4039493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4039578Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4039600Z 2025-09-07T07:07:42.4039704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4039915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4039982Z return mod(**inputs) 2025-09-07T07:07:42.4040246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4040315Z outputs = self.model( 2025-09-07T07:07:42.4040569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4040648Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4040903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4040980Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4041209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4041300Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4041557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4041669Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4041950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4042102Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4042106Z 2025-09-07T07:07:42.4042224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4042421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4042485Z return mod(**inputs) 2025-09-07T07:07:42.4042747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4042829Z outputs = self.model( 2025-09-07T07:07:42.4043090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4043162Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4043421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4043492Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4043711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4043795Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4044063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4044178Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4044430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4044510Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4044514Z 2025-09-07T07:07:42.4044622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4044824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4044894Z return mod(**inputs) 2025-09-07T07:07:42.4045145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4045219Z outputs = self.model( 2025-09-07T07:07:42.4045468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4045541Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4045828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4045899Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4046123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4046203Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4046460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4046573Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4046820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4046911Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4046915Z 2025-09-07T07:07:42.4046995Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4047074Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4047158Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4047235Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4047342Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4047539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4047620Z return mod(**inputs) 2025-09-07T07:07:42.4047884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4047950Z outputs = self.model( 2025-09-07T07:07:42.4048212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4048284Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4048545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4048618Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4048856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4048946Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4049208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4049325Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4049582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4049680Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4049986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4050138Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4050143Z 2025-09-07T07:07:42.4050256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4050459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4050532Z return mod(**inputs) 2025-09-07T07:07:42.4050796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4050864Z outputs = self.model( 2025-09-07T07:07:42.4051132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4051204Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4051473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4051546Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4051788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4051876Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4052135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4052250Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4052509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4052614Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4052913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4053019Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4053023Z 2025-09-07T07:07:42.4053137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4053344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4053418Z return mod(**inputs) 2025-09-07T07:07:42.4053682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4053765Z outputs = self.model( 2025-09-07T07:07:42.4054055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4054132Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4054473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4054545Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4054775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4054863Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4055143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4055261Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4055515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4055607Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4055611Z 2025-09-07T07:07:42.4055722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4055917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4055988Z return mod(**inputs) 2025-09-07T07:07:42.4056309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4056383Z outputs = self.model( 2025-09-07T07:07:42.4056642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4056714Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4056976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4057048Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4057279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4057358Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4057621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4057741Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4057746Z 2025-09-07T07:07:42.4057847Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4058074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4058140Z return mod(**inputs) 2025-09-07T07:07:42.4058403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4058472Z outputs = self.model( 2025-09-07T07:07:42.4058726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4058805Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4059058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4059136Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4059357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4059442Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4059696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4059815Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4059835Z 2025-09-07T07:07:42.4059947Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4060149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4060223Z return mod(**inputs) 2025-09-07T07:07:42.4060492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4060558Z outputs = self.model( 2025-09-07T07:07:42.4060829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4060900Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4061181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4061254Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4061481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4061567Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4061846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4061940Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4061944Z 2025-09-07T07:07:42.4062054Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4062272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4062358Z return mod(**inputs) 2025-09-07T07:07:42.4062633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4062715Z outputs = self.model( 2025-09-07T07:07:42.4063004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4063091Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4063375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4063451Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4063699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4063777Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4064095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4064245Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4064525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4064688Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4064694Z 2025-09-07T07:07:42.4064805Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4065025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4065096Z return mod(**inputs) 2025-09-07T07:07:42.4065382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4065455Z outputs = self.model( 2025-09-07T07:07:42.4065868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4065968Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4066252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4066340Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4066584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4066706Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4066993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4067099Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4067378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4067466Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4067470Z 2025-09-07T07:07:42.4067589Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4067819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4067891Z return mod(**inputs) 2025-09-07T07:07:42.4068173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4068247Z outputs = self.model( 2025-09-07T07:07:42.4068529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4068606Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4068904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4068986Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4069246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4069342Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4069614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4069725Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4069996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4070088Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4070092Z 2025-09-07T07:07:42.4070185Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4070270Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4070358Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4070440Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4070548Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4070772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4070863Z return mod(**inputs) 2025-09-07T07:07:42.4071142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4071214Z outputs = self.model( 2025-09-07T07:07:42.4071492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4071578Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4071851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4071935Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4072172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4072267Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4072537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4072645Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4072927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4073047Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4073372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4073519Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4073522Z 2025-09-07T07:07:42.4073631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4073856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4073939Z return mod(**inputs) 2025-09-07T07:07:42.4074216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4074285Z outputs = self.model( 2025-09-07T07:07:42.4074545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4074619Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4074870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4074949Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4075169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4075252Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4075526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4075626Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4075885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4075980Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4076277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4076385Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4076389Z 2025-09-07T07:07:42.4076504Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4076703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4076768Z return mod(**inputs) 2025-09-07T07:07:42.4077028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4077112Z outputs = self.model( 2025-09-07T07:07:42.4077380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4077452Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4077706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4077789Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4078009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4078095Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4078347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4078446Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4078706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4078791Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4078795Z 2025-09-07T07:07:42.4078907Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4079110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4079204Z return mod(**inputs) 2025-09-07T07:07:42.4079463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4079530Z outputs = self.model( 2025-09-07T07:07:42.4079793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4079866Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4080132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4080206Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4080447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4080548Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4080800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-09-07T07:07:42.4080888Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.4080892Z 2025-09-07T07:07:42.4080993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4081198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4081263Z return mod(**inputs) 2025-09-07T07:07:42.4081530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4081607Z outputs = self.model( 2025-09-07T07:07:42.4081861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4081938Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4082187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4082258Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4082484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4082561Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4082815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4082925Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4083175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4083349Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4083353Z 2025-09-07T07:07:42.4083454Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4083658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4083724Z return mod(**inputs) 2025-09-07T07:07:42.4083984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4084050Z outputs = self.model( 2025-09-07T07:07:42.4084301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4084378Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4084629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4084707Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4084926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4085003Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4085277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4085381Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4085638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4085717Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4085720Z 2025-09-07T07:07:42.4085829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4086025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4086092Z return mod(**inputs) 2025-09-07T07:07:42.4086367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4086435Z outputs = self.model( 2025-09-07T07:07:42.4086694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4086765Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4087016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4087092Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4087317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4087415Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4087668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4087785Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4088033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4088120Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4088124Z 2025-09-07T07:07:42.4088211Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4088292Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4088376Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4088451Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4088552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4088753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4088819Z return mod(**inputs) 2025-09-07T07:07:42.4089097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4089163Z outputs = self.model( 2025-09-07T07:07:42.4089414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4089494Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4089746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4089823Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4090038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4090115Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4090372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4090478Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4090736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4090831Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4091156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4091289Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4091292Z 2025-09-07T07:07:42.4091391Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4091596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4091662Z return mod(**inputs) 2025-09-07T07:07:42.4091924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4091992Z outputs = self.model( 2025-09-07T07:07:42.4092269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4092351Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4092605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4092685Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4092905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4092990Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4093241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4093363Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4093627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4093722Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4094018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4094125Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4094128Z 2025-09-07T07:07:42.4094237Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4094432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4094496Z return mod(**inputs) 2025-09-07T07:07:42.4094754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4094821Z outputs = self.model( 2025-09-07T07:07:42.4095076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4095168Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4095416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4095497Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4095713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4095798Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4096048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4096152Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4096409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4096491Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4096495Z 2025-09-07T07:07:42.4096602Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4096800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4096886Z return mod(**inputs) 2025-09-07T07:07:42.4097137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4097203Z outputs = self.model( 2025-09-07T07:07:42.4097461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4097532Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4097798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4097867Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4098103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4098189Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4098446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4098584Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4098587Z 2025-09-07T07:07:42.4098687Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4098886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4098949Z return mod(**inputs) 2025-09-07T07:07:42.4099195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4099285Z outputs = self.model( 2025-09-07T07:07:42.4099539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4099622Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4099871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4099943Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4100169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4100246Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4100506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4100621Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4100624Z 2025-09-07T07:07:42.4100727Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4100950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4101016Z return mod(**inputs) 2025-09-07T07:07:42.4101276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4101342Z outputs = self.model( 2025-09-07T07:07:42.4101616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4101688Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4101939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4102019Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4102238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4102324Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4102584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4102667Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4102671Z 2025-09-07T07:07:42.4102782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4103003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4103076Z return mod(**inputs) 2025-09-07T07:07:42.4103337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4103412Z outputs = self.model( 2025-09-07T07:07:42.4103672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4103747Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4104014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4104456Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4104706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4104793Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4105065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4105185Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4105483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4105656Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4105660Z 2025-09-07T07:07:42.4105910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4106149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4106222Z return mod(**inputs) 2025-09-07T07:07:42.4106508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4106593Z outputs = self.model( 2025-09-07T07:07:42.4106873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4106959Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4107245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4107323Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4107579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4107664Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4107971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4108082Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4108323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4108411Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4108415Z 2025-09-07T07:07:42.4108513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4108717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4108782Z return mod(**inputs) 2025-09-07T07:07:42.4109038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4109106Z outputs = self.model( 2025-09-07T07:07:42.4109361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4109440Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4109690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4109786Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4110003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4110080Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4110340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4110435Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4110691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4110778Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4110781Z 2025-09-07T07:07:42.4110884Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4110965Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4111050Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4111134Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4111230Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4111425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4111487Z return mod(**inputs) 2025-09-07T07:07:42.4111731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4111802Z outputs = self.model( 2025-09-07T07:07:42.4112062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4112143Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4112394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4112464Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4112690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4112767Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4113020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4113117Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4113367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4113473Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4113783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4113921Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4113925Z 2025-09-07T07:07:42.4114024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4114229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4114294Z return mod(**inputs) 2025-09-07T07:07:42.4114545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4114620Z outputs = self.model( 2025-09-07T07:07:42.4114868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4114950Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4115199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4115271Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4115496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4115602Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4115861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4115956Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4116214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4116308Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4116601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4116719Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4116740Z 2025-09-07T07:07:42.4116842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4117046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4117113Z return mod(**inputs) 2025-09-07T07:07:42.4117365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4117440Z outputs = self.model( 2025-09-07T07:07:42.4117690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4117770Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4118036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4118118Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4118342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4118420Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4118683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4118783Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4119065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4119153Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4119156Z 2025-09-07T07:07:42.4119267Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4119493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4119773Z return mod(**inputs) 2025-09-07T07:07:42.4120123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4120199Z outputs = self.model( 2025-09-07T07:07:42.4120491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4120576Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4120853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4120938Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4121166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4121254Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4121516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4121639Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4121900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4122051Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4122083Z 2025-09-07T07:07:42.4122195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4122396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4122469Z return mod(**inputs) 2025-09-07T07:07:42.4122722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4122788Z outputs = self.model( 2025-09-07T07:07:42.4123048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4123122Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4123401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4123474Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4123695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4123781Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4124033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4124146Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4124396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4124500Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4124513Z 2025-09-07T07:07:42.4124616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4124819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4124894Z return mod(**inputs) 2025-09-07T07:07:42.4125147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4125223Z outputs = self.model( 2025-09-07T07:07:42.4125476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4125549Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4125811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4125884Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4126113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4126209Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4126458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4126573Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4126822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4126912Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4126917Z 2025-09-07T07:07:42.4126994Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4127080Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4127158Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4127233Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4127345Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4127543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4127615Z return mod(**inputs) 2025-09-07T07:07:42.4127865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4127973Z outputs = self.model( 2025-09-07T07:07:42.4128228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4128298Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4128557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4128626Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4128854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4128941Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4129212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4129328Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4129586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4129690Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4129998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4130138Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4130141Z 2025-09-07T07:07:42.4130254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4130478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4130556Z return mod(**inputs) 2025-09-07T07:07:42.4130818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4130887Z outputs = self.model( 2025-09-07T07:07:42.4131158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4131234Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4131508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4131578Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4131795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4131880Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4132134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4132264Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4132514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4132619Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4132908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4133013Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4133016Z 2025-09-07T07:07:42.4133126Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4133323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4133395Z return mod(**inputs) 2025-09-07T07:07:42.4133650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4133719Z outputs = self.model( 2025-09-07T07:07:42.4133985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4134058Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4134339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4134411Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4134636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4134715Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4134967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4135081Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4135358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4135446Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4135450Z 2025-09-07T07:07:42.4135559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4135759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4135833Z return mod(**inputs) 2025-09-07T07:07:42.4136083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4136156Z outputs = self.model( 2025-09-07T07:07:42.4136406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4136507Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4136761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4136834Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4137058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4137137Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4137393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-09-07T07:07:42.4137471Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.4137475Z 2025-09-07T07:07:42.4137576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4137778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4137844Z return mod(**inputs) 2025-09-07T07:07:42.4138106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4138197Z outputs = self.model( 2025-09-07T07:07:42.4138457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4138536Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4138794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4138873Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4139097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4139182Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4139437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4139557Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4139562Z 2025-09-07T07:07:42.4139672Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4139872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4139943Z return mod(**inputs) 2025-09-07T07:07:42.4140212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4140279Z outputs = self.model( 2025-09-07T07:07:42.4140536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4140609Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4140866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4140937Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4141161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4141258Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4141515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4141641Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4141645Z 2025-09-07T07:07:42.4141747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4141963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4142032Z return mod(**inputs) 2025-09-07T07:07:42.4142303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4142382Z outputs = self.model( 2025-09-07T07:07:42.4142671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4142757Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4143033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4143109Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4143353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4143435Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4143711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4143797Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4143801Z 2025-09-07T07:07:42.4143915Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4144124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4144212Z return mod(**inputs) 2025-09-07T07:07:42.4144517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4144586Z outputs = self.model( 2025-09-07T07:07:42.4144850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4144924Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4145184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4145263Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4145501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4145594Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4145987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4146115Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4146398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4146599Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4146604Z 2025-09-07T07:07:42.4146726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4146957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4147029Z return mod(**inputs) 2025-09-07T07:07:42.4147274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4147339Z outputs = self.model( 2025-09-07T07:07:42.4147596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4147687Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4147938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4148008Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4148227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4148303Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4148546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4148651Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4148917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4149007Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4149010Z 2025-09-07T07:07:42.4149114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4149307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4149382Z return mod(**inputs) 2025-09-07T07:07:42.4149637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4149710Z outputs = self.model( 2025-09-07T07:07:42.4149959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4150029Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4150294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4150368Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4150617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4150695Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4150943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4151038Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4151282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4151372Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4151376Z 2025-09-07T07:07:42.4151453Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4151537Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4151609Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4151683Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4151788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4151980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4152051Z return mod(**inputs) 2025-09-07T07:07:42.4152299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4152380Z outputs = self.model( 2025-09-07T07:07:42.4152633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4152703Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4152953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4153022Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4153249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4153326Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4153592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4153695Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4153939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4154042Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4154333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4154468Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4154479Z 2025-09-07T07:07:42.4154580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4154792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4154869Z return mod(**inputs) 2025-09-07T07:07:42.4155123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4155196Z outputs = self.model( 2025-09-07T07:07:42.4155446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4155519Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4155777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4155847Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4156073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4156152Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4156404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4156536Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4156778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4156877Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4157155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4157265Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4157268Z 2025-09-07T07:07:42.4157365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4157553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4157624Z return mod(**inputs) 2025-09-07T07:07:42.4157869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4157941Z outputs = self.model( 2025-09-07T07:07:42.4158184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4158271Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4158527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4158599Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4158822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4158899Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4159151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4159254Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4159521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4159611Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4159614Z 2025-09-07T07:07:42.4159716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4159919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4159985Z return mod(**inputs) 2025-09-07T07:07:42.4160242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4160318Z outputs = self.model( 2025-09-07T07:07:42.4160575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4160683Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4160939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4161011Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4161238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4161319Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4161577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4161686Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4161945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4162095Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4162099Z 2025-09-07T07:07:42.4162223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4162427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4162493Z return mod(**inputs) 2025-09-07T07:07:42.4162762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4162830Z outputs = self.model( 2025-09-07T07:07:42.4163084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4163163Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4163415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4163492Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4163711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4163797Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4164049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4164156Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4164435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4164515Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4164519Z 2025-09-07T07:07:42.4164625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4164825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4164890Z return mod(**inputs) 2025-09-07T07:07:42.4165154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4165221Z outputs = self.model( 2025-09-07T07:07:42.4165497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4165571Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4165823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4165902Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4166119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4166203Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4166451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4166586Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4166838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4166926Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4166929Z 2025-09-07T07:07:42.4167017Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4167096Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4167181Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4167257Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4167356Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4167561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4167626Z return mod(**inputs) 2025-09-07T07:07:42.4167887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4167953Z outputs = self.model( 2025-09-07T07:07:42.4168207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4168305Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4168555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4168635Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4168859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4168942Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4169196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4169302Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4169557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4169653Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4169951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4170082Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4170108Z 2025-09-07T07:07:42.4170212Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4170414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4170479Z return mod(**inputs) 2025-09-07T07:07:42.4170740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4170806Z outputs = self.model( 2025-09-07T07:07:42.4171066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4171138Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4171407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4171487Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4171708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4171802Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4172049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4172151Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4172406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4172514Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4172804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4172911Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4172915Z 2025-09-07T07:07:42.4173019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4173210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4173276Z return mod(**inputs) 2025-09-07T07:07:42.4173533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4173599Z outputs = self.model( 2025-09-07T07:07:42.4173856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4173926Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4174178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4174271Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4174495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4174583Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4174842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4174948Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4175219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4175299Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4175302Z 2025-09-07T07:07:42.4175412Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4175607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4175680Z return mod(**inputs) 2025-09-07T07:07:42.4175930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4175995Z outputs = self.model( 2025-09-07T07:07:42.4176251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4176352Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4176600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4176668Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4176878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4176964Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4177206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4177342Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4177346Z 2025-09-07T07:07:42.4177448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4177650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4177718Z return mod(**inputs) 2025-09-07T07:07:42.4177971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4178047Z outputs = self.model( 2025-09-07T07:07:42.4178300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4178379Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4178650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4178724Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4178958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4179038Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4179306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4179426Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4179430Z 2025-09-07T07:07:42.4179539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4179741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4179806Z return mod(**inputs) 2025-09-07T07:07:42.4180072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4180157Z outputs = self.model( 2025-09-07T07:07:42.4180429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4180501Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4180768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4180847Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4181073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4181158Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4181418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4181502Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4181514Z 2025-09-07T07:07:42.4181619Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4181823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4181896Z return mod(**inputs) 2025-09-07T07:07:42.4182159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4182253Z outputs = self.model( 2025-09-07T07:07:42.4182517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4182590Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4182861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4182934Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4183184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4183270Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4183570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-09-07T07:07:42.4183667Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.4183672Z 2025-09-07T07:07:42.4183781Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4184004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4184073Z return mod(**inputs) 2025-09-07T07:07:42.4184357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4184426Z outputs = self.model( 2025-09-07T07:07:42.4184715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4184811Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4185069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4185148Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4185373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4185459Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4185974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4186088Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4186372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4186538Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4186576Z 2025-09-07T07:07:42.4186693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4186910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4186982Z return mod(**inputs) 2025-09-07T07:07:42.4187265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4187347Z outputs = self.model( 2025-09-07T07:07:42.4187608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4187680Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4187932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4188011Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4188232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4188317Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4188569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4188668Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4188945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4189026Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4189029Z 2025-09-07T07:07:42.4189141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4189341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4189414Z return mod(**inputs) 2025-09-07T07:07:42.4189673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4189752Z outputs = self.model( 2025-09-07T07:07:42.4190025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4190098Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4190355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4190425Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4190643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4190729Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4190980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4191101Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4191363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4191457Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4191461Z 2025-09-07T07:07:42.4191543Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4191626Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4191712Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4191789Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4191901Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4192109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4192176Z return mod(**inputs) 2025-09-07T07:07:42.4192442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4192512Z outputs = self.model( 2025-09-07T07:07:42.4192797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4192876Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4193140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4193227Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4193456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4193546Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4193808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4193915Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4194185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4194288Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4194609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4194747Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4194768Z 2025-09-07T07:07:42.4194877Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4195073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4195138Z return mod(**inputs) 2025-09-07T07:07:42.4195399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4195466Z outputs = self.model( 2025-09-07T07:07:42.4195727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4195802Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4196069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4196149Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4196365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4196451Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4196702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4196805Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4197052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4197165Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4197466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4197574Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4197577Z 2025-09-07T07:07:42.4197685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4197883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4197948Z return mod(**inputs) 2025-09-07T07:07:42.4198211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4198278Z outputs = self.model( 2025-09-07T07:07:42.4198542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4198617Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4198878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4198970Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4199197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4199285Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4199543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4199647Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4199901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4199982Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4199986Z 2025-09-07T07:07:42.4200097Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4200297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4200371Z return mod(**inputs) 2025-09-07T07:07:42.4200638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4200712Z outputs = self.model( 2025-09-07T07:07:42.4200979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4201050Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4201307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4201379Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4201604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4201682Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4201951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4202069Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4202320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4202483Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4202487Z 2025-09-07T07:07:42.4202593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4202805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4202870Z return mod(**inputs) 2025-09-07T07:07:42.4203132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4203223Z outputs = self.model( 2025-09-07T07:07:42.4203478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4203558Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4203828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4203909Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4204154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4204237Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4204515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4204630Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4204903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4205018Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4205023Z 2025-09-07T07:07:42.4205134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4205354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4205426Z return mod(**inputs) 2025-09-07T07:07:42.4205707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4205779Z outputs = self.model( 2025-09-07T07:07:42.4206054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4206141Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4206413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4206501Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4206739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4206822Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4207101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4207251Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4207527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4207617Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4207621Z 2025-09-07T07:07:42.4207713Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4207796Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4207878Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4207969Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4208081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4208323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4208394Z return mod(**inputs) 2025-09-07T07:07:42.4208671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4208752Z outputs = self.model( 2025-09-07T07:07:42.4209029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4209114Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4209388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4209465Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4209728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4209817Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4210094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4210208Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4210478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4210592Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4210909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4211059Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4211063Z 2025-09-07T07:07:42.4211172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4211410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4211485Z return mod(**inputs) 2025-09-07T07:07:42.4211764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4211848Z outputs = self.model( 2025-09-07T07:07:42.4212124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4212212Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4212488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4212568Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4212822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4212908Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4213198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4213317Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4213599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4213725Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4214043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4214167Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4214171Z 2025-09-07T07:07:42.4214280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4214501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4214572Z return mod(**inputs) 2025-09-07T07:07:42.4214876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4214957Z outputs = self.model( 2025-09-07T07:07:42.4215232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4215318Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4215590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4215667Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4215893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4215972Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4216254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4216365Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4216628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4216712Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4216716Z 2025-09-07T07:07:42.4216819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4217028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4217092Z return mod(**inputs) 2025-09-07T07:07:42.4217356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4217423Z outputs = self.model( 2025-09-07T07:07:42.4217690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4217784Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4218051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4218132Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4218358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4218446Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4218702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4218825Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4218828Z 2025-09-07T07:07:42.4218938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4219141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4219215Z return mod(**inputs) 2025-09-07T07:07:42.4219473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4219661Z outputs = self.model( 2025-09-07T07:07:42.4219940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4220056Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4220330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4220406Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4220655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4220740Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4221018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4221185Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4221190Z 2025-09-07T07:07:42.4221302Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4221523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4221595Z return mod(**inputs) 2025-09-07T07:07:42.4221870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4221952Z outputs = self.model( 2025-09-07T07:07:42.4222229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4222315Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4222612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4222700Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4222938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4223021Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4223300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4223389Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4223393Z 2025-09-07T07:07:42.4223509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4223724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4223793Z return mod(**inputs) 2025-09-07T07:07:42.4224073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4224145Z outputs = self.model( 2025-09-07T07:07:42.4224478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4224557Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4224837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4224915Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4225152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4225243Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4225518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4225631Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4225992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4226168Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4226173Z 2025-09-07T07:07:42.4226294Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4226515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4226624Z return mod(**inputs) 2025-09-07T07:07:42.4226910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4226983Z outputs = self.model( 2025-09-07T07:07:42.4227273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4227346Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4227615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4227692Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4227945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4228028Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4228285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4228395Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4228654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4228743Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4228747Z 2025-09-07T07:07:42.4228851Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4229067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4229145Z return mod(**inputs) 2025-09-07T07:07:42.4229408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4229482Z outputs = self.model( 2025-09-07T07:07:42.4229742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4229825Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4230085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4230157Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4230390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4230468Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4230734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4230859Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4231114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4231214Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4231217Z 2025-09-07T07:07:42.4231297Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4231383Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4231461Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4231536Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4231646Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4231845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4231918Z return mod(**inputs) 2025-09-07T07:07:42.4232177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4232247Z outputs = self.model( 2025-09-07T07:07:42.4232509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4232600Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4232863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4232934Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4233166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4233246Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4233502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4233608Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4233882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4233990Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4234299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4234434Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4234444Z 2025-09-07T07:07:42.4234545Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4234740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4234813Z return mod(**inputs) 2025-09-07T07:07:42.4235081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4235160Z outputs = self.model( 2025-09-07T07:07:42.4235414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4235486Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4235746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4235820Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4236045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4236121Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4236373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4236476Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4236730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4236851Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4237149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4237269Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4237273Z 2025-09-07T07:07:42.4237376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4237579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4237653Z return mod(**inputs) 2025-09-07T07:07:42.4237910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4237984Z outputs = self.model( 2025-09-07T07:07:42.4238243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4238318Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4238588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4238660Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4238910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4238990Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4239252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4239359Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4239610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4239698Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4239703Z 2025-09-07T07:07:42.4239803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4240023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4240088Z return mod(**inputs) 2025-09-07T07:07:42.4240339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4240413Z outputs = self.model( 2025-09-07T07:07:42.4240663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4240745Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4240994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4241081Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4241308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4241388Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4241647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-09-07T07:07:42.4241728Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.4241731Z 2025-09-07T07:07:42.4241838Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4242033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4242097Z return mod(**inputs) 2025-09-07T07:07:42.4242355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4242421Z outputs = self.model( 2025-09-07T07:07:42.4242679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4242769Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4243020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4243096Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4243314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4243397Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4243647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4243761Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4244008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4244158Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4244163Z 2025-09-07T07:07:42.4244274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4244470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4244539Z return mod(**inputs) 2025-09-07T07:07:42.4244809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4244874Z outputs = self.model( 2025-09-07T07:07:42.4245132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4245202Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4245460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4245531Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4245756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4245849Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4246098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4246211Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4246457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4246543Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4246546Z 2025-09-07T07:07:42.4246647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4246842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4246927Z return mod(**inputs) 2025-09-07T07:07:42.4247178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4247254Z outputs = self.model( 2025-09-07T07:07:42.4247506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4247585Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4247834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4247902Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4248125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4248200Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4248462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4248568Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4248833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4248925Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4248928Z 2025-09-07T07:07:42.4249009Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4249094Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4249169Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4249245Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4249355Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4249552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4249623Z return mod(**inputs) 2025-09-07T07:07:42.4249875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4249943Z outputs = self.model( 2025-09-07T07:07:42.4250208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4250283Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4250546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4250638Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4250878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4250958Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4251216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4251332Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4251596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4251723Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4252017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4252151Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4252161Z 2025-09-07T07:07:42.4252263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4252463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4252533Z return mod(**inputs) 2025-09-07T07:07:42.4252788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4252862Z outputs = self.model( 2025-09-07T07:07:42.4253144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4253220Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4253483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4253554Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4253783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4253860Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4254111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4254225Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4254476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4254593Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4254887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4255002Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4255005Z 2025-09-07T07:07:42.4255108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4255308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4255380Z return mod(**inputs) 2025-09-07T07:07:42.4255638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4255715Z outputs = self.model( 2025-09-07T07:07:42.4255974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4256048Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4256321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4256394Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4256626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4256727Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4256981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4257096Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4257353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4257444Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4257449Z 2025-09-07T07:07:42.4257553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4257784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4257852Z return mod(**inputs) 2025-09-07T07:07:42.4258117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4258195Z outputs = self.model( 2025-09-07T07:07:42.4258461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4258543Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4258800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4258871Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4259123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4259207Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4259469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4259591Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4259596Z 2025-09-07T07:07:42.4259705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4259903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4259970Z return mod(**inputs) 2025-09-07T07:07:42.4260235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4260301Z outputs = self.model( 2025-09-07T07:07:42.4260566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4260641Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4260929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4261008Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4261234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4261323Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4261580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4261705Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4261709Z 2025-09-07T07:07:42.4261813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4262015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4262090Z return mod(**inputs) 2025-09-07T07:07:42.4262367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4262444Z outputs = self.model( 2025-09-07T07:07:42.4262717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4262820Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4263101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4263177Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4263419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4263502Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4263773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4263871Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4263875Z 2025-09-07T07:07:42.4264008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4264218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4264287Z return mod(**inputs) 2025-09-07T07:07:42.4264552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4264620Z outputs = self.model( 2025-09-07T07:07:42.4264880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4264963Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4265238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4265318Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4265547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4265627Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4266003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4266114Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4266447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4266618Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4266622Z 2025-09-07T07:07:42.4266745Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4266979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4267053Z return mod(**inputs) 2025-09-07T07:07:42.4267367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4267437Z outputs = self.model( 2025-09-07T07:07:42.4267708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4267785Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4268043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4268122Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4268347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4268435Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4268697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4268805Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4269065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4269146Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4269183Z 2025-09-07T07:07:42.4269298Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4269500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4269572Z return mod(**inputs) 2025-09-07T07:07:42.4269832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4269899Z outputs = self.model( 2025-09-07T07:07:42.4270166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4270240Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4270524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4270599Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4270821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4270907Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4271165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4271271Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4271530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4271645Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4271649Z 2025-09-07T07:07:42.4271734Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4271815Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4271903Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4271981Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4272092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4272293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4272360Z return mod(**inputs) 2025-09-07T07:07:42.4272627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4272695Z outputs = self.model( 2025-09-07T07:07:42.4272959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4273032Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4273295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4273395Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4273623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4273710Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4273971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4274076Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4274396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4274489Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4274798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4274932Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4274936Z 2025-09-07T07:07:42.4275046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4275252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4275337Z return mod(**inputs) 2025-09-07T07:07:42.4275606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4275672Z outputs = self.model( 2025-09-07T07:07:42.4275935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4276008Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4276275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4276351Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4276616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4276709Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4276976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4277086Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4277358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4277459Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4277779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4277913Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4277917Z 2025-09-07T07:07:42.4278035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4278251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4278327Z return mod(**inputs) 2025-09-07T07:07:42.4278597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4278668Z outputs = self.model( 2025-09-07T07:07:42.4278932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4279015Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4279281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4279351Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4279569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4279677Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4279925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4280028Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4280283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4280369Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4280380Z 2025-09-07T07:07:42.4280488Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4280701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4280780Z return mod(**inputs) 2025-09-07T07:07:42.4281051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4281130Z outputs = self.model( 2025-09-07T07:07:42.4281385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4281459Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4281722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4281814Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4282047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4282126Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4282386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4282506Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4282775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4282945Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4282950Z 2025-09-07T07:07:42.4283052Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4283257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4283321Z return mod(**inputs) 2025-09-07T07:07:42.4283571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4283646Z outputs = self.model( 2025-09-07T07:07:42.4283898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4283991Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4284243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4284316Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4284541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4284618Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4284876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4284984Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4285239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4285318Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4285321Z 2025-09-07T07:07:42.4285424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4285628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4285713Z return mod(**inputs) 2025-09-07T07:07:42.4285973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4286039Z outputs = self.model( 2025-09-07T07:07:42.4286298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4286378Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4286627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4286705Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4286922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4287000Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4287267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4287376Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4287640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4287746Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4287750Z 2025-09-07T07:07:42.4287838Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4287918Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4287997Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4288080Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4288185Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4288392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4288458Z return mod(**inputs) 2025-09-07T07:07:42.4288735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4288812Z outputs = self.model( 2025-09-07T07:07:42.4289079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4289162Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4289427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4289499Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4289734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4289812Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4290101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4290220Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4290477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4290574Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4290868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4291006Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4291011Z 2025-09-07T07:07:42.4291112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4291316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4291380Z return mod(**inputs) 2025-09-07T07:07:42.4291644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4291739Z outputs = self.model( 2025-09-07T07:07:42.4291998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4292077Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4292337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4292415Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4292640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4292720Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4292984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4293094Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4293363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4293459Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4293758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4293895Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4293898Z 2025-09-07T07:07:42.4294002Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4294211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4294276Z return mod(**inputs) 2025-09-07T07:07:42.4294543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4294613Z outputs = self.model( 2025-09-07T07:07:42.4294881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4294976Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4295237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4295319Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4295542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4295621Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4295889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4295998Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4296284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4296370Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4296374Z 2025-09-07T07:07:42.4296488Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4296692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4296760Z return mod(**inputs) 2025-09-07T07:07:42.4297026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4297093Z outputs = self.model( 2025-09-07T07:07:42.4297358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4297432Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4297690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4297789Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4298020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4298107Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4298373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-09-07T07:07:42.4298457Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.4298468Z 2025-09-07T07:07:42.4298573Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4298776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4298848Z return mod(**inputs) 2025-09-07T07:07:42.4299112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4299187Z outputs = self.model( 2025-09-07T07:07:42.4299449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4299524Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4299798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4299898Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4300131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4300210Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4300470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4300600Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4300604Z 2025-09-07T07:07:42.4300710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4300923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4301004Z return mod(**inputs) 2025-09-07T07:07:42.4301269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4301337Z outputs = self.model( 2025-09-07T07:07:42.4301595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4301676Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4301931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4302015Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4302238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4302330Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4302596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4302715Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4302719Z 2025-09-07T07:07:42.4302833Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4303038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4303106Z return mod(**inputs) 2025-09-07T07:07:42.4303373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4303441Z outputs = self.model( 2025-09-07T07:07:42.4303707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4303781Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4304060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4304134Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4304359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4304448Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4304703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4304792Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4304796Z 2025-09-07T07:07:42.4304898Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4305095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4305168Z return mod(**inputs) 2025-09-07T07:07:42.4305425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4305503Z outputs = self.model( 2025-09-07T07:07:42.4305922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4306040Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4306531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4306638Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4307030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4307134Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4307503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4307640Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4308122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4308370Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4308375Z 2025-09-07T07:07:42.4308527Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4308862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4308951Z return mod(**inputs) 2025-09-07T07:07:42.4309362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4309436Z outputs = self.model( 2025-09-07T07:07:42.4309688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4309802Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4310057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4310137Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4310355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4310434Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4310693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4310791Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4311048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4311128Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4311131Z 2025-09-07T07:07:42.4311241Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4311467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4311534Z return mod(**inputs) 2025-09-07T07:07:42.4311792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4311860Z outputs = self.model( 2025-09-07T07:07:42.4312118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4312189Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4312438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4312517Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4312738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4312826Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4313080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4313179Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4313437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4313546Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4313549Z 2025-09-07T07:07:42.4313637Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4313714Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4313790Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4313872Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4313972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4314180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4314253Z return mod(**inputs) 2025-09-07T07:07:42.4314528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4314597Z outputs = self.model( 2025-09-07T07:07:42.4314847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4314928Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4315176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4315252Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4315468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4315545Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4315816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4315917Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4316171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4316269Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4316560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4316703Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4316706Z 2025-09-07T07:07:42.4316809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4317022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4317085Z return mod(**inputs) 2025-09-07T07:07:42.4317339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4317422Z outputs = self.model( 2025-09-07T07:07:42.4317668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4317745Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4317993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4318068Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4318280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4318355Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4318608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4318701Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4318951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4319042Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4319334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4319458Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4319461Z 2025-09-07T07:07:42.4319788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4319998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4320061Z return mod(**inputs) 2025-09-07T07:07:42.4320318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4320383Z outputs = self.model( 2025-09-07T07:07:42.4320681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4320761Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4321006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4321084Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4321297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4321378Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4321621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-09-07T07:07:42.4321714Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:07:42.4322010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4322093Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4322103Z 2025-09-07T07:07:42.4322211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4322403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4322469Z return mod(**inputs) 2025-09-07T07:07:42.4322722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4322786Z outputs = self.model( 2025-09-07T07:07:42.4323040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4323109Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4323363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4323470Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4323685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4323765Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4324006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4324121Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4324362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-09-07T07:07:42.4324506Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:07:42.4324509Z 2025-09-07T07:07:42.4324616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4324805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4324877Z return mod(**inputs) 2025-09-07T07:07:42.4325121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4325191Z outputs = self.model( 2025-09-07T07:07:42.4325437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4325533Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4325783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4325852Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4326071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4326146Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4326392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4326520Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4326763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-09-07T07:07:42.4326848Z key_states = self.k_proj(current_states) 2025-09-07T07:07:42.4326851Z 2025-09-07T07:07:42.4326950Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4327137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4327207Z return mod(**inputs) 2025-09-07T07:07:42.4327455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4327528Z outputs = self.model( 2025-09-07T07:07:42.4327789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4327868Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4328114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4328183Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4328402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4328477Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4328727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4328831Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4329073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-09-07T07:07:42.4329166Z value_states = self.v_proj(current_states) 2025-09-07T07:07:42.4329185Z 2025-09-07T07:07:42.4329264Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4329347Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4329420Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4329494Z cudagraph partition due to non gpu ops 2025-09-07T07:07:42.4329601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4329791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4329860Z return mod(**inputs) 2025-09-07T07:07:42.4330138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4330210Z outputs = self.model( 2025-09-07T07:07:42.4330461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4330533Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4330803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4330872Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4331089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4331185Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4331432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4331548Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4331795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4331898Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4332188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:07:42.4332343Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:07:42.4332347Z 2025-09-07T07:07:42.4332447Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4332637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4332710Z return mod(**inputs) 2025-09-07T07:07:42.4332957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4333028Z outputs = self.model( 2025-09-07T07:07:42.4333275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4333345Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4333622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4333693Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4333908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4333983Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4334221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4334331Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4334568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-09-07T07:07:42.4334666Z attn_output, attn_weights = attention_interface( 2025-09-07T07:07:42.4334939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:07:42.4335044Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:07:42.4335063Z 2025-09-07T07:07:42.4335162Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4335348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4335419Z return mod(**inputs) 2025-09-07T07:07:42.4335666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4335738Z outputs = self.model( 2025-09-07T07:07:42.4335983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4336051Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4336303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4336374Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4336596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4336675Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4336930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-09-07T07:07:42.4337057Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:07:42.4337316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-09-07T07:07:42.4337404Z attn_output = self.out_proj(attn_output) 2025-09-07T07:07:42.4337407Z 2025-09-07T07:07:42.4337505Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4337701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4337765Z return mod(**inputs) 2025-09-07T07:07:42.4338013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4338105Z outputs = self.model( 2025-09-07T07:07:42.4338350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4338429Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4338679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4338756Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4338973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4339052Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4339333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4339458Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4339464Z 2025-09-07T07:07:42.4339576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4339778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4339846Z return mod(**inputs) 2025-09-07T07:07:42.4340123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4340196Z outputs = self.model( 2025-09-07T07:07:42.4340477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4340554Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4340832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4340921Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4341176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4341268Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4341539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-09-07T07:07:42.4341675Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:07:42.4341679Z 2025-09-07T07:07:42.4341790Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4342000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4342078Z return mod(**inputs) 2025-09-07T07:07:42.4342350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4342429Z outputs = self.model( 2025-09-07T07:07:42.4342699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4342780Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4343058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4343162Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4343412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4343497Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4343787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-09-07T07:07:42.4343875Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:07:42.4343879Z 2025-09-07T07:07:42.4343987Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4344213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4344284Z return mod(**inputs) 2025-09-07T07:07:42.4344584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-09-07T07:07:42.4344657Z outputs = self.model( 2025-09-07T07:07:42.4344929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-09-07T07:07:42.4345018Z decoder_outputs = self.decoder( 2025-09-07T07:07:42.4345294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-09-07T07:07:42.4345378Z layer_outputs = decoder_layer( 2025-09-07T07:07:42.4345623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:07:42.4345849Z return super().__call__(*args, **kwargs) 2025-09-07T07:07:42.4347662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-09-07T07:07:42.4347939Z hidden_states = residual + hidden_states 2025-09-07T07:07:42.4347948Z 2025-09-07T07:07:42.4348095Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4348362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4348454Z return mod(**inputs) 2025-09-07T07:07:42.4348773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1422, in forward 2025-09-07T07:07:42.4348865Z lm_logits = self.lm_head(outputs[0]) 2025-09-07T07:07:42.4348869Z 2025-09-07T07:07:42.4348993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:07:42.4349243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:07:42.4349327Z return mod(**inputs) 2025-09-07T07:07:42.4349770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1429, in forward 2025-09-07T07:07:42.4349957Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:07:42.4349969Z 2025-09-07T07:07:57.0066102Z Compilation time (from dynamo_timed): 30.305467005 2025-09-07T07:07:57.0153474Z pass 2025-09-07T07:07:57.0153924Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:07:57.0154875Z TIMING: _recursive_pre_grad_passes:0.0156 _recursive_joint_graph_passes:0.80927 _recursive_post_grad_passes:0.15831 async_compile.wait:0.79628 code_gen:14.12558 inductor_compile:17.2476 backend_compile:24.45867 gc:0.0015 entire_frame_compile:30.30547 total_wall_time:30.30547 2025-09-07T07:07:57.0155999Z STATS: call_* op count: 1014 | FakeTensorMode.__torch_dispatch__:33758 | FakeTensor.__torch_dispatch__:10654 | ProxyTorchDispatchMode.__torch_dispatch__:12417 2025-09-07T07:07:57.0156626Z Dynamo produced 1 graphs covering 1014 ops with 0 graph breaks (0 unique) 2025-09-07T07:08:00.1607086Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:08:00.1608352Z import pynvml # type: ignore[import] 2025-09-07T07:08:02.9183246Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:08:02.9185182Z from pkg_resources import resource_filename 2025-09-07T07:08:03.6106558Z 2025-09-07T07:08:06.3234694Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:08:06.3239016Z loading model: 0it [00:02, ?it/s] 2025-09-07T07:08:06.3244572Z cpu eval MBartForCausalLM 2025-09-07T07:08:07.9771579Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:08:08.6063780Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:08:09.2487386Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:08:17.0124358Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0130318Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0136013Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0141130Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0143852Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0144201Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0144811Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0145074Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0145317Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0145570Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0146084Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0146323Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0146609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0147067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0147443Z return mod(**inputs) 2025-09-07T07:08:17.0147894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0148355Z outputs = self.model.decoder( 2025-09-07T07:08:17.0148834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0149288Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0149772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0150198Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0150649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0151129Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0151617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0152166Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0152425Z 2025-09-07T07:08:17.0152564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0152976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0153341Z return mod(**inputs) 2025-09-07T07:08:17.0153770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0154261Z outputs = self.model.decoder( 2025-09-07T07:08:17.0154734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0155276Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0155801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0156220Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0156659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0157133Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0157605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0158133Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0158295Z 2025-09-07T07:08:17.0158461Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0158874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0159248Z return mod(**inputs) 2025-09-07T07:08:17.0159663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0160114Z outputs = self.model.decoder( 2025-09-07T07:08:17.0160572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0161016Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0161403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0161850Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0162310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0162789Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0163258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0163726Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0163890Z 2025-09-07T07:08:17.0163980Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0164218Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0164449Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0164670Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0164935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0165351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0165726Z return mod(**inputs) 2025-09-07T07:08:17.0166218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0166926Z outputs = self.model.decoder( 2025-09-07T07:08:17.0167368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0167804Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0168218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0168631Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0169056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0169534Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0169981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0170437Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0170929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0171465Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0171691Z 2025-09-07T07:08:17.0171812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0172197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0172549Z return mod(**inputs) 2025-09-07T07:08:17.0172947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0173383Z outputs = self.model.decoder( 2025-09-07T07:08:17.0173791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0174210Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0174607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0175007Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0175446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0175897Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0176353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0176817Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0177318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0177866Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0178046Z 2025-09-07T07:08:17.0178158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0178555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0178908Z return mod(**inputs) 2025-09-07T07:08:17.0179301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0179735Z outputs = self.model.decoder( 2025-09-07T07:08:17.0180143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0180565Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0180945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0181346Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0181772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0182266Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0182728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0183165Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0183314Z 2025-09-07T07:08:17.0183436Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0183842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0184219Z return mod(**inputs) 2025-09-07T07:08:17.0184629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0185061Z outputs = self.model.decoder( 2025-09-07T07:08:17.0185485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0186047Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0186455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0186871Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0187353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0187844Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0188124Z 2025-09-07T07:08:17.0188286Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0188703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0189071Z return mod(**inputs) 2025-09-07T07:08:17.0189484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0189917Z outputs = self.model.decoder( 2025-09-07T07:08:17.0190380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0190814Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0191202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0191597Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0192035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0192518Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0192956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0193346Z return self.act(input) 2025-09-07T07:08:17.0193491Z 2025-09-07T07:08:17.0193618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0194015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0194375Z return mod(**inputs) 2025-09-07T07:08:17.0194775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0195201Z outputs = self.model.decoder( 2025-09-07T07:08:17.0195611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0196033Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0196419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0196812Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0197236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0197693Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0197853Z 2025-09-07T07:08:17.0197967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0198356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0198713Z return mod(**inputs) 2025-09-07T07:08:17.0199101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0199520Z outputs = self.model.decoder( 2025-09-07T07:08:17.0199931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0200352Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0200721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0201114Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0201540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0201989Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0202440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0202993Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0203223Z 2025-09-07T07:08:17.0203339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0203727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0204087Z return mod(**inputs) 2025-09-07T07:08:17.0204465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0204860Z outputs = self.model.decoder( 2025-09-07T07:08:17.0205284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0205687Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0206046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0206419Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0206820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0207244Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0207667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0208077Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0208215Z 2025-09-07T07:08:17.0208339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0208744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0209076Z return mod(**inputs) 2025-09-07T07:08:17.0209449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0209851Z outputs = self.model.decoder( 2025-09-07T07:08:17.0210239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0210637Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0210993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0211364Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0211760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0212177Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0212635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0213083Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0213230Z 2025-09-07T07:08:17.0213321Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0213537Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0213756Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0213971Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0214211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0214573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0214911Z return mod(**inputs) 2025-09-07T07:08:17.0215292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0215700Z outputs = self.model.decoder( 2025-09-07T07:08:17.0216097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0216488Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0216847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0217242Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0217642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0218062Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0218475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0218898Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0219359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0220199Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0220404Z 2025-09-07T07:08:17.0220527Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0220918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0221282Z return mod(**inputs) 2025-09-07T07:08:17.0221688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0222120Z outputs = self.model.decoder( 2025-09-07T07:08:17.0222573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0223036Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0223452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0223855Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0224281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0224720Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0225163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0225667Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0226234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0226757Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0226939Z 2025-09-07T07:08:17.0227055Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0227478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0227881Z return mod(**inputs) 2025-09-07T07:08:17.0228283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0228710Z outputs = self.model.decoder( 2025-09-07T07:08:17.0229136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0229561Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0229940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0230342Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0230757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0231202Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0231652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0232099Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0232245Z 2025-09-07T07:08:17.0232364Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0232783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0233134Z return mod(**inputs) 2025-09-07T07:08:17.0233526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0233954Z outputs = self.model.decoder( 2025-09-07T07:08:17.0234359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0234784Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0235160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0235553Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0235995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0236466Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0236655Z 2025-09-07T07:08:17.0236761Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0237126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0237463Z return mod(**inputs) 2025-09-07T07:08:17.0237824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0238204Z outputs = self.model.decoder( 2025-09-07T07:08:17.0238605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0238990Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0239338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0239693Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0240079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0240512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0240903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0241252Z return self.act(input) 2025-09-07T07:08:17.0241363Z 2025-09-07T07:08:17.0241467Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0241827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0242174Z return mod(**inputs) 2025-09-07T07:08:17.0242544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0242936Z outputs = self.model.decoder( 2025-09-07T07:08:17.0243330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0243713Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0244060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0244424Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0244808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0245201Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0245343Z 2025-09-07T07:08:17.0245448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0245811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0246140Z return mod(**inputs) 2025-09-07T07:08:17.0246492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0246904Z outputs = self.model.decoder( 2025-09-07T07:08:17.0247285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0247669Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0248011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0248374Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0248775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:08:17.0249181Z hidden_states = residual + hidden_states 2025-09-07T07:08:17.0249323Z 2025-09-07T07:08:17.0249460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0249830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0250154Z return mod(**inputs) 2025-09-07T07:08:17.0250520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0250904Z outputs = self.model.decoder( 2025-09-07T07:08:17.0251283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0251660Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0252014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0252409Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0252806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0253223Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0253645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0254122Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0254329Z 2025-09-07T07:08:17.0254443Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0254810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0255136Z return mod(**inputs) 2025-09-07T07:08:17.0255507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0255909Z outputs = self.model.decoder( 2025-09-07T07:08:17.0256297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0256739Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0257093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0257464Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0257862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0258287Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0258701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0259114Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0259263Z 2025-09-07T07:08:17.0259372Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0259752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0260101Z return mod(**inputs) 2025-09-07T07:08:17.0260480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0260891Z outputs = self.model.decoder( 2025-09-07T07:08:17.0261320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0261731Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0262098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0262478Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0262888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0263336Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0263805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0264238Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0264400Z 2025-09-07T07:08:17.0264488Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0264723Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0264957Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0265174Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0265405Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0265903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0266288Z return mod(**inputs) 2025-09-07T07:08:17.0266711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0267172Z outputs = self.model.decoder( 2025-09-07T07:08:17.0267580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0267986Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0268352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0268723Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0269115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0269533Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0269944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0270365Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0270823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0271341Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0271539Z 2025-09-07T07:08:17.0271646Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0272015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0272351Z return mod(**inputs) 2025-09-07T07:08:17.0272722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0273153Z outputs = self.model.decoder( 2025-09-07T07:08:17.0273562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0273962Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0274323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0274688Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0275091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0275514Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0275929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0276380Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0276829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0277302Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0277476Z 2025-09-07T07:08:17.0277583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0277954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0278281Z return mod(**inputs) 2025-09-07T07:08:17.0278679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0279082Z outputs = self.model.decoder( 2025-09-07T07:08:17.0279495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0279907Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0280262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0280639Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0281044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0281492Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0281912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0282313Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0282461Z 2025-09-07T07:08:17.0282570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0282958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0283311Z return mod(**inputs) 2025-09-07T07:08:17.0283697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0284131Z outputs = self.model.decoder( 2025-09-07T07:08:17.0284552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0284952Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0285318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0285720Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0286154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0286630Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0286832Z 2025-09-07T07:08:17.0286955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0287350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0287712Z return mod(**inputs) 2025-09-07T07:08:17.0288173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0288619Z outputs = self.model.decoder( 2025-09-07T07:08:17.0289049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0289480Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0289866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0290270Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0290696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0291191Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0291608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0291995Z return self.act(input) 2025-09-07T07:08:17.0292125Z 2025-09-07T07:08:17.0292238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0292639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0292984Z return mod(**inputs) 2025-09-07T07:08:17.0293391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0293840Z outputs = self.model.decoder( 2025-09-07T07:08:17.0294256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0294685Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0295049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0295415Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0295837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0296251Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0296395Z 2025-09-07T07:08:17.0296531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0296896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0297256Z return mod(**inputs) 2025-09-07T07:08:17.0297637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0298042Z outputs = self.model.decoder( 2025-09-07T07:08:17.0298434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0298866Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0299229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0299627Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0300033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0300461Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0300913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0301404Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0301624Z 2025-09-07T07:08:17.0301746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0302141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0302490Z return mod(**inputs) 2025-09-07T07:08:17.0302886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0303331Z outputs = self.model.decoder( 2025-09-07T07:08:17.0303741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0304163Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0304545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0304941Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0305362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0305920Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0306380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0306823Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0306993Z 2025-09-07T07:08:17.0307102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0307486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0307832Z return mod(**inputs) 2025-09-07T07:08:17.0308215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0308653Z outputs = self.model.decoder( 2025-09-07T07:08:17.0309054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0309436Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0309777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0310136Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0310523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0310939Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0311395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0311827Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0311994Z 2025-09-07T07:08:17.0312082Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0312319Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0312551Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0312773Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0313037Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0313432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0313767Z return mod(**inputs) 2025-09-07T07:08:17.0314147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0314545Z outputs = self.model.decoder( 2025-09-07T07:08:17.0314943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0315354Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0315758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0316141Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0316563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0317015Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0317459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0317908Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0318388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0318919Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0319126Z 2025-09-07T07:08:17.0319241Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0319759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0320123Z return mod(**inputs) 2025-09-07T07:08:17.0320513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0321000Z outputs = self.model.decoder( 2025-09-07T07:08:17.0321398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0321832Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0322216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0322611Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0323047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0323504Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0323970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0324370Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0324828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0325296Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0325461Z 2025-09-07T07:08:17.0325576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0325963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0326308Z return mod(**inputs) 2025-09-07T07:08:17.0326731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0327157Z outputs = self.model.decoder( 2025-09-07T07:08:17.0327571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0327992Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0328375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0328767Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0329192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0329636Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0330072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0330504Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0330700Z 2025-09-07T07:08:17.0330814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0331209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0331577Z return mod(**inputs) 2025-09-07T07:08:17.0331963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0332399Z outputs = self.model.decoder( 2025-09-07T07:08:17.0332823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0333287Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0333675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0334066Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0334493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0334969Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0335159Z 2025-09-07T07:08:17.0335279Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0335663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0336046Z return mod(**inputs) 2025-09-07T07:08:17.0336441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0336875Z outputs = self.model.decoder( 2025-09-07T07:08:17.0337336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0337725Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0338102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0338506Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0338954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0339427Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0339845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0340219Z return self.act(input) 2025-09-07T07:08:17.0340348Z 2025-09-07T07:08:17.0340462Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0340853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0341197Z return mod(**inputs) 2025-09-07T07:08:17.0341635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0342054Z outputs = self.model.decoder( 2025-09-07T07:08:17.0342471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0342886Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0343256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0343648Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0344068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0344496Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0344648Z 2025-09-07T07:08:17.0344766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0345147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0345500Z return mod(**inputs) 2025-09-07T07:08:17.0345959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0346432Z outputs = self.model.decoder( 2025-09-07T07:08:17.0346839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0347257Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0347624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0348000Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0348403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:08:17.0348800Z hidden_states = residual + hidden_states 2025-09-07T07:08:17.0348949Z 2025-09-07T07:08:17.0349056Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0349427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0349760Z return mod(**inputs) 2025-09-07T07:08:17.0350123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0350525Z outputs = self.model.decoder( 2025-09-07T07:08:17.0350920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0351340Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0351704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0352070Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0352496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0352926Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0353358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0353868Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0354079Z 2025-09-07T07:08:17.0354185Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0354567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0354904Z return mod(**inputs) 2025-09-07T07:08:17.0355278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0355661Z outputs = self.model.decoder( 2025-09-07T07:08:17.0356042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0356454Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0356802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0357164Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0357550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0357969Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0358389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0358775Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0358908Z 2025-09-07T07:08:17.0359016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0359364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0359691Z return mod(**inputs) 2025-09-07T07:08:17.0360050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0360466Z outputs = self.model.decoder( 2025-09-07T07:08:17.0360836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0361223Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0361569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0361938Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0362337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0362750Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0363172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0363590Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0363736Z 2025-09-07T07:08:17.0363828Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0364054Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0364267Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0364483Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0364726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0365126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0365453Z return mod(**inputs) 2025-09-07T07:08:17.0365836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0366231Z outputs = self.model.decoder( 2025-09-07T07:08:17.0366612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0366997Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0367351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0367745Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0368136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0368546Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0368941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0369347Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0369789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0370272Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0370475Z 2025-09-07T07:08:17.0370588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0370942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0371266Z return mod(**inputs) 2025-09-07T07:08:17.0371632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0372025Z outputs = self.model.decoder( 2025-09-07T07:08:17.0372420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0372807Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0373163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0373532Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0373934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0374374Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0374804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0375211Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0375653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0376107Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0376267Z 2025-09-07T07:08:17.0376369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0376727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0377046Z return mod(**inputs) 2025-09-07T07:08:17.0377408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0377796Z outputs = self.model.decoder( 2025-09-07T07:08:17.0378172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0378565Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0378918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0379314Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0379732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0380179Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0380625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0381053Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0381203Z 2025-09-07T07:08:17.0381325Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0381734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0382085Z return mod(**inputs) 2025-09-07T07:08:17.0382481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0382917Z outputs = self.model.decoder( 2025-09-07T07:08:17.0383361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0383796Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0384189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0384602Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0385065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0385552Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0385842Z 2025-09-07T07:08:17.0385968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0386376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0386742Z return mod(**inputs) 2025-09-07T07:08:17.0387141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0387533Z outputs = self.model.decoder( 2025-09-07T07:08:17.0387924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0388316Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0388675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0389045Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0389463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0389900Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0390296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0390656Z return self.act(input) 2025-09-07T07:08:17.0390766Z 2025-09-07T07:08:17.0390867Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0391228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0391589Z return mod(**inputs) 2025-09-07T07:08:17.0391979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0392408Z outputs = self.model.decoder( 2025-09-07T07:08:17.0392825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0393305Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0393661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0394056Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0394452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0394858Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0395013Z 2025-09-07T07:08:17.0395117Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0395482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0395808Z return mod(**inputs) 2025-09-07T07:08:17.0396167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0396572Z outputs = self.model.decoder( 2025-09-07T07:08:17.0396988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0397385Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0397742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0398112Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0398502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0398916Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0399324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0399809Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0400017Z 2025-09-07T07:08:17.0400123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0400484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0400817Z return mod(**inputs) 2025-09-07T07:08:17.0401185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0401625Z outputs = self.model.decoder( 2025-09-07T07:08:17.0402025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0402426Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0402796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0403208Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0403626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0404095Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0404513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0404924Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0405057Z 2025-09-07T07:08:17.0405168Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0405519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0405839Z return mod(**inputs) 2025-09-07T07:08:17.0406211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0406611Z outputs = self.model.decoder( 2025-09-07T07:08:17.0406993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0407396Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0407753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0408121Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0408553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0408960Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0409378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0409786Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0409928Z 2025-09-07T07:08:17.0410019Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0410233Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0410450Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0410661Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0410920Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0411287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0411616Z return mod(**inputs) 2025-09-07T07:08:17.0411989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0412385Z outputs = self.model.decoder( 2025-09-07T07:08:17.0412772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0413155Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0413512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0413913Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0414344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0414791Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0415237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0415692Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0416152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0416654Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0416843Z 2025-09-07T07:08:17.0416958Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0417323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0417656Z return mod(**inputs) 2025-09-07T07:08:17.0418060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0418458Z outputs = self.model.decoder( 2025-09-07T07:08:17.0418841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0419241Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0419723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0420127Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0420554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0420994Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0421446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0421899Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0422417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0422932Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0423187Z 2025-09-07T07:08:17.0423303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0423702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0424059Z return mod(**inputs) 2025-09-07T07:08:17.0424464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0424911Z outputs = self.model.decoder( 2025-09-07T07:08:17.0425335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0425819Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0426268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0426674Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0427100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0427561Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0428015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0428464Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0428618Z 2025-09-07T07:08:17.0428742Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0429185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0429555Z return mod(**inputs) 2025-09-07T07:08:17.0429968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0430406Z outputs = self.model.decoder( 2025-09-07T07:08:17.0430824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0431253Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0431639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0448556Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0449086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0449577Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0449803Z 2025-09-07T07:08:17.0449927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0450490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0450860Z return mod(**inputs) 2025-09-07T07:08:17.0451279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0451730Z outputs = self.model.decoder( 2025-09-07T07:08:17.0452162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0452598Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0452988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0453383Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0453818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0454292Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0454713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0455090Z return self.act(input) 2025-09-07T07:08:17.0455223Z 2025-09-07T07:08:17.0455391Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0455802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0456159Z return mod(**inputs) 2025-09-07T07:08:17.0456555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0456985Z outputs = self.model.decoder( 2025-09-07T07:08:17.0457405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0457831Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0458211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0458641Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0459076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0459511Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0459665Z 2025-09-07T07:08:17.0459792Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0460179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0460533Z return mod(**inputs) 2025-09-07T07:08:17.0460932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0461358Z outputs = self.model.decoder( 2025-09-07T07:08:17.0461804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0462218Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0462606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0462993Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0463409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:08:17.0463839Z hidden_states = residual + hidden_states 2025-09-07T07:08:17.0463993Z 2025-09-07T07:08:17.0464107Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0464504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0464867Z return mod(**inputs) 2025-09-07T07:08:17.0465267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0465818Z outputs = self.model.decoder( 2025-09-07T07:08:17.0466258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0466691Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0467073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0467480Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0467916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0468385Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0468847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0469363Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0469603Z 2025-09-07T07:08:17.0469722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0470128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0470488Z return mod(**inputs) 2025-09-07T07:08:17.0470892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0471344Z outputs = self.model.decoder( 2025-09-07T07:08:17.0471767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0472207Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0472596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0473000Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0473438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0473928Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0474391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0474836Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0474986Z 2025-09-07T07:08:17.0475104Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0475511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0475881Z return mod(**inputs) 2025-09-07T07:08:17.0476291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0476735Z outputs = self.model.decoder( 2025-09-07T07:08:17.0477186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0477629Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0478025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0478437Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0478864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0479321Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0479778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0480270Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0480425Z 2025-09-07T07:08:17.0480522Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0480751Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0480982Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0481234Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0481492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0481879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0482248Z return mod(**inputs) 2025-09-07T07:08:17.0482668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0483105Z outputs = self.model.decoder( 2025-09-07T07:08:17.0483530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0484020Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0484396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0484796Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0485223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0485670Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0486113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0486582Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0487071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0487593Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0487793Z 2025-09-07T07:08:17.0487907Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0488296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0488647Z return mod(**inputs) 2025-09-07T07:08:17.0489044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0489486Z outputs = self.model.decoder( 2025-09-07T07:08:17.0489894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0490317Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0490692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0491082Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0491510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0491947Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0492407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0492855Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0493343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0493835Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0494024Z 2025-09-07T07:08:17.0494137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0494531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0494885Z return mod(**inputs) 2025-09-07T07:08:17.0495278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0495698Z outputs = self.model.decoder( 2025-09-07T07:08:17.0496125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0496572Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0496955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0497358Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0497772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0498213Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0498654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0499091Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0499238Z 2025-09-07T07:08:17.0499351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0499744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0500101Z return mod(**inputs) 2025-09-07T07:08:17.0500511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0500942Z outputs = self.model.decoder( 2025-09-07T07:08:17.0501358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0501807Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0502194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0502595Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0503014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0503477Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0503678Z 2025-09-07T07:08:17.0503798Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0504198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0504604Z return mod(**inputs) 2025-09-07T07:08:17.0505015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0505461Z outputs = self.model.decoder( 2025-09-07T07:08:17.0505987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0506439Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0506833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0507240Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0507700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0508183Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0508623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0509011Z return self.act(input) 2025-09-07T07:08:17.0509137Z 2025-09-07T07:08:17.0509252Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0509658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0510020Z return mod(**inputs) 2025-09-07T07:08:17.0510436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0510876Z outputs = self.model.decoder( 2025-09-07T07:08:17.0511301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0511735Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0512128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0512560Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0512985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0513426Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0513586Z 2025-09-07T07:08:17.0513704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0514112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0514455Z return mod(**inputs) 2025-09-07T07:08:17.0514847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0515276Z outputs = self.model.decoder( 2025-09-07T07:08:17.0515691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0516114Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0516482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0516871Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0517321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0517768Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0518208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0518703Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0518932Z 2025-09-07T07:08:17.0519046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0519438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0519952Z return mod(**inputs) 2025-09-07T07:08:17.0520399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0520828Z outputs = self.model.decoder( 2025-09-07T07:08:17.0521243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0521661Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0522035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0522417Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0522841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0523321Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0523773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0524203Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0524349Z 2025-09-07T07:08:17.0524463Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0524853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0525208Z return mod(**inputs) 2025-09-07T07:08:17.0525602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0526019Z outputs = self.model.decoder( 2025-09-07T07:08:17.0526431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0526852Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0527232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0527658Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0528073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0528522Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0528942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0529375Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0529526Z 2025-09-07T07:08:17.0529622Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0529852Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0530082Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0530309Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0530569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0530958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0531311Z return mod(**inputs) 2025-09-07T07:08:17.0531705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0532125Z outputs = self.model.decoder( 2025-09-07T07:08:17.0532575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0533022Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0533412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0533820Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0534251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0534701Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0535182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0535626Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0536120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0536649Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0536854Z 2025-09-07T07:08:17.0536968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0537363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0537713Z return mod(**inputs) 2025-09-07T07:08:17.0538106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0538552Z outputs = self.model.decoder( 2025-09-07T07:08:17.0538970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0539389Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0539775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0540168Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0540584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0541030Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0541486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0541934Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0542417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0542940Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0543121Z 2025-09-07T07:08:17.0543232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0543640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0544004Z return mod(**inputs) 2025-09-07T07:08:17.0544407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0544832Z outputs = self.model.decoder( 2025-09-07T07:08:17.0545251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0545677Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0546129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0546534Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0546982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0547429Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0547885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0548347Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0548496Z 2025-09-07T07:08:17.0548611Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0549011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0549365Z return mod(**inputs) 2025-09-07T07:08:17.0549758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0550178Z outputs = self.model.decoder( 2025-09-07T07:08:17.0551440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0551887Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0552281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0552700Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0553124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0553603Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0553800Z 2025-09-07T07:08:17.0553914Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0554313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0554717Z return mod(**inputs) 2025-09-07T07:08:17.0555107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0555544Z outputs = self.model.decoder( 2025-09-07T07:08:17.0555959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0556379Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0556758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0557148Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0557569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0558045Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0558453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0558820Z return self.act(input) 2025-09-07T07:08:17.0558943Z 2025-09-07T07:08:17.0559051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0559422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0559757Z return mod(**inputs) 2025-09-07T07:08:17.0560137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0560524Z outputs = self.model.decoder( 2025-09-07T07:08:17.0560914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0561315Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0561689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0562073Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0562503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0562934Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0563084Z 2025-09-07T07:08:17.0563203Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0563639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0563987Z return mod(**inputs) 2025-09-07T07:08:17.0564380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0564788Z outputs = self.model.decoder( 2025-09-07T07:08:17.0565188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0565583Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0565936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0566309Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0566733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:08:17.0567147Z hidden_states = residual + hidden_states 2025-09-07T07:08:17.0567289Z 2025-09-07T07:08:17.0567394Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0567617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0567689Z return mod(**inputs) 2025-09-07T07:08:17.0567972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0568051Z outputs = self.model.decoder( 2025-09-07T07:08:17.0568352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0568433Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0568675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0568767Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0569036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0569153Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0569431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0569589Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0569592Z 2025-09-07T07:08:17.0569704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0569907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0570001Z return mod(**inputs) 2025-09-07T07:08:17.0570264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0570345Z outputs = self.model.decoder( 2025-09-07T07:08:17.0570617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0570697Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0570942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0571026Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0571307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0571414Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0571688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0571788Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0571792Z 2025-09-07T07:08:17.0571904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0572126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0572220Z return mod(**inputs) 2025-09-07T07:08:17.0572491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0572577Z outputs = self.model.decoder( 2025-09-07T07:08:17.0572845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0572930Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0573169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0573261Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0573551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0573665Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0573932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0574020Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0574023Z 2025-09-07T07:08:17.0574111Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0574193Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0574273Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0574357Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0574460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0574688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0574759Z return mod(**inputs) 2025-09-07T07:08:17.0575033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0575118Z outputs = self.model.decoder( 2025-09-07T07:08:17.0575391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0575477Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0575714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0575805Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0576078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0576185Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0576469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0576598Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0576925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0577070Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0577074Z 2025-09-07T07:08:17.0577183Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0577403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0577474Z return mod(**inputs) 2025-09-07T07:08:17.0577757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0577838Z outputs = self.model.decoder( 2025-09-07T07:08:17.0580431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0581433Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0581818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0582124Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0582542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0582672Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0582969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0583081Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0583441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0583570Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0583578Z 2025-09-07T07:08:17.0583789Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0584040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0584169Z return mod(**inputs) 2025-09-07T07:08:17.0584487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0584578Z outputs = self.model.decoder( 2025-09-07T07:08:17.0584882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0584966Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0585267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0585363Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0585667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0586013Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0586318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0586427Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0586433Z 2025-09-07T07:08:17.0586557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0586864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0586957Z return mod(**inputs) 2025-09-07T07:08:17.0587255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0587354Z outputs = self.model.decoder( 2025-09-07T07:08:17.0587697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0587789Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0588917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0589053Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0589508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0589693Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0589699Z 2025-09-07T07:08:17.0589864Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0590191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0590281Z return mod(**inputs) 2025-09-07T07:08:17.0590725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0590831Z outputs = self.model.decoder( 2025-09-07T07:08:17.0591273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0591373Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0591783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0591887Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0592295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0592480Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0592762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0592849Z return self.act(input) 2025-09-07T07:08:17.0592855Z 2025-09-07T07:08:17.0592973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0593232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0593317Z return mod(**inputs) 2025-09-07T07:08:17.0593603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0593699Z outputs = self.model.decoder( 2025-09-07T07:08:17.0593980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0594059Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0594339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0594428Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0594748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0594850Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0594855Z 2025-09-07T07:08:17.0594978Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0595211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0595284Z return mod(**inputs) 2025-09-07T07:08:17.0595577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0595656Z outputs = self.model.decoder( 2025-09-07T07:08:17.0595954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0596034Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0596282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0596420Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0596703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0596822Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0597104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0597281Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0597285Z 2025-09-07T07:08:17.0597403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0597640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0597721Z return mod(**inputs) 2025-09-07T07:08:17.0598000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0598088Z outputs = self.model.decoder( 2025-09-07T07:08:17.0598441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0598520Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0598782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0598889Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0599177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0599286Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0599572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0599669Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0599673Z 2025-09-07T07:08:17.0599804Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0600065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0600137Z return mod(**inputs) 2025-09-07T07:08:17.0600418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0600506Z outputs = self.model.decoder( 2025-09-07T07:08:17.0600799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0600885Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0601136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0601221Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0601544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0601654Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0601950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0602047Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0602053Z 2025-09-07T07:08:17.0602152Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0602242Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0602326Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0602416Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0602529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0602748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0602826Z return mod(**inputs) 2025-09-07T07:08:17.0603137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0603249Z outputs = self.model.decoder( 2025-09-07T07:08:17.0603543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0603631Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0603883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0603970Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0604263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0604374Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0604668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0604779Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0605124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0605287Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0605291Z 2025-09-07T07:08:17.0605408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0605662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0605737Z return mod(**inputs) 2025-09-07T07:08:17.0606027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0606107Z outputs = self.model.decoder( 2025-09-07T07:08:17.0606409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0606497Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0606747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0606859Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0607142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0607250Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0607541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0607646Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0607983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0608106Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0608110Z 2025-09-07T07:08:17.0608250Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0608475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0608547Z return mod(**inputs) 2025-09-07T07:08:17.0608847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0608931Z outputs = self.model.decoder( 2025-09-07T07:08:17.0609223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0609304Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0609549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0609646Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0609926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0610066Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0610346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0610447Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0610451Z 2025-09-07T07:08:17.0610577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0610788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0610865Z return mod(**inputs) 2025-09-07T07:08:17.0611139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0611224Z outputs = self.model.decoder( 2025-09-07T07:08:17.0611502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0611581Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0611838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0611924Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0612211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0612371Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0612375Z 2025-09-07T07:08:17.0612488Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0612716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0612788Z return mod(**inputs) 2025-09-07T07:08:17.0613075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0613156Z outputs = self.model.decoder( 2025-09-07T07:08:17.0613443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0613549Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0613796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0613892Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0614175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0614321Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0614549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0614626Z return self.act(input) 2025-09-07T07:08:17.0614630Z 2025-09-07T07:08:17.0614750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0614992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0615074Z return mod(**inputs) 2025-09-07T07:08:17.0615359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0615447Z outputs = self.model.decoder( 2025-09-07T07:08:17.0615731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0615808Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0616058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0616143Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0616430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0616522Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0616561Z 2025-09-07T07:08:17.0616675Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0616905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0616977Z return mod(**inputs) 2025-09-07T07:08:17.0617272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0617355Z outputs = self.model.decoder( 2025-09-07T07:08:17.0617643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0617730Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0617981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0618075Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0618361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:08:17.0618459Z hidden_states = residual + hidden_states 2025-09-07T07:08:17.0618465Z 2025-09-07T07:08:17.0618579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0618799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0618917Z return mod(**inputs) 2025-09-07T07:08:17.0619209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0619298Z outputs = self.model.decoder( 2025-09-07T07:08:17.0619847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0619982Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0620321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0620414Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0620802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0620914Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0621200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0621385Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0621391Z 2025-09-07T07:08:17.0621507Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0621735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0621810Z return mod(**inputs) 2025-09-07T07:08:17.0622166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0622251Z outputs = self.model.decoder( 2025-09-07T07:08:17.0622538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0622624Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0622869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0622964Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0623244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0623353Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0623641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0623730Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0623735Z 2025-09-07T07:08:17.0623853Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0624119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0624200Z return mod(**inputs) 2025-09-07T07:08:17.0624483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0624564Z outputs = self.model.decoder( 2025-09-07T07:08:17.0624855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0624937Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0625192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0625278Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0625560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0625676Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0626083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0626197Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0626248Z 2025-09-07T07:08:17.0626339Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0626435Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0626521Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0626605Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0626728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0626947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0627027Z return mod(**inputs) 2025-09-07T07:08:17.0627314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0627397Z outputs = self.model.decoder( 2025-09-07T07:08:17.0627717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0627801Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0628054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0628145Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0628429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0628548Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0628830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0628970Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0629303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0629467Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0629471Z 2025-09-07T07:08:17.0629586Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0629810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0629891Z return mod(**inputs) 2025-09-07T07:08:17.0630174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0630263Z outputs = self.model.decoder( 2025-09-07T07:08:17.0630545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0630633Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0630896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0631012Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0631303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0631409Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0631705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0631818Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0632146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0632279Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0632283Z 2025-09-07T07:08:17.0632396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0632630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0632705Z return mod(**inputs) 2025-09-07T07:08:17.0632984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0633106Z outputs = self.model.decoder( 2025-09-07T07:08:17.0633416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0633503Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0633781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0633869Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0634258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0634372Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0634708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0634804Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0634808Z 2025-09-07T07:08:17.0634933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0635172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0635242Z return mod(**inputs) 2025-09-07T07:08:17.0635513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0635588Z outputs = self.model.decoder( 2025-09-07T07:08:17.0635861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0635959Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0636186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0636312Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0636581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0636712Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0636716Z 2025-09-07T07:08:17.0636823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0637039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0637105Z return mod(**inputs) 2025-09-07T07:08:17.0637369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0637452Z outputs = self.model.decoder( 2025-09-07T07:08:17.0637715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0637829Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0638056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0638136Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0638408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0638528Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0638754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0638825Z return self.act(input) 2025-09-07T07:08:17.0638828Z 2025-09-07T07:08:17.0638932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0639147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0639216Z return mod(**inputs) 2025-09-07T07:08:17.0639483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0639559Z outputs = self.model.decoder( 2025-09-07T07:08:17.0639831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0639936Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0640163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0640251Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0640518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0640611Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0640617Z 2025-09-07T07:08:17.0640720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0640947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0641024Z return mod(**inputs) 2025-09-07T07:08:17.0641349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0641438Z outputs = self.model.decoder( 2025-09-07T07:08:17.0641722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0641805Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0642051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0642139Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0642457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0642570Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0642851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:08:17.0643017Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:08:17.0643023Z 2025-09-07T07:08:17.0643133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0643361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0643427Z return mod(**inputs) 2025-09-07T07:08:17.0643699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0643774Z outputs = self.model.decoder( 2025-09-07T07:08:17.0644037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0644145Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0644375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0644466Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0644730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0644841Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0645103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:08:17.0645187Z key_states = self.k_proj(current_states) 2025-09-07T07:08:17.0645192Z 2025-09-07T07:08:17.0645306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0645510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0645587Z return mod(**inputs) 2025-09-07T07:08:17.0645862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0645941Z outputs = self.model.decoder( 2025-09-07T07:08:17.0646209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0646316Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0646547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0646625Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0646891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0646991Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0647251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:08:17.0647350Z value_states = self.v_proj(current_states) 2025-09-07T07:08:17.0647385Z 2025-09-07T07:08:17.0647469Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0647558Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0647636Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0647794Z cudagraph partition due to non gpu ops 2025-09-07T07:08:17.0647898Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0648104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0648182Z return mod(**inputs) 2025-09-07T07:08:17.0648439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0648521Z outputs = self.model.decoder( 2025-09-07T07:08:17.0648805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0648890Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0649119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0649201Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0649469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0649566Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0649828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0649925Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0650226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:08:17.0650373Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:08:17.0650401Z 2025-09-07T07:08:17.0650507Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0650716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0650781Z return mod(**inputs) 2025-09-07T07:08:17.0651051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0651126Z outputs = self.model.decoder( 2025-09-07T07:08:17.0651404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0651489Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0651728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0651821Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0652092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0652200Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0652479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:08:17.0652607Z attn_output, attn_weights = attention_interface( 2025-09-07T07:08:17.0652929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:08:17.0653047Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:08:17.0653051Z 2025-09-07T07:08:17.0653167Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0653376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0653449Z return mod(**inputs) 2025-09-07T07:08:17.0653735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0653836Z outputs = self.model.decoder( 2025-09-07T07:08:17.0654130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0654208Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0654463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0654557Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0654840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:08:17.0654950Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:08:17.0655261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:08:17.0655357Z attn_output = self.out_proj(attn_output) 2025-09-07T07:08:17.0655369Z 2025-09-07T07:08:17.0655492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0655706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0655785Z return mod(**inputs) 2025-09-07T07:08:17.0656061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0656144Z outputs = self.model.decoder( 2025-09-07T07:08:17.0656430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0656506Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0656749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0656837Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0657143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0657269Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0657273Z 2025-09-07T07:08:17.0657380Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0657608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0657678Z return mod(**inputs) 2025-09-07T07:08:17.0657963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0658040Z outputs = self.model.decoder( 2025-09-07T07:08:17.0658327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0658406Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0658641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0658738Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0659010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:08:17.0659172Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:08:17.0659406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:08:17.0659483Z return self.act(input) 2025-09-07T07:08:17.0659487Z 2025-09-07T07:08:17.0659606Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0659836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0659915Z return mod(**inputs) 2025-09-07T07:08:17.0660201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0660281Z outputs = self.model.decoder( 2025-09-07T07:08:17.0660609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0660688Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0660945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0661029Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0661317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:08:17.0661408Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:08:17.0661411Z 2025-09-07T07:08:17.0661522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0661762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0661836Z return mod(**inputs) 2025-09-07T07:08:17.0662176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-09-07T07:08:17.0662255Z outputs = self.model.decoder( 2025-09-07T07:08:17.0662544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:08:17.0662629Z layer_outputs = decoder_layer( 2025-09-07T07:08:17.0662884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:08:17.0662974Z return super().__call__(*args, **kwargs) 2025-09-07T07:08:17.0663266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:08:17.0663362Z hidden_states = residual + hidden_states 2025-09-07T07:08:17.0663366Z 2025-09-07T07:08:17.0663479Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0663720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0663812Z return mod(**inputs) 2025-09-07T07:08:17.0664084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1880, in forward 2025-09-07T07:08:17.0664176Z logits = self.lm_head(outputs[0]) 2025-09-07T07:08:17.0664180Z 2025-09-07T07:08:17.0664287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:08:17.0664495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:08:17.0664571Z return mod(**inputs) 2025-09-07T07:08:17.0664846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1886, in forward 2025-09-07T07:08:17.0665016Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:08:17.0665021Z 2025-09-07T07:08:29.3871212Z Compilation time (from dynamo_timed): 18.245421277 2025-09-07T07:08:29.4147424Z pass 2025-09-07T07:08:29.4151974Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:08:29.4153303Z TIMING: _recursive_pre_grad_passes:0.00779 _recursive_joint_graph_passes:0.37196 _recursive_post_grad_passes:0.08077 async_compile.wait:0.74604 code_gen:11.38425 inductor_compile:12.7331 backend_compile:16.01429 gc:0.00218 entire_frame_compile:18.24542 total_wall_time:18.24542 2025-09-07T07:08:29.4155709Z STATS: call_* op count: 373 | FakeTensorMode.__torch_dispatch__:13260 | FakeTensor.__torch_dispatch__:4593 | ProxyTorchDispatchMode.__torch_dispatch__:4844 2025-09-07T07:08:29.4156245Z Dynamo produced 1 graphs covering 373 ops with 0 graph breaks (0 unique) 2025-09-07T07:08:32.1991206Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:08:32.1993510Z import pynvml # type: ignore[import] 2025-09-07T07:08:34.9897160Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:08:34.9898211Z from pkg_resources import resource_filename 2025-09-07T07:08:35.6663697Z 2025-09-07T07:08:40.8482712Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:08:40.8483178Z loading model: 0it [00:05, ?it/s] 2025-09-07T07:08:40.8508755Z cpu eval MBartForConditionalGeneration 2025-09-07T07:08:44.2658634Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:08:45.5641980Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:08:46.8701638Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:09:04.3262115Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3262603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3263007Z return mod(**inputs) 2025-09-07T07:09:04.3263635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1436, in forward 2025-09-07T07:09:04.3264228Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-09-07T07:09:04.3264778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 76, in shift_tokens_right 2025-09-07T07:09:04.3265344Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-09-07T07:09:04.3265591Z 2025-09-07T07:09:04.3266645Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3266899Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3267148Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3267409Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3267647Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3267872Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3268102Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3268338Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3268555Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3268872Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3269089Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3269314Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3269552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3269924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3270273Z return mod(**inputs) 2025-09-07T07:09:04.3270673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3271103Z outputs = self.model( 2025-09-07T07:09:04.3271505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3272012Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3272422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3272829Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3273196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3273571Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3274002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3274446Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3274980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3275493Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3275719Z 2025-09-07T07:09:04.3275836Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3276246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3276601Z return mod(**inputs) 2025-09-07T07:09:04.3277003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3277402Z outputs = self.model( 2025-09-07T07:09:04.3277825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3278253Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3278687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3279120Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3279511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3279909Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3280331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3280753Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3281181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3281617Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3281796Z 2025-09-07T07:09:04.3281911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3282318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3282677Z return mod(**inputs) 2025-09-07T07:09:04.3283071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3283486Z outputs = self.model( 2025-09-07T07:09:04.3283883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3284307Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3284726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3285148Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3285524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3285918Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3286342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3286846Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3288109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3288552Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3288724Z 2025-09-07T07:09:04.3288813Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3289045Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3289268Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3289486Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3289741Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3290140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3290500Z return mod(**inputs) 2025-09-07T07:09:04.3290910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3291335Z outputs = self.model( 2025-09-07T07:09:04.3291730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3292155Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3292571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3292981Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3293364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3293777Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3294200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3294633Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3295068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3295525Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3296014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3296533Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3296732Z 2025-09-07T07:09:04.3296844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3297236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3297590Z return mod(**inputs) 2025-09-07T07:09:04.3297983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3298424Z outputs = self.model( 2025-09-07T07:09:04.3298822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3299249Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3299665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3300116Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3300507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3300891Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3301320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3301761Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3302204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3302695Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3303174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3303697Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3303881Z 2025-09-07T07:09:04.3303994Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3304384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3304745Z return mod(**inputs) 2025-09-07T07:09:04.3305157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3305598Z outputs = self.model( 2025-09-07T07:09:04.3306223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3306681Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3307105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3307547Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3307939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3308360Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3308799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3309247Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3309722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3310217Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3310377Z 2025-09-07T07:09:04.3310501Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3310906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3311273Z return mod(**inputs) 2025-09-07T07:09:04.3311687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3312130Z outputs = self.model( 2025-09-07T07:09:04.3312540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3312981Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3313430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3313885Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3314390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3314797Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3315225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3315714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3315922Z 2025-09-07T07:09:04.3316043Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3316447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3316832Z return mod(**inputs) 2025-09-07T07:09:04.3317254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3317693Z outputs = self.model( 2025-09-07T07:09:04.3318097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3318540Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3318959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3319423Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3320058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3320452Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3320918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3321379Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3321802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3322176Z return self.act(input) 2025-09-07T07:09:04.3322297Z 2025-09-07T07:09:04.3322497Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3322895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3323243Z return mod(**inputs) 2025-09-07T07:09:04.3323641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3324053Z outputs = self.model( 2025-09-07T07:09:04.3324449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3324867Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3325283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3325731Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3326116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3326512Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3326932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3327365Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3327523Z 2025-09-07T07:09:04.3327636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3328029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3328378Z return mod(**inputs) 2025-09-07T07:09:04.3328763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3329177Z outputs = self.model( 2025-09-07T07:09:04.3329572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3330024Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3330431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3330853Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3331233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3331621Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3332040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3332488Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3332927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3333433Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3333657Z 2025-09-07T07:09:04.3333778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3334170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3334517Z return mod(**inputs) 2025-09-07T07:09:04.3334943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3335356Z outputs = self.model( 2025-09-07T07:09:04.3335756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3336176Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3336585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3337004Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3337391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3337800Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3338214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3338660Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3339095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3339520Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3339667Z 2025-09-07T07:09:04.3339788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3340181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3340550Z return mod(**inputs) 2025-09-07T07:09:04.3340969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3341401Z outputs = self.model( 2025-09-07T07:09:04.3341802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3342260Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3342688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3343115Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3343499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3343890Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3344327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3344781Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3345251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3345792Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3345963Z 2025-09-07T07:09:04.3346055Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3346299Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3346540Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3346776Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3347029Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3347434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3347801Z return mod(**inputs) 2025-09-07T07:09:04.3348210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3348645Z outputs = self.model( 2025-09-07T07:09:04.3349055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3349493Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3349921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3350374Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3350752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3351155Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3351591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3352040Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3352500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3352956Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3353476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3354020Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3354228Z 2025-09-07T07:09:04.3354353Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3354755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3355110Z return mod(**inputs) 2025-09-07T07:09:04.3355516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3355945Z outputs = self.model( 2025-09-07T07:09:04.3356357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3356775Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3357188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3357604Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3357985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3358383Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3358801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3359240Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3359678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3360122Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3360610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3361124Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3361305Z 2025-09-07T07:09:04.3361422Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3361816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3362179Z return mod(**inputs) 2025-09-07T07:09:04.3362562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3362962Z outputs = self.model( 2025-09-07T07:09:04.3363350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3363761Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3364174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3364598Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3364987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3365392Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3365846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3366302Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3366707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3367134Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3367289Z 2025-09-07T07:09:04.3367399Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3367785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3368136Z return mod(**inputs) 2025-09-07T07:09:04.3368522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3368919Z outputs = self.model( 2025-09-07T07:09:04.3369297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3369699Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3370083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3370476Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3370855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3371254Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3371673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3372118Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3372306Z 2025-09-07T07:09:04.3372413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3372783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3373118Z return mod(**inputs) 2025-09-07T07:09:04.3373491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3373906Z outputs = self.model( 2025-09-07T07:09:04.3374308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3374734Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3375153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3375578Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3375959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3376346Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3376767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3377240Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3377654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3378022Z return self.act(input) 2025-09-07T07:09:04.3378152Z 2025-09-07T07:09:04.3378262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3378653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3379001Z return mod(**inputs) 2025-09-07T07:09:04.3379398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3379815Z outputs = self.model( 2025-09-07T07:09:04.3380221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3380692Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3381112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3381543Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3381943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3382332Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3382750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3383179Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3383344Z 2025-09-07T07:09:04.3383480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3383883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3384240Z return mod(**inputs) 2025-09-07T07:09:04.3384638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3385066Z outputs = self.model( 2025-09-07T07:09:04.3385473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3386035Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3386467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3386914Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3387318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3387715Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3388143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-09-07T07:09:04.3388579Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3388727Z 2025-09-07T07:09:04.3388840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3389240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3389595Z return mod(**inputs) 2025-09-07T07:09:04.3389992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3390403Z outputs = self.model( 2025-09-07T07:09:04.3390800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3391268Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3391681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3392093Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3392463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3392852Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3393273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3393711Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3394143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3394653Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3394888Z 2025-09-07T07:09:04.3395004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3395396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3395746Z return mod(**inputs) 2025-09-07T07:09:04.3396150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3396566Z outputs = self.model( 2025-09-07T07:09:04.3396954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3397379Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3397811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3398230Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3398608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3399016Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3399441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3399885Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3400314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3400742Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3400888Z 2025-09-07T07:09:04.3401009Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3401403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3401753Z return mod(**inputs) 2025-09-07T07:09:04.3402169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3402599Z outputs = self.model( 2025-09-07T07:09:04.3402994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3403413Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3403822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3404256Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3404650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3405061Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3405497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3406029Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3406499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3406962Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3407122Z 2025-09-07T07:09:04.3407218Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3407451Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3407691Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3407921Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3408183Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3408588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3408962Z return mod(**inputs) 2025-09-07T07:09:04.3409382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3409826Z outputs = self.model( 2025-09-07T07:09:04.3410248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3410689Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3411121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3411618Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3412009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3412415Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3412857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3413313Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3413769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3414232Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3414746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3415287Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3415504Z 2025-09-07T07:09:04.3415622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3416029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3416388Z return mod(**inputs) 2025-09-07T07:09:04.3416787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3417212Z outputs = self.model( 2025-09-07T07:09:04.3417658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3418093Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3418528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3418941Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3419317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3419932Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3420365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3420797Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3421239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3421686Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3422180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3422739Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3422913Z 2025-09-07T07:09:04.3423029Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3423421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3423778Z return mod(**inputs) 2025-09-07T07:09:04.3424174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3424585Z outputs = self.model( 2025-09-07T07:09:04.3424975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3425395Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3425893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3426328Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3426714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3427111Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3427534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3428012Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3428447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3428869Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3429026Z 2025-09-07T07:09:04.3429140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3429535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3429889Z return mod(**inputs) 2025-09-07T07:09:04.3430313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3430733Z outputs = self.model( 2025-09-07T07:09:04.3431135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3431565Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3431961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3432345Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3432700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3433071Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3433508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3433959Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3434141Z 2025-09-07T07:09:04.3434250Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3434634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3434975Z return mod(**inputs) 2025-09-07T07:09:04.3435335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3435715Z outputs = self.model( 2025-09-07T07:09:04.3436070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3436455Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3436836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3437250Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3437588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3437947Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3438333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3438768Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3439154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3439486Z return self.act(input) 2025-09-07T07:09:04.3439604Z 2025-09-07T07:09:04.3439707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3440067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3440390Z return mod(**inputs) 2025-09-07T07:09:04.3440745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3441130Z outputs = self.model( 2025-09-07T07:09:04.3441489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3441902Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3442279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3442659Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3443008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3443369Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3443769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3444178Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3444321Z 2025-09-07T07:09:04.3444450Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3444825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3445159Z return mod(**inputs) 2025-09-07T07:09:04.3445538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3445925Z outputs = self.model( 2025-09-07T07:09:04.3446298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3446695Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3447084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3447495Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3447853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3448235Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3448628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3449047Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3449455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3449982Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3450199Z 2025-09-07T07:09:04.3450306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3450676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3451012Z return mod(**inputs) 2025-09-07T07:09:04.3451377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3451798Z outputs = self.model( 2025-09-07T07:09:04.3452175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3452576Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3452968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3453358Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3453717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3454088Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3454489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3454902Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3455320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3455754Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3455917Z 2025-09-07T07:09:04.3456024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3456419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3456746Z return mod(**inputs) 2025-09-07T07:09:04.3457128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3457547Z outputs = self.model( 2025-09-07T07:09:04.3457944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3458369Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3458778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3459218Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3459577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3459947Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3460339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3460752Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3461163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3461571Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3461715Z 2025-09-07T07:09:04.3461822Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3462038Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3462257Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3463242Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3463496Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3463883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3464237Z return mod(**inputs) 2025-09-07T07:09:04.3464638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3465056Z outputs = self.model( 2025-09-07T07:09:04.3465453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3465988Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3466428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3466891Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3467272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3467667Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3468080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3468521Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3468958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3469408Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3469884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3470491Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3470694Z 2025-09-07T07:09:04.3470806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3471199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3471550Z return mod(**inputs) 2025-09-07T07:09:04.3471938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3472423Z outputs = self.model( 2025-09-07T07:09:04.3472822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3473246Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3473660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3474070Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3474451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3474843Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3475288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3475720Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3476133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3476552Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3477006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3477476Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3477645Z 2025-09-07T07:09:04.3477752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3478139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3478480Z return mod(**inputs) 2025-09-07T07:09:04.3478859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3479254Z outputs = self.model( 2025-09-07T07:09:04.3479621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3480016Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3480412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3480810Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3481161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3481531Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3481931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3482369Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3482781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3483184Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3483334Z 2025-09-07T07:09:04.3483439Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3483806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3484143Z return mod(**inputs) 2025-09-07T07:09:04.3484536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3484944Z outputs = self.model( 2025-09-07T07:09:04.3485317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3485713Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3486114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3486519Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3486910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3487276Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3487675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3488117Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3488297Z 2025-09-07T07:09:04.3488402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3488768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3489104Z return mod(**inputs) 2025-09-07T07:09:04.3489492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3489886Z outputs = self.model( 2025-09-07T07:09:04.3490248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3490648Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3491029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3491411Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3491749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3492110Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3492538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3492983Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3493386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3493730Z return self.act(input) 2025-09-07T07:09:04.3493854Z 2025-09-07T07:09:04.3493959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3494328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3494659Z return mod(**inputs) 2025-09-07T07:09:04.3495028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3495427Z outputs = self.model( 2025-09-07T07:09:04.3495790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3496201Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3496591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3496987Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3497333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3497700Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3498090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3498485Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3498622Z 2025-09-07T07:09:04.3498726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3499086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3499413Z return mod(**inputs) 2025-09-07T07:09:04.3499781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3500158Z outputs = self.model( 2025-09-07T07:09:04.3500520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3500944Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3501341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3501739Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3502088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3502463Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3502876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-09-07T07:09:04.3503281Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3503419Z 2025-09-07T07:09:04.3503554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3503917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3504250Z return mod(**inputs) 2025-09-07T07:09:04.3504639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3505055Z outputs = self.model( 2025-09-07T07:09:04.3505445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3505974Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3506398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3506850Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3507230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3507631Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3508033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3508479Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3508919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3509412Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3509642Z 2025-09-07T07:09:04.3509755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3510143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3510495Z return mod(**inputs) 2025-09-07T07:09:04.3510893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3511333Z outputs = self.model( 2025-09-07T07:09:04.3511733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3512152Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3512567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3512981Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3513349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3513737Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3514161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3514596Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3515031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3515458Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3515613Z 2025-09-07T07:09:04.3515726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3516152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3516509Z return mod(**inputs) 2025-09-07T07:09:04.3516893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3517311Z outputs = self.model( 2025-09-07T07:09:04.3517707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3518164Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3518585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3519019Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3519378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3519946Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3520351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3520773Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3521189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3521604Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3521752Z 2025-09-07T07:09:04.3521898Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3522124Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3522338Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3522552Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3522794Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3523162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3523494Z return mod(**inputs) 2025-09-07T07:09:04.3523877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3524275Z outputs = self.model( 2025-09-07T07:09:04.3524659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3525062Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3525447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3525877Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3526237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3526609Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3526999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3527418Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3527829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3528250Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3528709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3529196Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3529394Z 2025-09-07T07:09:04.3529500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3529877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3530209Z return mod(**inputs) 2025-09-07T07:09:04.3530580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3530999Z outputs = self.model( 2025-09-07T07:09:04.3531373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3531769Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3532158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3532553Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3532914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3533292Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3533713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3534134Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3534541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3534969Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3535433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3535912Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3536074Z 2025-09-07T07:09:04.3536185Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3536554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3536884Z return mod(**inputs) 2025-09-07T07:09:04.3537254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3537644Z outputs = self.model( 2025-09-07T07:09:04.3538010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3538414Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3538813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3539201Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3539550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3539915Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3540319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3540760Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3541174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3541578Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3541717Z 2025-09-07T07:09:04.3541824Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3542194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3542526Z return mod(**inputs) 2025-09-07T07:09:04.3542893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3543287Z outputs = self.model( 2025-09-07T07:09:04.3543664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3544071Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3544488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3544906Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3545305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3545778Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3546221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3546686Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3546877Z 2025-09-07T07:09:04.3546996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3547377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3547712Z return mod(**inputs) 2025-09-07T07:09:04.3548115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3548506Z outputs = self.model( 2025-09-07T07:09:04.3548874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3549271Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3549665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3550061Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3550414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3550772Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3551195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3551632Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3552021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3552366Z return self.act(input) 2025-09-07T07:09:04.3552479Z 2025-09-07T07:09:04.3552581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3552943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3553278Z return mod(**inputs) 2025-09-07T07:09:04.3553650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3554037Z outputs = self.model( 2025-09-07T07:09:04.3554433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3554934Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3555319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3555732Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3556102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3556496Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3556919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3557349Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3557499Z 2025-09-07T07:09:04.3557620Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3558002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3558354Z return mod(**inputs) 2025-09-07T07:09:04.3558750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3559166Z outputs = self.model( 2025-09-07T07:09:04.3559553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3560000Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3560416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3560836Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3561212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3561596Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3562021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3562463Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3562920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3563417Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3563650Z 2025-09-07T07:09:04.3563765Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3564124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3564448Z return mod(**inputs) 2025-09-07T07:09:04.3564813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3565191Z outputs = self.model( 2025-09-07T07:09:04.3565570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3565955Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3566334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3566716Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3567057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3567417Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3567809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3568202Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3568588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3568971Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3569108Z 2025-09-07T07:09:04.3569211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3569583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3569902Z return mod(**inputs) 2025-09-07T07:09:04.3570249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3570624Z outputs = self.model( 2025-09-07T07:09:04.3570980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3571359Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3571733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3572100Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3572440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3572794Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3573179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3573567Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3573978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3574407Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3574562Z 2025-09-07T07:09:04.3574652Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3574869Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3575073Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3575280Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3575514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3575881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3576195Z return mod(**inputs) 2025-09-07T07:09:04.3576567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3576944Z outputs = self.model( 2025-09-07T07:09:04.3577305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3577697Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3578088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3578467Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3578819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3579186Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3579588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3579997Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3580407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3580829Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3581285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3581765Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3581959Z 2025-09-07T07:09:04.3582065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3582430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3582759Z return mod(**inputs) 2025-09-07T07:09:04.3583134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3583542Z outputs = self.model( 2025-09-07T07:09:04.3583935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3584353Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3584764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3585173Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3585549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3586045Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3586472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3586914Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3587345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3587767Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3588222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3588727Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3588891Z 2025-09-07T07:09:04.3589007Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3589370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3589741Z return mod(**inputs) 2025-09-07T07:09:04.3590112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3590503Z outputs = self.model( 2025-09-07T07:09:04.3590868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3591286Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3591669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3592057Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3592408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3592765Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3593158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3593574Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3594035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3594467Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3594613Z 2025-09-07T07:09:04.3594727Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3595122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3595459Z return mod(**inputs) 2025-09-07T07:09:04.3595853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3596262Z outputs = self.model( 2025-09-07T07:09:04.3596657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3597081Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3597496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3597916Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3598312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3598710Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3599132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3599602Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3599790Z 2025-09-07T07:09:04.3599911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3600289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3600637Z return mod(**inputs) 2025-09-07T07:09:04.3601034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3601448Z outputs = self.model( 2025-09-07T07:09:04.3601836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3602260Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3602677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3603094Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3603501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3603884Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3604304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3604769Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3605192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3605562Z return self.act(input) 2025-09-07T07:09:04.3605683Z 2025-09-07T07:09:04.3605796Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3606214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3606547Z return mod(**inputs) 2025-09-07T07:09:04.3606918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3607305Z outputs = self.model( 2025-09-07T07:09:04.3607682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3608080Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3608473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3608868Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3609238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3609626Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3610035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3610445Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3610589Z 2025-09-07T07:09:04.3610705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3611074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3611430Z return mod(**inputs) 2025-09-07T07:09:04.3611834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3612266Z outputs = self.model( 2025-09-07T07:09:04.3612637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3613056Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3613451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3613851Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3614215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3614610Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3615037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-09-07T07:09:04.3615470Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3615621Z 2025-09-07T07:09:04.3615741Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3616130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3616485Z return mod(**inputs) 2025-09-07T07:09:04.3616886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3617287Z outputs = self.model( 2025-09-07T07:09:04.3617663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3618078Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3618467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3618860Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3619215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3619760Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3620188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3620631Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3621118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3621625Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3621852Z 2025-09-07T07:09:04.3621966Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3622362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3622716Z return mod(**inputs) 2025-09-07T07:09:04.3623117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3623536Z outputs = self.model( 2025-09-07T07:09:04.3623963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3624400Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3624830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3625259Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3625650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3626133Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3626569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3627025Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3627488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3627923Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3628084Z 2025-09-07T07:09:04.3628198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3628669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3629021Z return mod(**inputs) 2025-09-07T07:09:04.3629415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3629832Z outputs = self.model( 2025-09-07T07:09:04.3630250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3630673Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3631090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3631506Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3631880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3632280Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3632704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3633145Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3633575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3634045Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3634204Z 2025-09-07T07:09:04.3634295Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3634534Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3634767Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3634984Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3635243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3635637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3635996Z return mod(**inputs) 2025-09-07T07:09:04.3636381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3636777Z outputs = self.model( 2025-09-07T07:09:04.3637146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3637551Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3637948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3638364Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3638730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3639098Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3639517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3639931Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3640335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3640745Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3641194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3641680Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3641864Z 2025-09-07T07:09:04.3641967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3642330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3642663Z return mod(**inputs) 2025-09-07T07:09:04.3643040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3643453Z outputs = self.model( 2025-09-07T07:09:04.3643821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3644218Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3644611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3645023Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3645373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3645748Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3646148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3646564Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3646979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3647398Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3647858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3648344Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3648507Z 2025-09-07T07:09:04.3648622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3648996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3649338Z return mod(**inputs) 2025-09-07T07:09:04.3649707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3650095Z outputs = self.model( 2025-09-07T07:09:04.3650460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3650858Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3651241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3651626Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3651977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3652337Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3652717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3653128Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3653557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3653963Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3654102Z 2025-09-07T07:09:04.3654218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3654576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3654912Z return mod(**inputs) 2025-09-07T07:09:04.3655275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3655652Z outputs = self.model( 2025-09-07T07:09:04.3656012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3656414Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3656801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3657193Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3657583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3657948Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3658345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3658965Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3659146Z 2025-09-07T07:09:04.3659263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3659640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3659972Z return mod(**inputs) 2025-09-07T07:09:04.3660345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3660738Z outputs = self.model( 2025-09-07T07:09:04.3661117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3661507Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3661904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3662303Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3662688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3663056Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3663447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3663885Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3664295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3664664Z return self.act(input) 2025-09-07T07:09:04.3664793Z 2025-09-07T07:09:04.3664906Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3665286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3665638Z return mod(**inputs) 2025-09-07T07:09:04.3666118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3666537Z outputs = self.model( 2025-09-07T07:09:04.3666921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3667351Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3667625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3667699Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3667950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3668043Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3668306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3668399Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3668404Z 2025-09-07T07:09:04.3668511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3668723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3668792Z return mod(**inputs) 2025-09-07T07:09:04.3669055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3669133Z outputs = self.model( 2025-09-07T07:09:04.3669395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3669496Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3669755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3669828Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3670060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3670142Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3670405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3670499Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3670753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3670916Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3670921Z 2025-09-07T07:09:04.3671027Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3671236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3671302Z return mod(**inputs) 2025-09-07T07:09:04.3671567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3671655Z outputs = self.model( 2025-09-07T07:09:04.3671912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3671992Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3672247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3672327Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3672551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3672632Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3672916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3673013Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3673274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3673358Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3673361Z 2025-09-07T07:09:04.3673473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3673685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3673756Z return mod(**inputs) 2025-09-07T07:09:04.3674054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3674129Z outputs = self.model( 2025-09-07T07:09:04.3674412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3674491Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3674761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3674849Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3675094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3675180Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3675435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3675527Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3675787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3675895Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3675899Z 2025-09-07T07:09:04.3675988Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3676067Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3676153Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3676230Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3676332Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3676537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3676600Z return mod(**inputs) 2025-09-07T07:09:04.3676868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3676936Z outputs = self.model( 2025-09-07T07:09:04.3677195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3677278Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3677536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3677617Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3677858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3677939Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3678202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3678294Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3678556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3678667Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3678976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3679116Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3679120Z 2025-09-07T07:09:04.3679221Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3679428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3679492Z return mod(**inputs) 2025-09-07T07:09:04.3679748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3679815Z outputs = self.model( 2025-09-07T07:09:04.3680068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3680164Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3680415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3680495Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3680712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3680791Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3681049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3681138Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3681395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3681492Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3681795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3681958Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3681963Z 2025-09-07T07:09:04.3682065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3682269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3682337Z return mod(**inputs) 2025-09-07T07:09:04.3682593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3682660Z outputs = self.model( 2025-09-07T07:09:04.3682908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3682988Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3683237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3683316Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3683536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3683620Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3683866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3683980Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3684247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3684330Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3684334Z 2025-09-07T07:09:04.3684444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3684648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3684716Z return mod(**inputs) 2025-09-07T07:09:04.3684990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3685074Z outputs = self.model( 2025-09-07T07:09:04.3685338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3685413Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3685666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3685743Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3685973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3686056Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3686330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3686459Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3686462Z 2025-09-07T07:09:04.3686564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3686761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3686834Z return mod(**inputs) 2025-09-07T07:09:04.3687087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3687160Z outputs = self.model( 2025-09-07T07:09:04.3687410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3687481Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3687738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3687810Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3688056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3688134Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3688392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3688509Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3688719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3688797Z return self.act(input) 2025-09-07T07:09:04.3688800Z 2025-09-07T07:09:04.3688902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3689103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3689169Z return mod(**inputs) 2025-09-07T07:09:04.3689420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3689497Z outputs = self.model( 2025-09-07T07:09:04.3689749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3689827Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3690096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3690167Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3690392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3690469Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3690727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3690812Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3690817Z 2025-09-07T07:09:04.3690928Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3691145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3691214Z return mod(**inputs) 2025-09-07T07:09:04.3691483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3691553Z outputs = self.model( 2025-09-07T07:09:04.3691820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3691894Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3692151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3692233Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3692524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3692614Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3692864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-09-07T07:09:04.3692946Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3692957Z 2025-09-07T07:09:04.3693061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3693262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3693336Z return mod(**inputs) 2025-09-07T07:09:04.3693592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3693666Z outputs = self.model( 2025-09-07T07:09:04.3693927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3694019Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3694291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3694365Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3694601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3694682Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3694944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3695045Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3695305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3695469Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3695474Z 2025-09-07T07:09:04.3695579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3695792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3695858Z return mod(**inputs) 2025-09-07T07:09:04.3696173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3696276Z outputs = self.model( 2025-09-07T07:09:04.3696539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3696617Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3696872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3696945Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3697177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3697259Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3697536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3697632Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3697889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3697982Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3697985Z 2025-09-07T07:09:04.3698090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3698299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3698366Z return mod(**inputs) 2025-09-07T07:09:04.3698646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3698717Z outputs = self.model( 2025-09-07T07:09:04.3698978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3699060Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3699317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3699398Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3699625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3699706Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3699972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3700064Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3700331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3700437Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3700441Z 2025-09-07T07:09:04.3700529Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3700610Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3700690Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3700780Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3700884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3701092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3701158Z return mod(**inputs) 2025-09-07T07:09:04.3701416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3701492Z outputs = self.model( 2025-09-07T07:09:04.3701750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3701832Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3702092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3702167Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3702420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3702501Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3702764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3702857Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3703116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3703224Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3703540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3703688Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3703691Z 2025-09-07T07:09:04.3703798Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3704013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3704086Z return mod(**inputs) 2025-09-07T07:09:04.3704371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3704453Z outputs = self.model( 2025-09-07T07:09:04.3704733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3704838Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3705110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3705189Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3705435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3705522Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3705906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3706010Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3706292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3706397Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3706721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3706874Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3706880Z 2025-09-07T07:09:04.3706990Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3707210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3707284Z return mod(**inputs) 2025-09-07T07:09:04.3707557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3707644Z outputs = self.model( 2025-09-07T07:09:04.3707901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3707983Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3708239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3708324Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3708549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3708630Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3708892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3709002Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3709266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3709351Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3709354Z 2025-09-07T07:09:04.3709457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3709665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3709734Z return mod(**inputs) 2025-09-07T07:09:04.3710004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3710112Z outputs = self.model( 2025-09-07T07:09:04.3710373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3710446Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3710695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3710774Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3710997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3711083Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3711363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3711488Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3711492Z 2025-09-07T07:09:04.3711604Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3711807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3711882Z return mod(**inputs) 2025-09-07T07:09:04.3712139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3712208Z outputs = self.model( 2025-09-07T07:09:04.3712472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3712544Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3712808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3712882Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3713135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3713219Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3713491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3713626Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3713853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3713935Z return self.act(input) 2025-09-07T07:09:04.3713939Z 2025-09-07T07:09:04.3714048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3714257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3714333Z return mod(**inputs) 2025-09-07T07:09:04.3714604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3714686Z outputs = self.model( 2025-09-07T07:09:04.3714957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3715052Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3715329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3715401Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3715631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3715709Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3715968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3716053Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3716058Z 2025-09-07T07:09:04.3716162Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3716388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3716455Z return mod(**inputs) 2025-09-07T07:09:04.3716717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3716788Z outputs = self.model( 2025-09-07T07:09:04.3717043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3717122Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3717379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3717459Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3717695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3717784Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3718040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3718133Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3718398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3718551Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3718555Z 2025-09-07T07:09:04.3718666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3718868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3718937Z return mod(**inputs) 2025-09-07T07:09:04.3719209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3719296Z outputs = self.model( 2025-09-07T07:09:04.3719686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3719772Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3720048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3720119Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3720337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3720426Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3720685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3720788Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3721046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3721131Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3721135Z 2025-09-07T07:09:04.3721249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3721513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3721585Z return mod(**inputs) 2025-09-07T07:09:04.3721843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3721911Z outputs = self.model( 2025-09-07T07:09:04.3722183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3722258Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3722532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3722607Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3722865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3722946Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3723198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3723296Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3723548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3723648Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3723651Z 2025-09-07T07:09:04.3723735Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3723837Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3723926Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3724006Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3724120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3724324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3724390Z return mod(**inputs) 2025-09-07T07:09:04.3724666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3724740Z outputs = self.model( 2025-09-07T07:09:04.3725021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3725098Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3725367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3725455Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3725721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3725815Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3726087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3726192Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3726467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3726572Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3726896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3727040Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3727044Z 2025-09-07T07:09:04.3727162Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3727384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3727455Z return mod(**inputs) 2025-09-07T07:09:04.3727737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3727834Z outputs = self.model( 2025-09-07T07:09:04.3728112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3728191Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3728467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3728546Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3728789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3728885Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3729175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3729280Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3729553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3729657Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3729982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3730099Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3730103Z 2025-09-07T07:09:04.3730220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3730453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3730533Z return mod(**inputs) 2025-09-07T07:09:04.3730809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3730882Z outputs = self.model( 2025-09-07T07:09:04.3731162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3731242Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3731520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3731598Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3731837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3731930Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3732204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3732324Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3732597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3732686Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3732698Z 2025-09-07T07:09:04.3732807Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3733020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3733098Z return mod(**inputs) 2025-09-07T07:09:04.3733372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3733453Z outputs = self.model( 2025-09-07T07:09:04.3733725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3733804Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3734087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3734164Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3734410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3734526Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3734798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3734937Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3734940Z 2025-09-07T07:09:04.3735048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3735275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3735347Z return mod(**inputs) 2025-09-07T07:09:04.3735651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3735728Z outputs = self.model( 2025-09-07T07:09:04.3736006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3736095Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3736367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3736450Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3736690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3736774Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3737079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3737211Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3737449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3737526Z return self.act(input) 2025-09-07T07:09:04.3737530Z 2025-09-07T07:09:04.3737639Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3737863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3737934Z return mod(**inputs) 2025-09-07T07:09:04.3738220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3738291Z outputs = self.model( 2025-09-07T07:09:04.3738578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3738657Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3738953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3739038Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3739279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3739373Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3739646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3739734Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3739739Z 2025-09-07T07:09:04.3739857Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3740072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3740151Z return mod(**inputs) 2025-09-07T07:09:04.3740430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3740511Z outputs = self.model( 2025-09-07T07:09:04.3740788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3740866Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3741163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3741238Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3741481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3741564Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3741835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-09-07T07:09:04.3741930Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3741936Z 2025-09-07T07:09:04.3742044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3742276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3742347Z return mod(**inputs) 2025-09-07T07:09:04.3742625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3742705Z outputs = self.model( 2025-09-07T07:09:04.3742979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3743061Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3743333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3743415Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3743671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3743760Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3744037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3744135Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3744417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3744579Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3744583Z 2025-09-07T07:09:04.3744692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3744910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3744979Z return mod(**inputs) 2025-09-07T07:09:04.3745259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3745351Z outputs = self.model( 2025-09-07T07:09:04.3745631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3745796Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3746081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3746169Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3746416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3746510Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3746789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3746896Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3747163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3747247Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3747251Z 2025-09-07T07:09:04.3747361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3747582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3747649Z return mod(**inputs) 2025-09-07T07:09:04.3747917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3747985Z outputs = self.model( 2025-09-07T07:09:04.3748249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3748324Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3748588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3748664Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3748911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3749003Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3749260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3749360Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3749614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3749701Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3749705Z 2025-09-07T07:09:04.3749796Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3749894Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3749986Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3750067Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3750173Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3750382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3750447Z return mod(**inputs) 2025-09-07T07:09:04.3750716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3750785Z outputs = self.model( 2025-09-07T07:09:04.3751045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3751124Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3751381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3751464Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3751722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3751810Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3752063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3752158Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3752420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3752518Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3752820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3752955Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3752960Z 2025-09-07T07:09:04.3753065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3753274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3753340Z return mod(**inputs) 2025-09-07T07:09:04.3753604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3753690Z outputs = self.model( 2025-09-07T07:09:04.3753962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3754036Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3754297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3754382Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3754622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3754714Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3754997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3755094Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3755374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3755479Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3755803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3755922Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3755926Z 2025-09-07T07:09:04.3756044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3756279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3756349Z return mod(**inputs) 2025-09-07T07:09:04.3756619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3756687Z outputs = self.model( 2025-09-07T07:09:04.3756955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3757030Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3757285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3757367Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3757591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3757679Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3757933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3758057Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3758336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3758427Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3758432Z 2025-09-07T07:09:04.3758547Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3758763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3758840Z return mod(**inputs) 2025-09-07T07:09:04.3759121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3759194Z outputs = self.model( 2025-09-07T07:09:04.3759479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3759558Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3759836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3759912Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3760150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3760262Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3760534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3760669Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3760673Z 2025-09-07T07:09:04.3760784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3761006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3761076Z return mod(**inputs) 2025-09-07T07:09:04.3761376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3761457Z outputs = self.model( 2025-09-07T07:09:04.3761733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3761819Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3762089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3762168Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3762414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3762498Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3762815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3762947Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3763177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3763258Z return self.act(input) 2025-09-07T07:09:04.3763262Z 2025-09-07T07:09:04.3763374Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3763592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3763663Z return mod(**inputs) 2025-09-07T07:09:04.3763944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3764016Z outputs = self.model( 2025-09-07T07:09:04.3764288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3764374Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3764665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3764748Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3764985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3765070Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3765349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3765435Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3765439Z 2025-09-07T07:09:04.3765554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3765765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3765846Z return mod(**inputs) 2025-09-07T07:09:04.3766119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3766193Z outputs = self.model( 2025-09-07T07:09:04.3766477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3766554Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3766851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3766928Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3767164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3767254Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3767526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3767634Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3767924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3768088Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3768099Z 2025-09-07T07:09:04.3768209Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3768435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3768517Z return mod(**inputs) 2025-09-07T07:09:04.3768776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3768852Z outputs = self.model( 2025-09-07T07:09:04.3769112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3769201Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3769468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3769541Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3769771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3769853Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3770108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3770206Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3770461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3770550Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3770553Z 2025-09-07T07:09:04.3770657Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3770863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3770951Z return mod(**inputs) 2025-09-07T07:09:04.3771211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3771289Z outputs = self.model( 2025-09-07T07:09:04.3771549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3771629Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3771904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3771980Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3772225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3772311Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3772592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3772690Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3772964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3773084Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3773088Z 2025-09-07T07:09:04.3773174Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3773268Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3773353Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3773442Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3773553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3773768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3773847Z return mod(**inputs) 2025-09-07T07:09:04.3774146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3774226Z outputs = self.model( 2025-09-07T07:09:04.3774500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3774580Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3774871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3774942Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3775173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3775252Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3775519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3775623Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3775887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3775997Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3776314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3776464Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3776468Z 2025-09-07T07:09:04.3776576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3776796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3776871Z return mod(**inputs) 2025-09-07T07:09:04.3777133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3777227Z outputs = self.model( 2025-09-07T07:09:04.3777504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3777582Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3777863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3777942Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3778188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3778272Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3778547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3778646Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3778904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3779013Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3779310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3779449Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3779453Z 2025-09-07T07:09:04.3779555Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3779763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3779842Z return mod(**inputs) 2025-09-07T07:09:04.3780117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3780197Z outputs = self.model( 2025-09-07T07:09:04.3780489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3780570Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3780878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3780959Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3781216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3781302Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3781592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-09-07T07:09:04.3781692Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:09:04.3781973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3782132Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3782139Z 2025-09-07T07:09:04.3782254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3782494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3782564Z return mod(**inputs) 2025-09-07T07:09:04.3782844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3782926Z outputs = self.model( 2025-09-07T07:09:04.3783210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3783298Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3783567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3783645Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3783893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3783998Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3784303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3784437Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3784442Z 2025-09-07T07:09:04.3784564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3784794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3784869Z return mod(**inputs) 2025-09-07T07:09:04.3785169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3785247Z outputs = self.model( 2025-09-07T07:09:04.3785552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3785637Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3786074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3786175Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3786421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3786542Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3786821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-09-07T07:09:04.3786960Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3787199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3787277Z return self.act(input) 2025-09-07T07:09:04.3787284Z 2025-09-07T07:09:04.3787410Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3787655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3787739Z return mod(**inputs) 2025-09-07T07:09:04.3788032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3788110Z outputs = self.model( 2025-09-07T07:09:04.3788418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3788498Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3788804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3788883Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3789146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3789245Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3789534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-09-07T07:09:04.3789632Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3789635Z 2025-09-07T07:09:04.3789750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3789976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3790048Z return mod(**inputs) 2025-09-07T07:09:04.3790351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3790433Z outputs = self.model( 2025-09-07T07:09:04.3790718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-09-07T07:09:04.3790806Z encoder_outputs = self.encoder( 2025-09-07T07:09:04.3791105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-09-07T07:09:04.3791185Z layer_outputs = encoder_layer( 2025-09-07T07:09:04.3791436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3791524Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3791808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-09-07T07:09:04.3791895Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3791899Z 2025-09-07T07:09:04.3792017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3792232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3792303Z return mod(**inputs) 2025-09-07T07:09:04.3792590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3792665Z outputs = self.model( 2025-09-07T07:09:04.3792951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3793031Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3793328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3793412Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3793655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3793748Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3794031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3794135Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3794406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3794557Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3794561Z 2025-09-07T07:09:04.3794670Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3794867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3794940Z return mod(**inputs) 2025-09-07T07:09:04.3795196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3795262Z outputs = self.model( 2025-09-07T07:09:04.3795519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3795606Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3795865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3795938Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3796155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3796244Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3796493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3796603Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3796854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3796942Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3796946Z 2025-09-07T07:09:04.3797051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3797273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3797349Z return mod(**inputs) 2025-09-07T07:09:04.3797598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3797673Z outputs = self.model( 2025-09-07T07:09:04.3797926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3797998Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3798255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3798324Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3798547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3798626Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3798878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3798987Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3799237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3799350Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3799353Z 2025-09-07T07:09:04.3799434Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3799521Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3799598Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3799675Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3799783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3799979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3800053Z return mod(**inputs) 2025-09-07T07:09:04.3800321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3800389Z outputs = self.model( 2025-09-07T07:09:04.3800646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3800718Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3800975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3801044Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3801262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3801347Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3801610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3801717Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3801966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3802070Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3802364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3802495Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3802498Z 2025-09-07T07:09:04.3802607Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3802800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3802872Z return mod(**inputs) 2025-09-07T07:09:04.3803130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3803218Z outputs = self.model( 2025-09-07T07:09:04.3803491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3803567Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3803834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3803908Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3804132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3804219Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3804475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3804585Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3804841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3804950Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3805252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3805381Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3805385Z 2025-09-07T07:09:04.3805506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3805702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3805774Z return mod(**inputs) 2025-09-07T07:09:04.3806028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3806094Z outputs = self.model( 2025-09-07T07:09:04.3806355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3806450Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3806708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3806777Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3807001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3807077Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3807322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3807425Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3807693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3807783Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3807788Z 2025-09-07T07:09:04.3807890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3808088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3808161Z return mod(**inputs) 2025-09-07T07:09:04.3808409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3808483Z outputs = self.model( 2025-09-07T07:09:04.3808733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3808810Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3809061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3809132Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3809360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3809459Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3809718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3809830Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3810082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3810238Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3810241Z 2025-09-07T07:09:04.3810343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3810549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3810617Z return mod(**inputs) 2025-09-07T07:09:04.3810868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3810947Z outputs = self.model( 2025-09-07T07:09:04.3811195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3811276Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3811541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3811620Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3811840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3811916Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3812177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3812285Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3812563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3812645Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3812648Z 2025-09-07T07:09:04.3812753Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3812963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3813030Z return mod(**inputs) 2025-09-07T07:09:04.3813293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3813361Z outputs = self.model( 2025-09-07T07:09:04.3813625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3813719Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3813976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3814062Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3814285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3814373Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3814632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3814741Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3815004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3815093Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3815096Z 2025-09-07T07:09:04.3815188Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3815268Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3815369Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3815457Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3815562Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3815772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3815840Z return mod(**inputs) 2025-09-07T07:09:04.3816101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3816177Z outputs = self.model( 2025-09-07T07:09:04.3816435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3816516Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3816784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3816865Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3817091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3817172Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3817441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3817570Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3817834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3817931Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3818230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3818374Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3818379Z 2025-09-07T07:09:04.3818486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3818712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3818784Z return mod(**inputs) 2025-09-07T07:09:04.3819062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3819133Z outputs = self.model( 2025-09-07T07:09:04.3819389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3819470Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3819923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3820010Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3820283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3820367Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3820638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3820748Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3821018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3821116Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3821419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3821528Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3821531Z 2025-09-07T07:09:04.3821637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3821850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3821961Z return mod(**inputs) 2025-09-07T07:09:04.3822229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3822300Z outputs = self.model( 2025-09-07T07:09:04.3822559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3822642Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3822897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3822976Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3823198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3823281Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3823552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3823670Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3823950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3824064Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3824068Z 2025-09-07T07:09:04.3824185Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3824398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3824468Z return mod(**inputs) 2025-09-07T07:09:04.3824750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3824821Z outputs = self.model( 2025-09-07T07:09:04.3825104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3825224Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3825498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3825582Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3826043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3826146Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3826431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3826579Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3826583Z 2025-09-07T07:09:04.3826694Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3826931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3827017Z return mod(**inputs) 2025-09-07T07:09:04.3827292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3827375Z outputs = self.model( 2025-09-07T07:09:04.3827650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3827730Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3828012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3828085Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3828323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3828406Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3828671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3828821Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3829038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3829119Z return self.act(input) 2025-09-07T07:09:04.3829123Z 2025-09-07T07:09:04.3829226Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3829432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3829506Z return mod(**inputs) 2025-09-07T07:09:04.3829763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3829838Z outputs = self.model( 2025-09-07T07:09:04.3830097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3830179Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3830446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3830518Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3830742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3830846Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3831103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.3831183Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3831187Z 2025-09-07T07:09:04.3831299Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3831499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3831565Z return mod(**inputs) 2025-09-07T07:09:04.3831842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3831911Z outputs = self.model( 2025-09-07T07:09:04.3832170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3832242Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3832491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3832568Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3832784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3832870Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3833136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3833239Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3833496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3833645Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3833650Z 2025-09-07T07:09:04.3833760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3833954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3834024Z return mod(**inputs) 2025-09-07T07:09:04.3834278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3834346Z outputs = self.model( 2025-09-07T07:09:04.3834611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3834701Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3834961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3835032Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3835250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3835337Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3835587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3835691Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3835939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3836027Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3836030Z 2025-09-07T07:09:04.3836133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3836329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3836402Z return mod(**inputs) 2025-09-07T07:09:04.3836650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3836740Z outputs = self.model( 2025-09-07T07:09:04.3837008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3837077Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3837326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3837394Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3837614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3837690Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3837953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3838055Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3838299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3838392Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3838395Z 2025-09-07T07:09:04.3838473Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3838556Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3838631Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3838705Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3838811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3839018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3839091Z return mod(**inputs) 2025-09-07T07:09:04.3839340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3839405Z outputs = self.model( 2025-09-07T07:09:04.3839664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3839736Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3839994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3840064Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3840282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3840370Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3840619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3840745Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3841000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3841105Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3841406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3841539Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3841543Z 2025-09-07T07:09:04.3841653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3841852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3841928Z return mod(**inputs) 2025-09-07T07:09:04.3842190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3842260Z outputs = self.model( 2025-09-07T07:09:04.3842535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3842623Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3842880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3842953Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3843184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3843272Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3843541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3843648Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3843930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3844035Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3844329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3844441Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3844444Z 2025-09-07T07:09:04.3844554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3844754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3844827Z return mod(**inputs) 2025-09-07T07:09:04.3845108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3845178Z outputs = self.model( 2025-09-07T07:09:04.3845436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3845508Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3845767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3845840Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3846064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3846143Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3846391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3846495Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3846757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3846861Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3846866Z 2025-09-07T07:09:04.3846964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3847154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3847229Z return mod(**inputs) 2025-09-07T07:09:04.3847482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3847558Z outputs = self.model( 2025-09-07T07:09:04.3847811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3847889Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3848143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3848215Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3848442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3848518Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3848777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-09-07T07:09:04.3848874Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3848877Z 2025-09-07T07:09:04.3848979Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3849180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3849245Z return mod(**inputs) 2025-09-07T07:09:04.3849502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3849570Z outputs = self.model( 2025-09-07T07:09:04.3849817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3849916Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3850166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3850245Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3850462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3850546Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3850794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3850899Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3851172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3851323Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3851327Z 2025-09-07T07:09:04.3851436Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3851630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3851698Z return mod(**inputs) 2025-09-07T07:09:04.3851955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3852022Z outputs = self.model( 2025-09-07T07:09:04.3852276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3852346Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3852608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3852676Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3852922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3853007Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3853258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3853373Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3853622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3853702Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3853706Z 2025-09-07T07:09:04.3853814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3854011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3854085Z return mod(**inputs) 2025-09-07T07:09:04.3854353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3854421Z outputs = self.model( 2025-09-07T07:09:04.3854686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3854779Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3855045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3855118Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3855351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3855429Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3855687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3855808Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3856088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3856185Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3856188Z 2025-09-07T07:09:04.3856270Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3856348Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3856436Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3856512Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3856622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3856821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3856885Z return mod(**inputs) 2025-09-07T07:09:04.3857164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3857236Z outputs = self.model( 2025-09-07T07:09:04.3857496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3857567Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3857822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3857896Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3858117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3858205Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3858456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3858569Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3858819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3858934Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3859230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3859360Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3859364Z 2025-09-07T07:09:04.3859470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3859667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3859739Z return mod(**inputs) 2025-09-07T07:09:04.3859991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3860058Z outputs = self.model( 2025-09-07T07:09:04.3860314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3860385Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3860643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3860715Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3860949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3861035Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3861284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3861398Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3861646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3861742Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3862066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3862174Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3862177Z 2025-09-07T07:09:04.3862288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3862482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3862558Z return mod(**inputs) 2025-09-07T07:09:04.3862810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3862877Z outputs = self.model( 2025-09-07T07:09:04.3863136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3863227Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3863492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3863564Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3863792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3863880Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3864138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3864254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3864511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3864602Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3864605Z 2025-09-07T07:09:04.3864709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3864931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3865006Z return mod(**inputs) 2025-09-07T07:09:04.3865265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3865343Z outputs = self.model( 2025-09-07T07:09:04.3865600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3865673Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3866032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3866109Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3866343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3866431Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3866727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3866867Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3866871Z 2025-09-07T07:09:04.3866985Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3867247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3867315Z return mod(**inputs) 2025-09-07T07:09:04.3867580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3867649Z outputs = self.model( 2025-09-07T07:09:04.3867904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3867987Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3868242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3868342Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3868566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3868647Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3868914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3869035Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3869262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3869334Z return self.act(input) 2025-09-07T07:09:04.3869337Z 2025-09-07T07:09:04.3869449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3869665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3869737Z return mod(**inputs) 2025-09-07T07:09:04.3870011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3870079Z outputs = self.model( 2025-09-07T07:09:04.3870347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3870421Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3870677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3870759Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3870982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3871073Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3871336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.3871440Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3871451Z 2025-09-07T07:09:04.3871557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3871759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3871836Z return mod(**inputs) 2025-09-07T07:09:04.3872097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3872173Z outputs = self.model( 2025-09-07T07:09:04.3872442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3872515Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3872779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3872852Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3873087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3873166Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3873442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3873551Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3873807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3873966Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3873970Z 2025-09-07T07:09:04.3874091Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3874297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3874365Z return mod(**inputs) 2025-09-07T07:09:04.3874639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3874717Z outputs = self.model( 2025-09-07T07:09:04.3874981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3875066Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3875324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3875397Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3875629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3875740Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3876005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3876108Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3876368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3876459Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3876462Z 2025-09-07T07:09:04.3876567Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3876778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3876844Z return mod(**inputs) 2025-09-07T07:09:04.3877111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3877179Z outputs = self.model( 2025-09-07T07:09:04.3877437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3877538Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3877805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3877884Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3878116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3878196Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3878477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3878574Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3878834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3878923Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3878928Z 2025-09-07T07:09:04.3879015Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3879095Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3879174Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3879256Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3879357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3879575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3879640Z return mod(**inputs) 2025-09-07T07:09:04.3879894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3879968Z outputs = self.model( 2025-09-07T07:09:04.3880222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3880302Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3880563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3880685Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3880922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3881003Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3881265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3881365Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3881618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3881726Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3882037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3882183Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3882187Z 2025-09-07T07:09:04.3882290Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3882495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3882563Z return mod(**inputs) 2025-09-07T07:09:04.3882820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3882895Z outputs = self.model( 2025-09-07T07:09:04.3883161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3883241Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3883496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3883583Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3883819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3883899Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3884165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3884265Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3884528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3884625Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3884924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3885042Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3885045Z 2025-09-07T07:09:04.3885150Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3885362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3885433Z return mod(**inputs) 2025-09-07T07:09:04.3885699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3885795Z outputs = self.model( 2025-09-07T07:09:04.3886057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3886137Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3886393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3886463Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3886699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3886780Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3887066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3887167Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3887443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3887525Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3887528Z 2025-09-07T07:09:04.3887628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3887834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3887909Z return mod(**inputs) 2025-09-07T07:09:04.3888185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3888254Z outputs = self.model( 2025-09-07T07:09:04.3888508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3888588Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3888839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3888918Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3889144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3889230Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3889483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3889594Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3889870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3890041Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3890044Z 2025-09-07T07:09:04.3890153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3890349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3890415Z return mod(**inputs) 2025-09-07T07:09:04.3890676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3890742Z outputs = self.model( 2025-09-07T07:09:04.3890998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3891070Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3891327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3891399Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3891619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3891706Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3891981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3892097Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3892361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3892440Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3892443Z 2025-09-07T07:09:04.3892553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3892750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3892826Z return mod(**inputs) 2025-09-07T07:09:04.3893100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3893169Z outputs = self.model( 2025-09-07T07:09:04.3893433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3893507Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3893770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3893842Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3894074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3894155Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3894430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3894552Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3894810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3894908Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3894911Z 2025-09-07T07:09:04.3894992Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3895073Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3895162Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3916125Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3916432Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3916666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3916742Z return mod(**inputs) 2025-09-07T07:09:04.3917071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3917272Z outputs = self.model( 2025-09-07T07:09:04.3917543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3917626Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3917890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3917977Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3918207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3918306Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3918563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3918691Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3918947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3919051Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3919358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3919532Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3919541Z 2025-09-07T07:09:04.3919812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3920025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3920095Z return mod(**inputs) 2025-09-07T07:09:04.3920363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3920437Z outputs = self.model( 2025-09-07T07:09:04.3920786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3920867Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3921146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3921226Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3921452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3921535Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3921796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3921909Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3922219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3922324Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3922628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3922751Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3922757Z 2025-09-07T07:09:04.3922866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3923085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3923155Z return mod(**inputs) 2025-09-07T07:09:04.3923428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3923499Z outputs = self.model( 2025-09-07T07:09:04.3923766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3923883Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3924144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3924228Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3924460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3924544Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3924813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3924923Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3925192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3925278Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3925283Z 2025-09-07T07:09:04.3925400Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3925614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3925682Z return mod(**inputs) 2025-09-07T07:09:04.3925953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3926051Z outputs = self.model( 2025-09-07T07:09:04.3926319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3926400Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3926652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3926734Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3926957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3927046Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3927315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-09-07T07:09:04.3927397Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3927411Z 2025-09-07T07:09:04.3927515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3927720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3927798Z return mod(**inputs) 2025-09-07T07:09:04.3928063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3928135Z outputs = self.model( 2025-09-07T07:09:04.3928415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3928489Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3928750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3928821Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3929041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3929128Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3929375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3929503Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3929507Z 2025-09-07T07:09:04.3929610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3929808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3929883Z return mod(**inputs) 2025-09-07T07:09:04.3930153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3930225Z outputs = self.model( 2025-09-07T07:09:04.3930476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3930549Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3930804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3930874Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3931149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3931230Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3931485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3931603Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3931813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3931887Z return self.act(input) 2025-09-07T07:09:04.3931891Z 2025-09-07T07:09:04.3931993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3932215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3932281Z return mod(**inputs) 2025-09-07T07:09:04.3932532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3932604Z outputs = self.model( 2025-09-07T07:09:04.3932852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3932932Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3933184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3933276Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3933492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3933573Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3933827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.3933908Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3933912Z 2025-09-07T07:09:04.3934017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3934213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3934279Z return mod(**inputs) 2025-09-07T07:09:04.3934552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3934621Z outputs = self.model( 2025-09-07T07:09:04.3934882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3934956Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3935209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3935288Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3935514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3935599Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3935843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3935949Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3936215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3936364Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3936368Z 2025-09-07T07:09:04.3936477Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3936669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3936739Z return mod(**inputs) 2025-09-07T07:09:04.3936981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3937045Z outputs = self.model( 2025-09-07T07:09:04.3937295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3937366Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3937615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3937688Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3937907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3938000Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3938245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3938349Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3938592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3938680Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3938683Z 2025-09-07T07:09:04.3938783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3938974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3939049Z return mod(**inputs) 2025-09-07T07:09:04.3939320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3939397Z outputs = self.model( 2025-09-07T07:09:04.3939655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3939728Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3939993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3940067Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3940297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3940394Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3940664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3940767Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3941046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3941152Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3941156Z 2025-09-07T07:09:04.3941247Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3941344Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3941427Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3941511Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3941633Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3941851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3941931Z return mod(**inputs) 2025-09-07T07:09:04.3942233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3942308Z outputs = self.model( 2025-09-07T07:09:04.3942595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3942677Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3942969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3943045Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3943284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3943379Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3943658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3943776Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3944054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3944168Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3944496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3944675Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3944679Z 2025-09-07T07:09:04.3944796Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3945010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3945087Z return mod(**inputs) 2025-09-07T07:09:04.3945373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3945451Z outputs = self.model( 2025-09-07T07:09:04.3945877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3945967Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3946262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3946344Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3946597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3946685Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3946966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3947082Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3947364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3947470Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3947754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3947863Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3947874Z 2025-09-07T07:09:04.3947974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3948163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3948235Z return mod(**inputs) 2025-09-07T07:09:04.3948481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3948556Z outputs = self.model( 2025-09-07T07:09:04.3948800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3948888Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3949139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3949207Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3949430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3949506Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3949746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3949847Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3950086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3950173Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3950178Z 2025-09-07T07:09:04.3950276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3950475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3950539Z return mod(**inputs) 2025-09-07T07:09:04.3950781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3950876Z outputs = self.model( 2025-09-07T07:09:04.3951122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3951201Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3951446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3951515Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3951738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3951833Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3952089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3952193Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3952437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3952595Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3952599Z 2025-09-07T07:09:04.3952700Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3952903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3952969Z return mod(**inputs) 2025-09-07T07:09:04.3953241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3953311Z outputs = self.model( 2025-09-07T07:09:04.3953568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3953646Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3953891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3953968Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3954181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3954260Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3954513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3954617Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3954882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3954960Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3954963Z 2025-09-07T07:09:04.3955067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3955255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3955319Z return mod(**inputs) 2025-09-07T07:09:04.3955568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3955633Z outputs = self.model( 2025-09-07T07:09:04.3955885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3955955Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3956197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3956277Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3956488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3956570Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3956827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3956929Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3957177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3957260Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3957264Z 2025-09-07T07:09:04.3957350Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3957426Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3957509Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3957583Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3957696Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3957898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3957963Z return mod(**inputs) 2025-09-07T07:09:04.3958216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3958280Z outputs = self.model( 2025-09-07T07:09:04.3958526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3958603Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3958863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3958940Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3959157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3959234Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3959485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3959589Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3959839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3959933Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3960222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3960351Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3960355Z 2025-09-07T07:09:04.3960470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3960667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3960731Z return mod(**inputs) 2025-09-07T07:09:04.3960983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3961051Z outputs = self.model( 2025-09-07T07:09:04.3961293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3961370Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3961612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3961688Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3961900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3961978Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3962226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3962329Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3962601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3962693Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3962979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3963081Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3963085Z 2025-09-07T07:09:04.3963180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3963376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3963441Z return mod(**inputs) 2025-09-07T07:09:04.3963707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3963773Z outputs = self.model( 2025-09-07T07:09:04.3964015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3964097Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3964340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3964415Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3964626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3964726Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3964973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3965078Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3965340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3965424Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3965427Z 2025-09-07T07:09:04.3965535Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3965733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3965797Z return mod(**inputs) 2025-09-07T07:09:04.3966063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3966130Z outputs = self.model( 2025-09-07T07:09:04.3966389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3966491Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3966741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3966811Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3967027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3967111Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3967355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3967478Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3967482Z 2025-09-07T07:09:04.3967579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3967773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3967846Z return mod(**inputs) 2025-09-07T07:09:04.3968094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3968165Z outputs = self.model( 2025-09-07T07:09:04.3968411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3968498Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3968753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3968822Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3969046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3969122Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3969374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.3969503Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.3969710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.3969786Z return self.act(input) 2025-09-07T07:09:04.3969791Z 2025-09-07T07:09:04.3969889Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3970087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3970149Z return mod(**inputs) 2025-09-07T07:09:04.3970392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3970464Z outputs = self.model( 2025-09-07T07:09:04.3970719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3970800Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3971050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3971127Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3971338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3971415Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3971666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.3971745Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.3971749Z 2025-09-07T07:09:04.3971855Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3972046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3972109Z return mod(**inputs) 2025-09-07T07:09:04.3972389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3972454Z outputs = self.model( 2025-09-07T07:09:04.3972705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3972776Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3973019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3973096Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3973309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3973394Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3973639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:09:04.3973725Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.3973729Z 2025-09-07T07:09:04.3973830Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3974023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3974094Z return mod(**inputs) 2025-09-07T07:09:04.3974365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3974436Z outputs = self.model( 2025-09-07T07:09:04.3974687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3974758Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3975021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3975095Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3975337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3975432Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3975684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3975790Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3976046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3976200Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3976204Z 2025-09-07T07:09:04.3976302Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3976501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3976580Z return mod(**inputs) 2025-09-07T07:09:04.3976829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3976903Z outputs = self.model( 2025-09-07T07:09:04.3977148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3977225Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3977469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3977536Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3977756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3977833Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3978089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3978201Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3978451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3978529Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3978532Z 2025-09-07T07:09:04.3978632Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3978829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3978892Z return mod(**inputs) 2025-09-07T07:09:04.3979143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3979209Z outputs = self.model( 2025-09-07T07:09:04.3979452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3979532Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3979786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3979864Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3980091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3980206Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3980448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3980541Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3980788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3980872Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3980876Z 2025-09-07T07:09:04.3980961Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3981044Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3981119Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3981219Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3981321Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3981523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3981590Z return mod(**inputs) 2025-09-07T07:09:04.3981838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3981914Z outputs = self.model( 2025-09-07T07:09:04.3982166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3982252Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3982522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3982597Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3982826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3982906Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3983166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3983275Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3983522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3983625Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3983918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.3984061Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.3984088Z 2025-09-07T07:09:04.3984191Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3984401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3984468Z return mod(**inputs) 2025-09-07T07:09:04.3984726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3984805Z outputs = self.model( 2025-09-07T07:09:04.3985062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3985142Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3985399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3985472Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3985819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3985918Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3986200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3986305Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3986609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3986712Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3987029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.3987148Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.3987151Z 2025-09-07T07:09:04.3987254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3987458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3987541Z return mod(**inputs) 2025-09-07T07:09:04.3987795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3987870Z outputs = self.model( 2025-09-07T07:09:04.3988119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3988200Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3988448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3988525Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3988741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3988871Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3989133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.3989228Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.3989482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.3989566Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.3989570Z 2025-09-07T07:09:04.3989671Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3989878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3989943Z return mod(**inputs) 2025-09-07T07:09:04.3990201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3990269Z outputs = self.model( 2025-09-07T07:09:04.3990527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3990619Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3990872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3990952Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3991173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3991256Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3991507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3991615Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3991873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.3992027Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.3992031Z 2025-09-07T07:09:04.3992142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3992340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3992430Z return mod(**inputs) 2025-09-07T07:09:04.3992693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3992760Z outputs = self.model( 2025-09-07T07:09:04.3993032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3993108Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3993379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3993451Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3993698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3993787Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3994049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3994167Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3994423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.3994504Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.3994518Z 2025-09-07T07:09:04.3994621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3994822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3994912Z return mod(**inputs) 2025-09-07T07:09:04.3995171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3995250Z outputs = self.model( 2025-09-07T07:09:04.3995505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3995580Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3995850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3995921Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3996145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3996223Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3996473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3996587Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3996852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.3996943Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.3996947Z 2025-09-07T07:09:04.3997027Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3997108Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3997189Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3997263Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.3997369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.3997566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.3997630Z return mod(**inputs) 2025-09-07T07:09:04.3997887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.3997957Z outputs = self.model( 2025-09-07T07:09:04.3998216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.3998288Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.3998543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.3998631Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.3998850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.3998936Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.3999184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.3999296Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.3999548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.3999662Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.3999959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4000091Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4000094Z 2025-09-07T07:09:04.4000203Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4000399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4000470Z return mod(**inputs) 2025-09-07T07:09:04.4000720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4000786Z outputs = self.model( 2025-09-07T07:09:04.4001063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4001139Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4001403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4001472Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4001690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4001777Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4002028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4002142Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4002393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4002496Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4002811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4002918Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4002922Z 2025-09-07T07:09:04.4003033Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4003237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4003311Z return mod(**inputs) 2025-09-07T07:09:04.4003567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4003636Z outputs = self.model( 2025-09-07T07:09:04.4003906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4003984Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4004265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4004343Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4004584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4004689Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4004943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4005060Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4005313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4005402Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4005406Z 2025-09-07T07:09:04.4005511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4005721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4005811Z return mod(**inputs) 2025-09-07T07:09:04.4006062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4006134Z outputs = self.model( 2025-09-07T07:09:04.4006385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4006456Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4006712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4006782Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4007014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4007118Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4007387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4007507Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4007511Z 2025-09-07T07:09:04.4007614Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4007826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4007893Z return mod(**inputs) 2025-09-07T07:09:04.4008156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4008224Z outputs = self.model( 2025-09-07T07:09:04.4008482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4008565Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4008822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4008924Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4009147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4009226Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4009493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4009609Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4009834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4009904Z return self.act(input) 2025-09-07T07:09:04.4009908Z 2025-09-07T07:09:04.4010018Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4010222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4010290Z return mod(**inputs) 2025-09-07T07:09:04.4010557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4010625Z outputs = self.model( 2025-09-07T07:09:04.4010887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4010978Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4011236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4011314Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4011535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4011621Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4011879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4011984Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4011989Z 2025-09-07T07:09:04.4012093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4012301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4012378Z return mod(**inputs) 2025-09-07T07:09:04.4012639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4012715Z outputs = self.model( 2025-09-07T07:09:04.4012974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4013046Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4013663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4013742Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4013981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4014063Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4014331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4014443Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4014704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4014866Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4014869Z 2025-09-07T07:09:04.4014972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4015183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4015270Z return mod(**inputs) 2025-09-07T07:09:04.4015532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4015608Z outputs = self.model( 2025-09-07T07:09:04.4015865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4015948Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4016207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4016280Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4016513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4016592Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4016856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4016959Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4017220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4017301Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4017322Z 2025-09-07T07:09:04.4017426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4017634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4017699Z return mod(**inputs) 2025-09-07T07:09:04.4017963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4018030Z outputs = self.model( 2025-09-07T07:09:04.4018295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4018377Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4018636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4018711Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4018921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4018997Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4019243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4019335Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4019722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4019868Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4019878Z 2025-09-07T07:09:04.4019969Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4020053Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4020135Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4020222Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4020326Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4020536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4020603Z return mod(**inputs) 2025-09-07T07:09:04.4020858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4020937Z outputs = self.model( 2025-09-07T07:09:04.4021191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4021276Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4021531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4021629Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4021853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4021933Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4022193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4022292Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4022541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4022644Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4022935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4023080Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4023084Z 2025-09-07T07:09:04.4023188Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4023397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4023490Z return mod(**inputs) 2025-09-07T07:09:04.4023756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4023837Z outputs = self.model( 2025-09-07T07:09:04.4024109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4024193Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4024461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4024537Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4024807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4024893Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4025170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4025276Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4025554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4025656Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4026057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4026212Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4026217Z 2025-09-07T07:09:04.4026330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4026553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4026624Z return mod(**inputs) 2025-09-07T07:09:04.4026894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4026971Z outputs = self.model( 2025-09-07T07:09:04.4027218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4027296Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4027544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4027622Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4027840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4027932Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4028182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4028276Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4028526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4028606Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4028610Z 2025-09-07T07:09:04.4028707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4028903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4028966Z return mod(**inputs) 2025-09-07T07:09:04.4029271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4029336Z outputs = self.model( 2025-09-07T07:09:04.4029586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4029658Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4029901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4029999Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4030212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4030296Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4030538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-09-07T07:09:04.4030615Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.4030619Z 2025-09-07T07:09:04.4030726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4030918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4031009Z return mod(**inputs) 2025-09-07T07:09:04.4031256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4031323Z outputs = self.model( 2025-09-07T07:09:04.4031573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4031642Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4031894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4031962Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4032200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4032277Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4032521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4032632Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4032881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4033039Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4033042Z 2025-09-07T07:09:04.4033142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4033337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4033409Z return mod(**inputs) 2025-09-07T07:09:04.4033662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4033756Z outputs = self.model( 2025-09-07T07:09:04.4034022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4034095Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4034361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4034437Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4034671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4034750Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4035007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4035124Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4035389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4035478Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4035483Z 2025-09-07T07:09:04.4035582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4035778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4035871Z return mod(**inputs) 2025-09-07T07:09:04.4036127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4036201Z outputs = self.model( 2025-09-07T07:09:04.4036456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4036534Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4036790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4036863Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4037108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4037188Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4037446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4037553Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4037802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4037896Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4037899Z 2025-09-07T07:09:04.4037976Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4038061Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4038154Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4038229Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4038339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4038542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4038615Z return mod(**inputs) 2025-09-07T07:09:04.4038866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4038935Z outputs = self.model( 2025-09-07T07:09:04.4039192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4039264Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4039520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4039591Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4039818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4039915Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4040162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4040274Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4040524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4040626Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4040922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4041052Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4041062Z 2025-09-07T07:09:04.4041163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4041358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4041434Z return mod(**inputs) 2025-09-07T07:09:04.4041686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4041759Z outputs = self.model( 2025-09-07T07:09:04.4042023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4042095Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4042351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4042420Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4042645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4042723Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4042990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4043102Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4043349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4043452Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4043742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4043856Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4043859Z 2025-09-07T07:09:04.4043959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4044184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4044260Z return mod(**inputs) 2025-09-07T07:09:04.4044522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4044599Z outputs = self.model( 2025-09-07T07:09:04.4044867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4044942Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4045202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4045274Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4045500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4045579Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4045829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4045959Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4046210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4046301Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4046305Z 2025-09-07T07:09:04.4046406Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4046611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4046676Z return mod(**inputs) 2025-09-07T07:09:04.4046925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4046998Z outputs = self.model( 2025-09-07T07:09:04.4047249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4047327Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4047579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4047649Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4047876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4047974Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4048231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4048351Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4048355Z 2025-09-07T07:09:04.4048463Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4048669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4048733Z return mod(**inputs) 2025-09-07T07:09:04.4048977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4049058Z outputs = self.model( 2025-09-07T07:09:04.4049302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4049372Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4049610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4049684Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4049890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4049971Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4050223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4050333Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4050541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4050609Z return self.act(input) 2025-09-07T07:09:04.4050612Z 2025-09-07T07:09:04.4050715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4050902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4050970Z return mod(**inputs) 2025-09-07T07:09:04.4051206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4051268Z outputs = self.model( 2025-09-07T07:09:04.4051514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4051582Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4051826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4051914Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4052121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4052203Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4052442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4052526Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4052529Z 2025-09-07T07:09:04.4052624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4052816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4052877Z return mod(**inputs) 2025-09-07T07:09:04.4053113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4053188Z outputs = self.model( 2025-09-07T07:09:04.4053430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4053508Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4053779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4053848Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4054072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4054149Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4054404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4054505Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4054758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4054933Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4054938Z 2025-09-07T07:09:04.4055051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4055251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4055314Z return mod(**inputs) 2025-09-07T07:09:04.4055562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4055626Z outputs = self.model( 2025-09-07T07:09:04.4055867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4055960Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4056207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4056289Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4056500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4056576Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4056829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4056921Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4057169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4057247Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4057250Z 2025-09-07T07:09:04.4057354Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4057550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4057629Z return mod(**inputs) 2025-09-07T07:09:04.4057876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4057941Z outputs = self.model( 2025-09-07T07:09:04.4058195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4058263Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4058509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4058587Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4058801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4058885Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4059131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4059230Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4059481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4059583Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4059586Z 2025-09-07T07:09:04.4059672Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4059749Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4059831Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4059904Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4060001Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4060206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4060269Z return mod(**inputs) 2025-09-07T07:09:04.4060524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4060606Z outputs = self.model( 2025-09-07T07:09:04.4060851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4060928Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4061172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4061247Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4061460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4061537Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4061803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4061900Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4062150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4062245Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4062537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4062670Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4062674Z 2025-09-07T07:09:04.4062775Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4062981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4063048Z return mod(**inputs) 2025-09-07T07:09:04.4063310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4063395Z outputs = self.model( 2025-09-07T07:09:04.4063648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4063727Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4063977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4064058Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4064276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4064352Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4064608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4064702Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4064963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4065060Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4065358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4065485Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4065488Z 2025-09-07T07:09:04.4065588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4065901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4065972Z return mod(**inputs) 2025-09-07T07:09:04.4066237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4066304Z outputs = self.model( 2025-09-07T07:09:04.4066576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4066662Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4066951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4067036Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4067273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4067367Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4067652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4067747Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4068010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4068112Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4068117Z 2025-09-07T07:09:04.4068224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4068419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4068485Z return mod(**inputs) 2025-09-07T07:09:04.4068743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4068811Z outputs = self.model( 2025-09-07T07:09:04.4069070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4069144Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4069390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4069470Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4069691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4069811Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4070061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4070174Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4070422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4070573Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4070577Z 2025-09-07T07:09:04.4070683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4070879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4070952Z return mod(**inputs) 2025-09-07T07:09:04.4071201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4071270Z outputs = self.model( 2025-09-07T07:09:04.4071525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4071595Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4071873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4071942Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4072164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4072243Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4072490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4072606Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4072875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4072963Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4072967Z 2025-09-07T07:09:04.4073067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4073271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4073346Z return mod(**inputs) 2025-09-07T07:09:04.4073603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4073677Z outputs = self.model( 2025-09-07T07:09:04.4073932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4074011Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4074286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4074362Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4074591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4074674Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4074940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4075055Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4075325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4075425Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4075429Z 2025-09-07T07:09:04.4075515Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4075616Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4075721Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4075797Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4075912Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4076114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4076189Z return mod(**inputs) 2025-09-07T07:09:04.4076457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4076531Z outputs = self.model( 2025-09-07T07:09:04.4076815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4076902Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4077164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4077238Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4077471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4077550Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4077806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4077946Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4078200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4078306Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4078607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4078745Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4078757Z 2025-09-07T07:09:04.4078861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4079083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4079160Z return mod(**inputs) 2025-09-07T07:09:04.4079421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4079498Z outputs = self.model( 2025-09-07T07:09:04.4079756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4079829Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4080093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4080165Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4080415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4080500Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4080760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4080878Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4081137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4081241Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4081540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4081656Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4081660Z 2025-09-07T07:09:04.4081762Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4081964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4082056Z return mod(**inputs) 2025-09-07T07:09:04.4082318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4082395Z outputs = self.model( 2025-09-07T07:09:04.4082659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4082734Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4082998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4083069Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4083300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4083380Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4083640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4083759Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4084015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4084123Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4084127Z 2025-09-07T07:09:04.4084228Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4084436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4084502Z return mod(**inputs) 2025-09-07T07:09:04.4084761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4084836Z outputs = self.model( 2025-09-07T07:09:04.4085095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4085176Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4085487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4085563Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4085798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4085877Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4086138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-09-07T07:09:04.4086221Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.4086224Z 2025-09-07T07:09:04.4086335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4086549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4086618Z return mod(**inputs) 2025-09-07T07:09:04.4086883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4086949Z outputs = self.model( 2025-09-07T07:09:04.4087211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4087285Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4087540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4087621Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4087842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4087929Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4088186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4088325Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4088337Z 2025-09-07T07:09:04.4088441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4088639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4088715Z return mod(**inputs) 2025-09-07T07:09:04.4088969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4089045Z outputs = self.model( 2025-09-07T07:09:04.4089301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4089374Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4089641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4089715Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4089948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4090026Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4090281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4090425Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4090642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4090719Z return self.act(input) 2025-09-07T07:09:04.4090722Z 2025-09-07T07:09:04.4090823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4091035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4091107Z return mod(**inputs) 2025-09-07T07:09:04.4091389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4091492Z outputs = self.model( 2025-09-07T07:09:04.4091776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4091866Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4092165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4092244Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4092500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4092586Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4092907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4093001Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4093005Z 2025-09-07T07:09:04.4093119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4093347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4093419Z return mod(**inputs) 2025-09-07T07:09:04.4093711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4093789Z outputs = self.model( 2025-09-07T07:09:04.4094052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4094124Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4094379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4094462Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4094705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4094794Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4095059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4095161Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4095488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4095654Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4095658Z 2025-09-07T07:09:04.4095775Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4095990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4096070Z return mod(**inputs) 2025-09-07T07:09:04.4096347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4096423Z outputs = self.model( 2025-09-07T07:09:04.4096710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4096804Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4097090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4097168Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4097403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4097504Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4097765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4097871Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4098147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4098231Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4098242Z 2025-09-07T07:09:04.4098343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4098548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4098620Z return mod(**inputs) 2025-09-07T07:09:04.4098877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4098951Z outputs = self.model( 2025-09-07T07:09:04.4099206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4099300Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4099566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4099639Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4099875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4099954Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4100200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4100303Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4100551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4100642Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4100646Z 2025-09-07T07:09:04.4100727Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4100812Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4100905Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4100980Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4101086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4101279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4101352Z return mod(**inputs) 2025-09-07T07:09:04.4101601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4101666Z outputs = self.model( 2025-09-07T07:09:04.4101921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4101994Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4102253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4102326Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4102550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4102639Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4102894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4103021Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4103277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4103374Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4103681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4103817Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4103822Z 2025-09-07T07:09:04.4103933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4104156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4104236Z return mod(**inputs) 2025-09-07T07:09:04.4104511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4104585Z outputs = self.model( 2025-09-07T07:09:04.4104866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4104944Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4105223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4105300Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4105552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4105647Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4106015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4106135Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4106413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4106526Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4106845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4106963Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4106967Z 2025-09-07T07:09:04.4107091Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4107308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4107410Z return mod(**inputs) 2025-09-07T07:09:04.4107669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4107736Z outputs = self.model( 2025-09-07T07:09:04.4108003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4108076Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4108338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4108410Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4108631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4108722Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4108974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4109081Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4109331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4109442Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4109445Z 2025-09-07T07:09:04.4109546Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4109747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4109822Z return mod(**inputs) 2025-09-07T07:09:04.4110073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4110149Z outputs = self.model( 2025-09-07T07:09:04.4110405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4110498Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4110758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4110829Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4111056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4111134Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4111390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4111495Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4111763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4111924Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4111929Z 2025-09-07T07:09:04.4112032Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4112238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4112303Z return mod(**inputs) 2025-09-07T07:09:04.4112557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4112631Z outputs = self.model( 2025-09-07T07:09:04.4112881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4112960Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4113209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4113286Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4113518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4113597Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4113852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4113958Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4114213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4114292Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4114296Z 2025-09-07T07:09:04.4114396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4114595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4114661Z return mod(**inputs) 2025-09-07T07:09:04.4114918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4114986Z outputs = self.model( 2025-09-07T07:09:04.4115236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4115314Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4115576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4115653Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4115868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4115954Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4116201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4116316Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4116582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4116664Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4116667Z 2025-09-07T07:09:04.4116751Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4116828Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4116900Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4116981Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4117077Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4117272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4117335Z return mod(**inputs) 2025-09-07T07:09:04.4117596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4117671Z outputs = self.model( 2025-09-07T07:09:04.4117917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4117997Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4118239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4118310Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4118529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4118606Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4118855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4118958Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4119211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4119324Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4119822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4119966Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4119972Z 2025-09-07T07:09:04.4120070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4120270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4120336Z return mod(**inputs) 2025-09-07T07:09:04.4120590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4120664Z outputs = self.model( 2025-09-07T07:09:04.4120918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4120999Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4121253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4121332Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4121592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4121669Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4121918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4122019Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4122266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4122359Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4122675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4122787Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4122790Z 2025-09-07T07:09:04.4122886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4123091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4123155Z return mod(**inputs) 2025-09-07T07:09:04.4123409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4123474Z outputs = self.model( 2025-09-07T07:09:04.4123717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4123819Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4124065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4124144Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4124357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4124432Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4124686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4124789Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4125044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4125123Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4125126Z 2025-09-07T07:09:04.4125232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4125423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4125512Z return mod(**inputs) 2025-09-07T07:09:04.4125771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4125837Z outputs = self.model( 2025-09-07T07:09:04.4126088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4126158Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4126399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4126476Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4126688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4126775Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4127021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4127137Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4127147Z 2025-09-07T07:09:04.4127243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4127454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4127524Z return mod(**inputs) 2025-09-07T07:09:04.4127767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4127839Z outputs = self.model( 2025-09-07T07:09:04.4128082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4128149Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4128403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4128472Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4128704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4128783Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4129023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4129145Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4129348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4129420Z return self.act(input) 2025-09-07T07:09:04.4129424Z 2025-09-07T07:09:04.4129523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4129736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4129803Z return mod(**inputs) 2025-09-07T07:09:04.4130052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4130124Z outputs = self.model( 2025-09-07T07:09:04.4130366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4130443Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4130686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4130754Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4130975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4131051Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4131311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4131411Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4131414Z 2025-09-07T07:09:04.4131512Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4131711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4131775Z return mod(**inputs) 2025-09-07T07:09:04.4132032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4132096Z outputs = self.model( 2025-09-07T07:09:04.4132350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4132419Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4132669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4132752Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4132967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4133050Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4133294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:09:04.4133403Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.4133406Z 2025-09-07T07:09:04.4133513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4133703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4133772Z return mod(**inputs) 2025-09-07T07:09:04.4134019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4134087Z outputs = self.model( 2025-09-07T07:09:04.4134361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4134432Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4134692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4134764Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4134989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4135067Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4135320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4135427Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4135701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4135866Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4135872Z 2025-09-07T07:09:04.4135974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4136176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4136251Z return mod(**inputs) 2025-09-07T07:09:04.4136518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4136594Z outputs = self.model( 2025-09-07T07:09:04.4136853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4136934Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4137204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4137291Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4137512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4137586Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4137837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4137935Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4138179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4138270Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4138273Z 2025-09-07T07:09:04.4138378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4138586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4138651Z return mod(**inputs) 2025-09-07T07:09:04.4138918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4138985Z outputs = self.model( 2025-09-07T07:09:04.4139244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4139344Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4139609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4139692Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4139921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4140000Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4140268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4140372Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4140667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4140756Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4140761Z 2025-09-07T07:09:04.4140843Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4140930Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4141007Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4141091Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4141196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4141397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4141468Z return mod(**inputs) 2025-09-07T07:09:04.4141744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4141822Z outputs = self.model( 2025-09-07T07:09:04.4142083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4142162Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4142417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4142490Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4142719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4142798Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4143061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4143161Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4143418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4143548Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4143864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4144012Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4144016Z 2025-09-07T07:09:04.4144126Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4144347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4144413Z return mod(**inputs) 2025-09-07T07:09:04.4144670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4144747Z outputs = self.model( 2025-09-07T07:09:04.4145017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4145105Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4145377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4145461Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4145808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4145899Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4146165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4146264Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4146537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4146644Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4146971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4147094Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4147097Z 2025-09-07T07:09:04.4147204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4147414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4147481Z return mod(**inputs) 2025-09-07T07:09:04.4147738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4147818Z outputs = self.model( 2025-09-07T07:09:04.4148076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4148175Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4148438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4148512Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4148746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4148829Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4149091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4149190Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4149453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4149538Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4149541Z 2025-09-07T07:09:04.4149645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4149876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4149945Z return mod(**inputs) 2025-09-07T07:09:04.4150217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4150287Z outputs = self.model( 2025-09-07T07:09:04.4150550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4150631Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4150899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4150976Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4151195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4151273Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4151533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4151643Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4151904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4152075Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4152079Z 2025-09-07T07:09:04.4152187Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4152384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4152452Z return mod(**inputs) 2025-09-07T07:09:04.4152723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4152793Z outputs = self.model( 2025-09-07T07:09:04.4153084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4153159Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4153409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4153489Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4153707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4153794Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4154045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4154157Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4154430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4154515Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4154518Z 2025-09-07T07:09:04.4154631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4154830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4154912Z return mod(**inputs) 2025-09-07T07:09:04.4155160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4155227Z outputs = self.model( 2025-09-07T07:09:04.4155488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4155559Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4155815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4155885Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4156135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4156213Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4156461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4156574Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4156822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4156914Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4156917Z 2025-09-07T07:09:04.4156998Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4157076Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4157160Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4157237Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4157346Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4157543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4157607Z return mod(**inputs) 2025-09-07T07:09:04.4157867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4157952Z outputs = self.model( 2025-09-07T07:09:04.4158211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4158282Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4158535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4158612Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4158831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4158918Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4159186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4159300Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4159551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4159645Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4159940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4160073Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4160077Z 2025-09-07T07:09:04.4160203Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4160399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4160465Z return mod(**inputs) 2025-09-07T07:09:04.4160723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4160790Z outputs = self.model( 2025-09-07T07:09:04.4161053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4161126Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4161378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4161450Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4161667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4161754Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4162030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4162141Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4162384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4162479Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4162766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4162870Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4162873Z 2025-09-07T07:09:04.4162976Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4163168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4163241Z return mod(**inputs) 2025-09-07T07:09:04.4163488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4163555Z outputs = self.model( 2025-09-07T07:09:04.4163809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4163896Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4164152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4164221Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4164438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4164524Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4164780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4164894Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4165164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4165249Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4165260Z 2025-09-07T07:09:04.4165361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4165556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4165628Z return mod(**inputs) 2025-09-07T07:09:04.4165881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4165953Z outputs = self.model( 2025-09-07T07:09:04.4166202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4166288Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4166545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4166615Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4166838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4166917Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4167162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4167287Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4167290Z 2025-09-07T07:09:04.4167391Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4167594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4167662Z return mod(**inputs) 2025-09-07T07:09:04.4167917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4168024Z outputs = self.model( 2025-09-07T07:09:04.4168266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4168342Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4168588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4168664Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4168874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4168949Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4169199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4169312Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4169526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4169595Z return self.act(input) 2025-09-07T07:09:04.4169598Z 2025-09-07T07:09:04.4169695Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4169912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4169977Z return mod(**inputs) 2025-09-07T07:09:04.4170232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4170299Z outputs = self.model( 2025-09-07T07:09:04.4170552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4170624Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4170874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4170973Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4171183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4171267Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4171508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4171587Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4171590Z 2025-09-07T07:09:04.4171698Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4171888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4171959Z return mod(**inputs) 2025-09-07T07:09:04.4172217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4172285Z outputs = self.model( 2025-09-07T07:09:04.4172537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4172604Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4172861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4172932Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4173157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4173235Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4173485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4173593Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4173844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4174023Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4174027Z 2025-09-07T07:09:04.4174131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4174333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4174405Z return mod(**inputs) 2025-09-07T07:09:04.4174675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4174748Z outputs = self.model( 2025-09-07T07:09:04.4175006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4175083Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4175334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4175405Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4175630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4175708Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4175991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4176086Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4176328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4176414Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4176417Z 2025-09-07T07:09:04.4176514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4176713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4176777Z return mod(**inputs) 2025-09-07T07:09:04.4177043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4177110Z outputs = self.model( 2025-09-07T07:09:04.4177354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4177431Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4177681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4177756Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4177969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4178045Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4178333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4178436Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4178688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4178773Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4178776Z 2025-09-07T07:09:04.4178856Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4178943Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4179020Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4179103Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4179204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4179402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4179474Z return mod(**inputs) 2025-09-07T07:09:04.4179723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4179818Z outputs = self.model( 2025-09-07T07:09:04.4180082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4180167Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4180423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4180493Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4180728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4180808Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4181077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4181177Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4181438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4181547Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4181851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4182048Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4182052Z 2025-09-07T07:09:04.4182156Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4182366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4182432Z return mod(**inputs) 2025-09-07T07:09:04.4182699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4182779Z outputs = self.model( 2025-09-07T07:09:04.4183072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4183156Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4183429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4183506Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4183748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4183833Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4184114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4184216Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4184506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4184619Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4184934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4185056Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4185061Z 2025-09-07T07:09:04.4185169Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4185390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4185460Z return mod(**inputs) 2025-09-07T07:09:04.4185828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4185919Z outputs = self.model( 2025-09-07T07:09:04.4186195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4186306Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4186579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4186656Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4186899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4186986Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4187259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4187355Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4187613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4187695Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4187699Z 2025-09-07T07:09:04.4187801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4188004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4188071Z return mod(**inputs) 2025-09-07T07:09:04.4188329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4188416Z outputs = self.model( 2025-09-07T07:09:04.4188668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4188745Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4188995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4189071Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4189292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4189371Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4189645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-09-07T07:09:04.4189728Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.4189733Z 2025-09-07T07:09:04.4189839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4190034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4190106Z return mod(**inputs) 2025-09-07T07:09:04.4190353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4190419Z outputs = self.model( 2025-09-07T07:09:04.4190703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4190778Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4191037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4191108Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4191331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4191419Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4191676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4191806Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4192057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4192217Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4192220Z 2025-09-07T07:09:04.4192341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4192540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4192613Z return mod(**inputs) 2025-09-07T07:09:04.4192865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4192940Z outputs = self.model( 2025-09-07T07:09:04.4193193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4193264Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4193524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4193597Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4193825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4193904Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4194154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4194269Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4194549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4194638Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4194641Z 2025-09-07T07:09:04.4194744Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4194964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4195028Z return mod(**inputs) 2025-09-07T07:09:04.4195280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4195354Z outputs = self.model( 2025-09-07T07:09:04.4195626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4195706Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4195956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4196027Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4196251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4196329Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4196587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4196692Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4196966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4197055Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4197058Z 2025-09-07T07:09:04.4197138Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4197222Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4197302Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4197383Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4197485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4197679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4197754Z return mod(**inputs) 2025-09-07T07:09:04.4198005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4198077Z outputs = self.model( 2025-09-07T07:09:04.4198332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4198418Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4198680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4198750Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4198975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4199054Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4199303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4199417Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4199663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4199765Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4200059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4200197Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4200201Z 2025-09-07T07:09:04.4200319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4200519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4200592Z return mod(**inputs) 2025-09-07T07:09:04.4200848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4200922Z outputs = self.model( 2025-09-07T07:09:04.4201177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4201251Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4201530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4201603Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4201835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4201915Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4202181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4202289Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4202546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4202652Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4202968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4203090Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4203095Z 2025-09-07T07:09:04.4203199Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4203405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4203485Z return mod(**inputs) 2025-09-07T07:09:04.4203759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4203840Z outputs = self.model( 2025-09-07T07:09:04.4204113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4204196Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4204469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4204566Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4204820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4204906Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4205191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4205313Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4205576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4205668Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4205671Z 2025-09-07T07:09:04.4205774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4205994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4206060Z return mod(**inputs) 2025-09-07T07:09:04.4206337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4206408Z outputs = self.model( 2025-09-07T07:09:04.4206672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4206774Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4207036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4207117Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4207347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4207425Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4207697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4207820Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4207824Z 2025-09-07T07:09:04.4207951Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4208154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4208222Z return mod(**inputs) 2025-09-07T07:09:04.4208495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4208564Z outputs = self.model( 2025-09-07T07:09:04.4208830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4208904Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4209184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4209260Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4209490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4209578Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4209842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4209970Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4210188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4210257Z return self.act(input) 2025-09-07T07:09:04.4210261Z 2025-09-07T07:09:04.4210372Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4210574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4210648Z return mod(**inputs) 2025-09-07T07:09:04.4210916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4211010Z outputs = self.model( 2025-09-07T07:09:04.4211277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4211352Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4211613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4211686Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4211916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4211996Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4212250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4212341Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4212346Z 2025-09-07T07:09:04.4212451Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4212659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4212724Z return mod(**inputs) 2025-09-07T07:09:04.4212999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4213073Z outputs = self.model( 2025-09-07T07:09:04.4213327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4213407Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4213660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4213740Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4213967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4214078Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4214364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4214472Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4214749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4214911Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4214915Z 2025-09-07T07:09:04.4215026Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4215246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4215336Z return mod(**inputs) 2025-09-07T07:09:04.4215615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4215699Z outputs = self.model( 2025-09-07T07:09:04.4215961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4216032Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4216290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4216367Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4216588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4216674Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4216927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4217028Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4217314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4217396Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4217399Z 2025-09-07T07:09:04.4217508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4217711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4217775Z return mod(**inputs) 2025-09-07T07:09:04.4218037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4218105Z outputs = self.model( 2025-09-07T07:09:04.4218364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4218437Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4218702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4218778Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4219002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4219108Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4219362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4219467Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4219975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4220068Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4220075Z 2025-09-07T07:09:04.4220170Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4220252Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4220344Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4220473Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4220581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4220793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4220860Z return mod(**inputs) 2025-09-07T07:09:04.4221130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4221199Z outputs = self.model( 2025-09-07T07:09:04.4221461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4221544Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4221829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4221914Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4222140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4222230Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4222489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4222591Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4222860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4222962Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4223287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4223431Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4223464Z 2025-09-07T07:09:04.4223574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4223796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4223866Z return mod(**inputs) 2025-09-07T07:09:04.4224148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4224221Z outputs = self.model( 2025-09-07T07:09:04.4224502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4224580Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4224851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4224934Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4225172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4225266Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4225548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4225657Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4226043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4226150Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4226484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4226604Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4226608Z 2025-09-07T07:09:04.4226726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4226949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4227033Z return mod(**inputs) 2025-09-07T07:09:04.4227338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4227412Z outputs = self.model( 2025-09-07T07:09:04.4227698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4227778Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4228058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4228147Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4228388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4228500Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4228771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4228880Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4229159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4229251Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4229254Z 2025-09-07T07:09:04.4229372Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4229581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4229657Z return mod(**inputs) 2025-09-07T07:09:04.4229929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4230001Z outputs = self.model( 2025-09-07T07:09:04.4230281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4230379Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4230658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4230734Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4230971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4231063Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4231333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4231460Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4231731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4231901Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4231906Z 2025-09-07T07:09:04.4232019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4232231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4232306Z return mod(**inputs) 2025-09-07T07:09:04.4232600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4232678Z outputs = self.model( 2025-09-07T07:09:04.4232950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4233024Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4233303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4233381Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4233626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4233726Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4234011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4234129Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4234399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4234493Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4234497Z 2025-09-07T07:09:04.4234605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4234825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4234896Z return mod(**inputs) 2025-09-07T07:09:04.4235184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4235268Z outputs = self.model( 2025-09-07T07:09:04.4235544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4235627Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4235901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4235977Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4236223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4236307Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4236588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4236704Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4237002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4237096Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4237100Z 2025-09-07T07:09:04.4237188Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4237280Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4237361Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4237449Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4237560Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4237771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4237847Z return mod(**inputs) 2025-09-07T07:09:04.4238119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4238198Z outputs = self.model( 2025-09-07T07:09:04.4238473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4238548Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4238828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4238937Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4239192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4239270Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4239533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4239650Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4239909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4240017Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4240352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4240496Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4240501Z 2025-09-07T07:09:04.4240604Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4240805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4240880Z return mod(**inputs) 2025-09-07T07:09:04.4241143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4241218Z outputs = self.model( 2025-09-07T07:09:04.4241495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4241572Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4241837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4241908Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4242137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4242218Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4242482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4242592Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4242853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4242962Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4243276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4243392Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4243396Z 2025-09-07T07:09:04.4243500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4243706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4243784Z return mod(**inputs) 2025-09-07T07:09:04.4244047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4244128Z outputs = self.model( 2025-09-07T07:09:04.4244390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4244474Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4244736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4244816Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4245054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4245135Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4245419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4245525Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4245780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4245872Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4245875Z 2025-09-07T07:09:04.4245978Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4246187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4246254Z return mod(**inputs) 2025-09-07T07:09:04.4246540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4246609Z outputs = self.model( 2025-09-07T07:09:04.4246871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4246949Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4247205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4247285Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4247507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4247602Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4247869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-09-07T07:09:04.4247962Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.4247966Z 2025-09-07T07:09:04.4248086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4248307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4248381Z return mod(**inputs) 2025-09-07T07:09:04.4248681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4248752Z outputs = self.model( 2025-09-07T07:09:04.4249039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4249115Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4249404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4249500Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4249740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4249833Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4250113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4250241Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4250245Z 2025-09-07T07:09:04.4250347Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4250547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4250621Z return mod(**inputs) 2025-09-07T07:09:04.4250880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4250959Z outputs = self.model( 2025-09-07T07:09:04.4251216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4251289Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4251552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4251642Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4251872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4251950Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4252214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4252335Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4252551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4252631Z return self.act(input) 2025-09-07T07:09:04.4252917Z 2025-09-07T07:09:04.4253024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4253233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4253303Z return mod(**inputs) 2025-09-07T07:09:04.4253571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4253648Z outputs = self.model( 2025-09-07T07:09:04.4253895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4253976Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4254243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4254327Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4254548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4254628Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4254889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4254973Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4254977Z 2025-09-07T07:09:04.4255089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4255284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4255350Z return mod(**inputs) 2025-09-07T07:09:04.4255611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4255679Z outputs = self.model( 2025-09-07T07:09:04.4255954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4256028Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4256279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4256360Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4256582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4256669Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4256922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4257029Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4257280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4257436Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4257439Z 2025-09-07T07:09:04.4257552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4257751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4257846Z return mod(**inputs) 2025-09-07T07:09:04.4258094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4258159Z outputs = self.model( 2025-09-07T07:09:04.4258416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4258486Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4258744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4258814Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4259054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4259134Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4259386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4259492Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4259746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4259834Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4259837Z 2025-09-07T07:09:04.4259935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4260133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4260224Z return mod(**inputs) 2025-09-07T07:09:04.4260477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4260552Z outputs = self.model( 2025-09-07T07:09:04.4260808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4260881Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4261135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4261206Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4261426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4261505Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4261768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4261899Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4262154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4262251Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4262254Z 2025-09-07T07:09:04.4262338Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4262426Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4262505Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4262583Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4262696Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4262894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4262967Z return mod(**inputs) 2025-09-07T07:09:04.4263226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4263295Z outputs = self.model( 2025-09-07T07:09:04.4263563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4263635Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4263899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4263990Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4264223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4264303Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4264563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4264673Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4264938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4268889Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4269242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4269383Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4269387Z 2025-09-07T07:09:04.4269500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4269697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4269772Z return mod(**inputs) 2025-09-07T07:09:04.4270028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4270097Z outputs = self.model( 2025-09-07T07:09:04.4270392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4270470Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4270767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4270847Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4271069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4271153Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4271407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4271509Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4271761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4271856Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4272178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4272287Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4272290Z 2025-09-07T07:09:04.4272398Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4272611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4272675Z return mod(**inputs) 2025-09-07T07:09:04.4272930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4272995Z outputs = self.model( 2025-09-07T07:09:04.4273250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4273324Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4273586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4273660Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4273895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4274012Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4274282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-09-07T07:09:04.4274392Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:09:04.4274662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4274751Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4274755Z 2025-09-07T07:09:04.4274873Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4275101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4275174Z return mod(**inputs) 2025-09-07T07:09:04.4275492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4275561Z outputs = self.model( 2025-09-07T07:09:04.4275826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4275896Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4276156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4276227Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4276458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4276553Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4276806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4276923Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4277171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-09-07T07:09:04.4277330Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:09:04.4277334Z 2025-09-07T07:09:04.4277435Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4277631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4277705Z return mod(**inputs) 2025-09-07T07:09:04.4277957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4278034Z outputs = self.model( 2025-09-07T07:09:04.4278306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4278385Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4278639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4278720Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4278942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4279017Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4279270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4279373Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4279620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-09-07T07:09:04.4279706Z key_states = self.k_proj(current_states) 2025-09-07T07:09:04.4279710Z 2025-09-07T07:09:04.4279809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4280007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4280087Z return mod(**inputs) 2025-09-07T07:09:04.4280331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4280401Z outputs = self.model( 2025-09-07T07:09:04.4280644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4280721Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4280963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4281040Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4281252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4281358Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4281609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4281714Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4281963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-09-07T07:09:04.4282045Z value_states = self.v_proj(current_states) 2025-09-07T07:09:04.4282049Z 2025-09-07T07:09:04.4282126Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4282209Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4282284Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4282416Z cudagraph partition due to non gpu ops 2025-09-07T07:09:04.4282521Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4282718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4282790Z return mod(**inputs) 2025-09-07T07:09:04.4283043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4283118Z outputs = self.model( 2025-09-07T07:09:04.4283369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4283450Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4283700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4283773Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4284000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4284098Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4284352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4284457Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4284705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4284809Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4285101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:09:04.4285236Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:09:04.4285239Z 2025-09-07T07:09:04.4285340Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4285544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4285612Z return mod(**inputs) 2025-09-07T07:09:04.4285865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4285941Z outputs = self.model( 2025-09-07T07:09:04.4286190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4286286Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4286536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4286606Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4286833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4286912Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4287169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4287303Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4287554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-09-07T07:09:04.4287659Z attn_output, attn_weights = attention_interface( 2025-09-07T07:09:04.4287950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:09:04.4288062Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:04.4288065Z 2025-09-07T07:09:04.4288166Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4288368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4288445Z return mod(**inputs) 2025-09-07T07:09:04.4288697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4288775Z outputs = self.model( 2025-09-07T07:09:04.4289024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4289102Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4289353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4289423Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4289645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4289725Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4289979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-09-07T07:09:04.4290085Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:09:04.4290363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-09-07T07:09:04.4290444Z attn_output = self.out_proj(attn_output) 2025-09-07T07:09:04.4290448Z 2025-09-07T07:09:04.4290548Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4290752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4290818Z return mod(**inputs) 2025-09-07T07:09:04.4291075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4291140Z outputs = self.model( 2025-09-07T07:09:04.4291386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4291468Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4291717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4291797Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4292018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4292113Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4292372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4292491Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4292495Z 2025-09-07T07:09:04.4292607Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4292807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4292881Z return mod(**inputs) 2025-09-07T07:09:04.4293138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4293207Z outputs = self.model( 2025-09-07T07:09:04.4293490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4293563Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4293830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4293900Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4294118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4294201Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4294451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-09-07T07:09:04.4294586Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:09:04.4294801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:09:04.4294876Z return self.act(input) 2025-09-07T07:09:04.4294880Z 2025-09-07T07:09:04.4294981Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4295178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4295249Z return mod(**inputs) 2025-09-07T07:09:04.4295500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4295572Z outputs = self.model( 2025-09-07T07:09:04.4295823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4295892Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4296152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4296240Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4296467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4296544Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4296795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-09-07T07:09:04.4296883Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:09:04.4296887Z 2025-09-07T07:09:04.4296987Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4297188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4297253Z return mod(**inputs) 2025-09-07T07:09:04.4297512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-09-07T07:09:04.4297579Z outputs = self.model( 2025-09-07T07:09:04.4297829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-09-07T07:09:04.4297908Z decoder_outputs = self.decoder( 2025-09-07T07:09:04.4298156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-09-07T07:09:04.4298253Z layer_outputs = decoder_layer( 2025-09-07T07:09:04.4298471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:04.4298548Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:04.4298810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-09-07T07:09:04.4298890Z hidden_states = residual + hidden_states 2025-09-07T07:09:04.4298895Z 2025-09-07T07:09:04.4299005Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4299201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4299283Z return mod(**inputs) 2025-09-07T07:09:04.4299538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1456, in forward 2025-09-07T07:09:04.4299658Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-09-07T07:09:04.4299662Z 2025-09-07T07:09:04.4299771Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:04.4299966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:04.4300037Z return mod(**inputs) 2025-09-07T07:09:04.4300290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1461, in forward 2025-09-07T07:09:04.4300472Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:09:04.4300478Z 2025-09-07T07:09:20.1871210Z Compilation time (from dynamo_timed): 30.718431175 2025-09-07T07:09:20.2101656Z pass 2025-09-07T07:09:20.2102128Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:09:20.2103052Z TIMING: _recursive_pre_grad_passes:0.01436 _recursive_joint_graph_passes:0.80593 _recursive_post_grad_passes:0.1755 async_compile.wait:0.7721 code_gen:14.15249 inductor_compile:17.41518 backend_compile:24.74383 gc:0.00064 entire_frame_compile:30.71843 total_wall_time:30.71843 2025-09-07T07:09:20.2104090Z STATS: call_* op count: 986 | FakeTensorMode.__torch_dispatch__:33710 | FakeTensor.__torch_dispatch__:11299 | ProxyTorchDispatchMode.__torch_dispatch__:12456 2025-09-07T07:09:20.2104663Z Dynamo produced 1 graphs covering 986 ops with 0 graph breaks (0 unique) 2025-09-07T07:09:23.4584909Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:09:23.4586435Z import pynvml # type: ignore[import] 2025-09-07T07:09:26.2899354Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:09:26.2900349Z from pkg_resources import resource_filename 2025-09-07T07:09:27.0376557Z 2025-09-07T07:09:29.4819223Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:09:29.4819535Z loading model: 0it [00:02, ?it/s] 2025-09-07T07:09:29.4838030Z cpu eval MT5ForConditionalGeneration 2025-09-07T07:09:30.0833938Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:09:30.3454825Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:09:30.6087211Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:09:43.5060322Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5061119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5061583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5061985Z return mod(**inputs) 2025-09-07T07:09:43.5062445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5062899Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5063338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5063789Z layer_outputs = layer_module( 2025-09-07T07:09:43.5064248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5064781Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5065229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5065926Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5066394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5066851Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5067287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 421, in forward 2025-09-07T07:09:43.5067722Z position_bias = position_bias + causal_mask 2025-09-07T07:09:43.5067897Z 2025-09-07T07:09:43.5068071Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5068493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5068868Z return mod(**inputs) 2025-09-07T07:09:43.5069268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5069705Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5070109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5070528Z layer_outputs = layer_module( 2025-09-07T07:09:43.5070923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5071482Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5071912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5072359Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5072844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5073300Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5073740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5074159Z return self.weight * hidden_states 2025-09-07T07:09:43.5074313Z 2025-09-07T07:09:43.5074429Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5074842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5075208Z return mod(**inputs) 2025-09-07T07:09:43.5075618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5076047Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5076463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5076889Z layer_outputs = layer_module( 2025-09-07T07:09:43.5077274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5077671Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5078117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5078539Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5078959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5079393Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5079822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5080243Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5080396Z 2025-09-07T07:09:43.5080511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5080931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5081296Z return mod(**inputs) 2025-09-07T07:09:43.5081681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5082096Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5082505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5082925Z layer_outputs = layer_module( 2025-09-07T07:09:43.5083310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5083735Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5084155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5084594Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5085015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5085438Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5085861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5086275Z key_states = self.k(current_states) 2025-09-07T07:09:43.5086417Z 2025-09-07T07:09:43.5086540Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5086937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5087298Z return mod(**inputs) 2025-09-07T07:09:43.5087685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5088134Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5088543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5088956Z layer_outputs = layer_module( 2025-09-07T07:09:43.5089341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5089737Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5090155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5090594Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5091000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5091433Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5091846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5092321Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5092520Z 2025-09-07T07:09:43.5092638Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5093025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5093397Z return mod(**inputs) 2025-09-07T07:09:43.5093777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5094185Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5094576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5094980Z layer_outputs = layer_module( 2025-09-07T07:09:43.5095358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5095757Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5096188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5096602Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5097035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5097450Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5097861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5098355Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5098583Z 2025-09-07T07:09:43.5098694Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5099099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5099457Z return mod(**inputs) 2025-09-07T07:09:43.5099870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5100292Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5100706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5101124Z layer_outputs = layer_module( 2025-09-07T07:09:43.5101510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5101915Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5102329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5102757Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5103182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5103688Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5104117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5104545Z value_states = self.v(current_states) 2025-09-07T07:09:43.5104708Z 2025-09-07T07:09:43.5104826Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5105234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5105601Z return mod(**inputs) 2025-09-07T07:09:43.5106076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5106500Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5106927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5107348Z layer_outputs = layer_module( 2025-09-07T07:09:43.5107743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5108144Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5108570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5109033Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5109461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5109890Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5110311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5110781Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5110974Z 2025-09-07T07:09:43.5111091Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5111524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5111881Z return mod(**inputs) 2025-09-07T07:09:43.5112275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5112698Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5113112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5113529Z layer_outputs = layer_module( 2025-09-07T07:09:43.5113909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5114311Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5114746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5115166Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5115574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5115980Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5116392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5116836Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5117013Z 2025-09-07T07:09:43.5117134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5117518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5117872Z return mod(**inputs) 2025-09-07T07:09:43.5118257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5118688Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5119095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5119505Z layer_outputs = layer_module( 2025-09-07T07:09:43.5120089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5120504Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5120921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5121342Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5121745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5122162Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5122575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5123022Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5123206Z 2025-09-07T07:09:43.5123321Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5123718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5124141Z return mod(**inputs) 2025-09-07T07:09:43.5124526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5124929Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5125325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5125742Z layer_outputs = layer_module( 2025-09-07T07:09:43.5126124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5126520Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5126962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5127373Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5127785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5128203Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5128615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5129022Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5129175Z 2025-09-07T07:09:43.5129292Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5129716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5130069Z return mod(**inputs) 2025-09-07T07:09:43.5130461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5130870Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5131283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5131703Z layer_outputs = layer_module( 2025-09-07T07:09:43.5132085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5132484Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5132895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5133342Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5133784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5134246Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5134664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5135089Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5135245Z 2025-09-07T07:09:43.5135365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5135768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5136133Z return mod(**inputs) 2025-09-07T07:09:43.5136517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5136943Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5137358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5137775Z layer_outputs = layer_module( 2025-09-07T07:09:43.5138165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5138562Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5138984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5139434Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5139858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5140283Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5140713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5141148Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5141298Z 2025-09-07T07:09:43.5141438Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5141865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5142240Z return mod(**inputs) 2025-09-07T07:09:43.5142661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5143084Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5143504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5143932Z layer_outputs = layer_module( 2025-09-07T07:09:43.5144319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5144725Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5145156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5145684Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5146122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5146566Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5147003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5148244Z key_states = self.k(current_states) 2025-09-07T07:09:43.5148395Z 2025-09-07T07:09:43.5148519Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5148917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5149342Z return mod(**inputs) 2025-09-07T07:09:43.5149732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5150153Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5150569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5151013Z layer_outputs = layer_module( 2025-09-07T07:09:43.5151503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5151925Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5152333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5152747Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5153157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5153587Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5154012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5154497Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5154699Z 2025-09-07T07:09:43.5154813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5155206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5155563Z return mod(**inputs) 2025-09-07T07:09:43.5155976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5156390Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5156785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5157214Z layer_outputs = layer_module( 2025-09-07T07:09:43.5157593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5157984Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5158402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5158845Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5159273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5159706Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5160132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5160640Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5160885Z 2025-09-07T07:09:43.5161005Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5161409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5161763Z return mod(**inputs) 2025-09-07T07:09:43.5162176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5162581Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5162992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5163419Z layer_outputs = layer_module( 2025-09-07T07:09:43.5163791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5164179Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5164578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5164994Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5165411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5165827Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5166255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5166747Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5166982Z 2025-09-07T07:09:43.5167095Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5167489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5167843Z return mod(**inputs) 2025-09-07T07:09:43.5168214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5168624Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5169027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5169439Z layer_outputs = layer_module( 2025-09-07T07:09:43.5169818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5170208Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5170616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5171047Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5171452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5171862Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5172275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5172762Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5172986Z 2025-09-07T07:09:43.5173108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5173499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5173840Z return mod(**inputs) 2025-09-07T07:09:43.5174235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5174659Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5175076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5175569Z layer_outputs = layer_module( 2025-09-07T07:09:43.5175963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5176356Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5176768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5177204Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5177617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5178051Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5178485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5178914Z value_states = self.v(current_states) 2025-09-07T07:09:43.5179068Z 2025-09-07T07:09:43.5179193Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5179588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5179953Z return mod(**inputs) 2025-09-07T07:09:43.5180335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5180759Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5181164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5182402Z layer_outputs = layer_module( 2025-09-07T07:09:43.5182793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5183200Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5183621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5184043Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5184468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5184896Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5185324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5185924Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5186113Z 2025-09-07T07:09:43.5186230Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5186635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5187006Z return mod(**inputs) 2025-09-07T07:09:43.5187443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5187858Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5188276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5188695Z layer_outputs = layer_module( 2025-09-07T07:09:43.5189086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5189495Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5189923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5190378Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5190811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5191252Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5191688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5192131Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5192317Z 2025-09-07T07:09:43.5192433Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5192826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5193185Z return mod(**inputs) 2025-09-07T07:09:43.5193590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5194002Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5194406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5194813Z layer_outputs = layer_module( 2025-09-07T07:09:43.5195197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5195581Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5195986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5196401Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5196809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5197219Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5197649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5198101Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5198276Z 2025-09-07T07:09:43.5198398Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5198789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5199156Z return mod(**inputs) 2025-09-07T07:09:43.5199539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5199957Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5200364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5200771Z layer_outputs = layer_module( 2025-09-07T07:09:43.5201146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5201547Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5201958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5202372Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5202792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5203212Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5203627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5204038Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5204181Z 2025-09-07T07:09:43.5204279Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5204538Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5204940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5205295Z return mod(**inputs) 2025-09-07T07:09:43.5205702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5206127Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5206534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5206921Z layer_outputs = layer_module( 2025-09-07T07:09:43.5207283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5207654Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5208031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5208472Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5208880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5209292Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5209699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5210084Z return self.weight * hidden_states 2025-09-07T07:09:43.5210225Z 2025-09-07T07:09:43.5210330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5210693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5211025Z return mod(**inputs) 2025-09-07T07:09:43.5211377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5211765Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5212148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5212554Z layer_outputs = layer_module( 2025-09-07T07:09:43.5212910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5213297Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5213707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5214131Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5214566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5215024Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5215454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5215862Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5216023Z 2025-09-07T07:09:43.5216130Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5216498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5216826Z return mod(**inputs) 2025-09-07T07:09:43.5217206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5217590Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5217972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5218356Z layer_outputs = layer_module( 2025-09-07T07:09:43.5218706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5219079Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5219464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5220232Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5220638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5221066Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5221503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5221906Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5222050Z 2025-09-07T07:09:43.5222171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5222544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5222890Z return mod(**inputs) 2025-09-07T07:09:43.5223311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5223725Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5224134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5224535Z layer_outputs = layer_module( 2025-09-07T07:09:43.5224914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5225314Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5225786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5226233Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5226660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5227131Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5227621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5228046Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5228202Z 2025-09-07T07:09:43.5228318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5228716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5229072Z return mod(**inputs) 2025-09-07T07:09:43.5229453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5229862Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5230255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5230659Z layer_outputs = layer_module( 2025-09-07T07:09:43.5231035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5231428Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5231835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5232282Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5232702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5233152Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5233606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5233988Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5234135Z 2025-09-07T07:09:43.5234220Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5234472Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5234858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5235252Z return mod(**inputs) 2025-09-07T07:09:43.5235592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5235960Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5236319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5236680Z layer_outputs = layer_module( 2025-09-07T07:09:43.5237006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5237354Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5237737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5238114Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5238494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5238884Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5239281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5239656Z return self.weight * hidden_states 2025-09-07T07:09:43.5239787Z 2025-09-07T07:09:43.5239898Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5240248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5240561Z return mod(**inputs) 2025-09-07T07:09:43.5240907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5241275Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5241651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5242015Z layer_outputs = layer_module( 2025-09-07T07:09:43.5242353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5242709Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5243084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5243467Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5243837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5244225Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5244607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5244988Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5245131Z 2025-09-07T07:09:43.5245242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5245584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5245903Z return mod(**inputs) 2025-09-07T07:09:43.5246297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5246674Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5247037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5247423Z layer_outputs = layer_module( 2025-09-07T07:09:43.5247761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5248120Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5248491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5248874Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5249246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5249623Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5250000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5250369Z key_states = self.k(current_states) 2025-09-07T07:09:43.5250507Z 2025-09-07T07:09:43.5250612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5250977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5251304Z return mod(**inputs) 2025-09-07T07:09:43.5251678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5252051Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5252423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5252799Z layer_outputs = layer_module( 2025-09-07T07:09:43.5253144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5253508Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5253882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5254272Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5254661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5255058Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5255463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5255897Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5256085Z 2025-09-07T07:09:43.5256190Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5256561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5256892Z return mod(**inputs) 2025-09-07T07:09:43.5257245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5257630Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5258008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5258407Z layer_outputs = layer_module( 2025-09-07T07:09:43.5258753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5259112Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5259490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5259890Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5260298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5260681Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5261067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5261529Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5261743Z 2025-09-07T07:09:43.5261858Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5262227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5262552Z return mod(**inputs) 2025-09-07T07:09:43.5262933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5263341Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5263749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5264162Z layer_outputs = layer_module( 2025-09-07T07:09:43.5264533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5264930Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5265341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5265900Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5266323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5266756Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5267196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5267720Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5267952Z 2025-09-07T07:09:43.5268082Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5268435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5268765Z return mod(**inputs) 2025-09-07T07:09:43.5269122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5269543Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5269957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5270464Z layer_outputs = layer_module( 2025-09-07T07:09:43.5270867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5271283Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5271709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5272130Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5272555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5272987Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5273417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5273933Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5274165Z 2025-09-07T07:09:43.5274284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5274698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5275070Z return mod(**inputs) 2025-09-07T07:09:43.5275500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5276000Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5276416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5276824Z layer_outputs = layer_module( 2025-09-07T07:09:43.5277205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5277610Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5278011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5278438Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5278851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5279265Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5279672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5280075Z value_states = self.v(current_states) 2025-09-07T07:09:43.5280228Z 2025-09-07T07:09:43.5280341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5280730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5281106Z return mod(**inputs) 2025-09-07T07:09:43.5281504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5281914Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5282326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5282736Z layer_outputs = layer_module( 2025-09-07T07:09:43.5283114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5283472Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5283857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5284239Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5284620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5285008Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5285396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5285818Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5285996Z 2025-09-07T07:09:43.5286111Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5286501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5286872Z return mod(**inputs) 2025-09-07T07:09:43.5287254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5287637Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5288017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5288401Z layer_outputs = layer_module( 2025-09-07T07:09:43.5288750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5289111Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5289482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5289870Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5290286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5290676Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5291071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5291495Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5291660Z 2025-09-07T07:09:43.5291776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5292154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5292488Z return mod(**inputs) 2025-09-07T07:09:43.5292892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5293303Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5293706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5294114Z layer_outputs = layer_module( 2025-09-07T07:09:43.5294499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5294904Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5295327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5295774Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5296191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5296629Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5297039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5297491Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5297666Z 2025-09-07T07:09:43.5297779Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5298173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5298530Z return mod(**inputs) 2025-09-07T07:09:43.5298916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5299324Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5299722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5300149Z layer_outputs = layer_module( 2025-09-07T07:09:43.5300528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5300921Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5301336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5301742Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5302151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5302565Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5302974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5303379Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5303530Z 2025-09-07T07:09:43.5303644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5304040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5304404Z return mod(**inputs) 2025-09-07T07:09:43.5304795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5305232Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5305725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5306163Z layer_outputs = layer_module( 2025-09-07T07:09:43.5306550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5306958Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5307383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5307814Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5308264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5308700Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5309141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5309564Z return self.weight * hidden_states 2025-09-07T07:09:43.5309718Z 2025-09-07T07:09:43.5309830Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5310227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5310593Z return mod(**inputs) 2025-09-07T07:09:43.5310988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5311417Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5311835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5312259Z layer_outputs = layer_module( 2025-09-07T07:09:43.5312637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5313045Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5313454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5313885Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5314309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5314757Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5315223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5315686Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5315850Z 2025-09-07T07:09:43.5315972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5316370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5316731Z return mod(**inputs) 2025-09-07T07:09:43.5317112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5317516Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5317920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5318326Z layer_outputs = layer_module( 2025-09-07T07:09:43.5318713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5319116Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5319529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5320170Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5320646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5321099Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5321548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5321963Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5322113Z 2025-09-07T07:09:43.5322233Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5322618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5322978Z return mod(**inputs) 2025-09-07T07:09:43.5323389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5323779Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5324170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5324587Z layer_outputs = layer_module( 2025-09-07T07:09:43.5324941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5325316Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5325721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5326133Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5326563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5326995Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5327420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5327820Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5327970Z 2025-09-07T07:09:43.5328079Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5328455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5328791Z return mod(**inputs) 2025-09-07T07:09:43.5329152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5329534Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5329921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5330377Z layer_outputs = layer_module( 2025-09-07T07:09:43.5330755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5331161Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5331572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5332021Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5332440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5332876Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5333311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5333701Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5333856Z 2025-09-07T07:09:43.5333949Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5334216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5334619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5334976Z return mod(**inputs) 2025-09-07T07:09:43.5335398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5335815Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5336210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5336597Z layer_outputs = layer_module( 2025-09-07T07:09:43.5336945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5337321Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5337708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5338140Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5338546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5338999Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5339445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5339867Z return self.weight * hidden_states 2025-09-07T07:09:43.5340011Z 2025-09-07T07:09:43.5340132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5340527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5340901Z return mod(**inputs) 2025-09-07T07:09:43.5341305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5341722Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5342130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5342535Z layer_outputs = layer_module( 2025-09-07T07:09:43.5342911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5343310Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5343722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5344131Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5344543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5344962Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5345399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5345894Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5346049Z 2025-09-07T07:09:43.5346168Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5346579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5346955Z return mod(**inputs) 2025-09-07T07:09:43.5347345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5347755Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5348151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5348559Z layer_outputs = layer_module( 2025-09-07T07:09:43.5348944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5349344Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5349753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5350169Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5350619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5351034Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5351442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5351842Z key_states = self.k(current_states) 2025-09-07T07:09:43.5351994Z 2025-09-07T07:09:43.5352106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5352496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5352852Z return mod(**inputs) 2025-09-07T07:09:43.5353210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5353597Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5353980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5354370Z layer_outputs = layer_module( 2025-09-07T07:09:43.5354724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5355091Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5355486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5355866Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5356264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5356649Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5357023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5357455Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5357649Z 2025-09-07T07:09:43.5357755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5358115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5358438Z return mod(**inputs) 2025-09-07T07:09:43.5358782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5359154Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5359516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5359903Z layer_outputs = layer_module( 2025-09-07T07:09:43.5360231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5360586Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5360952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5361326Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5361692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5362060Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5362433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5362894Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5363111Z 2025-09-07T07:09:43.5363226Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5363595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5363927Z return mod(**inputs) 2025-09-07T07:09:43.5364291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5364698Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5365094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5365468Z layer_outputs = layer_module( 2025-09-07T07:09:43.5365828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5366201Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5366595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5366991Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5367392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5367784Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5368171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5368622Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5368825Z 2025-09-07T07:09:43.5368937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5369294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5369627Z return mod(**inputs) 2025-09-07T07:09:43.5370004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5370409Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5370817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5371201Z layer_outputs = layer_module( 2025-09-07T07:09:43.5371557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5371932Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5372319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5372704Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5373116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5373542Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5373955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5374471Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5374708Z 2025-09-07T07:09:43.5374816Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5375190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5375525Z return mod(**inputs) 2025-09-07T07:09:43.5375887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5376267Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5376649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5377032Z layer_outputs = layer_module( 2025-09-07T07:09:43.5377392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5377765Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5378145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5378540Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5378934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5379315Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5379686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5380068Z value_states = self.v(current_states) 2025-09-07T07:09:43.5380208Z 2025-09-07T07:09:43.5380314Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5380687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5381023Z return mod(**inputs) 2025-09-07T07:09:43.5381398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5381792Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5382171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5382560Z layer_outputs = layer_module( 2025-09-07T07:09:43.5382922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5383311Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5383717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5384132Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5384572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5384982Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5385399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5385935Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5386123Z 2025-09-07T07:09:43.5386254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5386663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5387042Z return mod(**inputs) 2025-09-07T07:09:43.5387443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5387884Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5388304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5388753Z layer_outputs = layer_module( 2025-09-07T07:09:43.5389135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5389539Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5389958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5390396Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5390804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5391232Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5391651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5392114Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5392291Z 2025-09-07T07:09:43.5392415Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5392809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5393183Z return mod(**inputs) 2025-09-07T07:09:43.5393573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5394015Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5394394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5394778Z layer_outputs = layer_module( 2025-09-07T07:09:43.5395139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5395514Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5395903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5396293Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5396702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5397097Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5397494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5397920Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5398089Z 2025-09-07T07:09:43.5398199Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5398574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5398961Z return mod(**inputs) 2025-09-07T07:09:43.5399353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5399735Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5400122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5400509Z layer_outputs = layer_module( 2025-09-07T07:09:43.5400870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5401246Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5401628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5402025Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5402417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5402811Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5403201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5403601Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5403742Z 2025-09-07T07:09:43.5403827Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5404081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5404448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5404775Z return mod(**inputs) 2025-09-07T07:09:43.5405140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5405528Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5405913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5406298Z layer_outputs = layer_module( 2025-09-07T07:09:43.5406654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5407030Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5407421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5407824Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5408247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5408666Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5409058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5409450Z return self.weight * hidden_states 2025-09-07T07:09:43.5409587Z 2025-09-07T07:09:43.5409713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5410074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5410402Z return mod(**inputs) 2025-09-07T07:09:43.5410768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5411144Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5411515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5411883Z layer_outputs = layer_module( 2025-09-07T07:09:43.5412230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5412591Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5412965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5413351Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5413762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5414181Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5414595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5414997Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5415151Z 2025-09-07T07:09:43.5415255Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5415621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5415952Z return mod(**inputs) 2025-09-07T07:09:43.5416298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5416675Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5417045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5417448Z layer_outputs = layer_module( 2025-09-07T07:09:43.5417809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5418173Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5418545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5418936Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5419322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5419865Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5420288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5420685Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5420839Z 2025-09-07T07:09:43.5420947Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5421321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5421665Z return mod(**inputs) 2025-09-07T07:09:43.5422028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5422462Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5422842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5423225Z layer_outputs = layer_module( 2025-09-07T07:09:43.5423585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5423958Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5424379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5424814Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5425276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5425783Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5426249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5426685Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5426852Z 2025-09-07T07:09:43.5426970Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5427383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5427744Z return mod(**inputs) 2025-09-07T07:09:43.5428150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5428564Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5428966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5429373Z layer_outputs = layer_module( 2025-09-07T07:09:43.5429742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5430135Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5430542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5430965Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5431384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5431827Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5432273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5432721Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5432867Z 2025-09-07T07:09:43.5432966Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5433227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5433613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5433973Z return mod(**inputs) 2025-09-07T07:09:43.5434356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5434778Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5435187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5435644Z layer_outputs = layer_module( 2025-09-07T07:09:43.5436002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5436384Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5436771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5437176Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5437562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5437977Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5438387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5438765Z return self.weight * hidden_states 2025-09-07T07:09:43.5438909Z 2025-09-07T07:09:43.5439016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5439388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5439731Z return mod(**inputs) 2025-09-07T07:09:43.5440125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5440512Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5440901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5441288Z layer_outputs = layer_module( 2025-09-07T07:09:43.5441648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5442020Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5442400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5442809Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5443201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5443605Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5443999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5444400Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5444547Z 2025-09-07T07:09:43.5444659Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5445036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5445378Z return mod(**inputs) 2025-09-07T07:09:43.5445742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5446134Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5446528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5446912Z layer_outputs = layer_module( 2025-09-07T07:09:43.5447243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5447593Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5447971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5448353Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5448728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5449105Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5449472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5449845Z key_states = self.k(current_states) 2025-09-07T07:09:43.5449976Z 2025-09-07T07:09:43.5450086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5450440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5450758Z return mod(**inputs) 2025-09-07T07:09:43.5451108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5451509Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5451890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5452247Z layer_outputs = layer_module( 2025-09-07T07:09:43.5452588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5452948Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5453321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5453701Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5454091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5454473Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5454849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5455277Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5455468Z 2025-09-07T07:09:43.5455577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5455921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5456240Z return mod(**inputs) 2025-09-07T07:09:43.5456598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5456969Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5457326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5457689Z layer_outputs = layer_module( 2025-09-07T07:09:43.5458046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5458432Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5458809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5459186Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5459570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5459965Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5460358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5460887Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5461101Z 2025-09-07T07:09:43.5461214Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5461592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5461937Z return mod(**inputs) 2025-09-07T07:09:43.5462302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5462703Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5463114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5463527Z layer_outputs = layer_module( 2025-09-07T07:09:43.5463911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5464314Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5464726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5465150Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5465701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5466144Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5466561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5466738Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5466742Z 2025-09-07T07:09:43.5466863Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5467079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5467151Z return mod(**inputs) 2025-09-07T07:09:43.5467427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5467506Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5467752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5467834Z layer_outputs = layer_module( 2025-09-07T07:09:43.5468060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5468149Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5468391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5468473Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5468739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5468825Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5469076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5469229Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5469234Z 2025-09-07T07:09:43.5469349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5469552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5469620Z return mod(**inputs) 2025-09-07T07:09:43.5469879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5469953Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5470210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5470303Z layer_outputs = layer_module( 2025-09-07T07:09:43.5470530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5470618Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5470861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5470951Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5471192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5471283Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5471523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5471603Z value_states = self.v(current_states) 2025-09-07T07:09:43.5471608Z 2025-09-07T07:09:43.5471720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5471926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5472001Z return mod(**inputs) 2025-09-07T07:09:43.5472245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5472338Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5472607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5472681Z layer_outputs = layer_module( 2025-09-07T07:09:43.5472915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5472994Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5473246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5473337Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5473603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5473695Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5473939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5474059Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5474063Z 2025-09-07T07:09:43.5474169Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5474370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5474444Z return mod(**inputs) 2025-09-07T07:09:43.5474708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5474792Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5475041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5475113Z layer_outputs = layer_module( 2025-09-07T07:09:43.5475344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5475426Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5475676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5475759Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5476012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5476095Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5476346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5476485Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5476489Z 2025-09-07T07:09:43.5476593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5476803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5476873Z return mod(**inputs) 2025-09-07T07:09:43.5477119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5477200Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5477449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5477528Z layer_outputs = layer_module( 2025-09-07T07:09:43.5477754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5477835Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5478096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5478174Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5478412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5478517Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5478763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5478873Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5478876Z 2025-09-07T07:09:43.5478977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5479184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5479250Z return mod(**inputs) 2025-09-07T07:09:43.5479512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5479585Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5479823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5505373Z layer_outputs = layer_module( 2025-09-07T07:09:43.5505809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5505931Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5506233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5506346Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5506768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5506876Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5507162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5507255Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5507267Z 2025-09-07T07:09:43.5507407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5507646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5507727Z return mod(**inputs) 2025-09-07T07:09:43.5508024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5508110Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5508387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5508469Z layer_outputs = layer_module( 2025-09-07T07:09:43.5508754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5508854Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5509113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5509211Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5509470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-09-07T07:09:43.5509623Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:09:43.5509628Z 2025-09-07T07:09:43.5509719Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5509838Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5510071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5510146Z return mod(**inputs) 2025-09-07T07:09:43.5510421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5510505Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5510768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5510891Z layer_outputs = layer_module( 2025-09-07T07:09:43.5511130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5511228Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5511484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5511584Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5511852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5511959Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5512263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5512352Z return self.weight * hidden_states 2025-09-07T07:09:43.5512357Z 2025-09-07T07:09:43.5512478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5512700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5512773Z return mod(**inputs) 2025-09-07T07:09:43.5513043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5513124Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5513403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5513492Z layer_outputs = layer_module( 2025-09-07T07:09:43.5513732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5513824Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5514082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5514183Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5514449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5514578Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5514842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5514953Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5514956Z 2025-09-07T07:09:43.5515094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5515314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5515386Z return mod(**inputs) 2025-09-07T07:09:43.5515656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5515736Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5516007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5516083Z layer_outputs = layer_module( 2025-09-07T07:09:43.5516322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5516413Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5516657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5516758Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5517003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5517119Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5517386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5517468Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5517472Z 2025-09-07T07:09:43.5517584Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5517787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5517862Z return mod(**inputs) 2025-09-07T07:09:43.5518108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5518183Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5518463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5518537Z layer_outputs = layer_module( 2025-09-07T07:09:43.5518766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5518847Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5519086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5519181Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5519419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5519540Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5519969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5520069Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5520083Z 2025-09-07T07:09:43.5520188Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5520397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5520475Z return mod(**inputs) 2025-09-07T07:09:43.5520727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5520807Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5521059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5521132Z layer_outputs = layer_module( 2025-09-07T07:09:43.5521372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5521483Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5521732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5521822Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5522063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5522188Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5522430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5522520Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5522523Z 2025-09-07T07:09:43.5522607Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5522716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5522919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5522988Z return mod(**inputs) 2025-09-07T07:09:43.5523243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5523316Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5523602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5523674Z layer_outputs = layer_module( 2025-09-07T07:09:43.5523902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5523991Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5524241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5524334Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5524582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5524723Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5524973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5525055Z return self.weight * hidden_states 2025-09-07T07:09:43.5525058Z 2025-09-07T07:09:43.5525170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5525372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5525446Z return mod(**inputs) 2025-09-07T07:09:43.5525689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5525761Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5526029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5526106Z layer_outputs = layer_module( 2025-09-07T07:09:43.5526344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5526425Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5526668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5526757Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5527003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5527097Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5527341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5527428Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5527449Z 2025-09-07T07:09:43.5527566Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5527770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5527845Z return mod(**inputs) 2025-09-07T07:09:43.5528092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5528171Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5528417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5528489Z layer_outputs = layer_module( 2025-09-07T07:09:43.5528721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5528801Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5529052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5529137Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5529385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5529480Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5529742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5529824Z key_states = self.k(current_states) 2025-09-07T07:09:43.5529828Z 2025-09-07T07:09:43.5529925Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5530120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5530182Z return mod(**inputs) 2025-09-07T07:09:43.5530411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5530490Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5530734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5530814Z layer_outputs = layer_module( 2025-09-07T07:09:43.5531030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5531108Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5531345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5531423Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5531666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5531748Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5532002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5532144Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5532149Z 2025-09-07T07:09:43.5532250Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5532455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5532522Z return mod(**inputs) 2025-09-07T07:09:43.5532767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5532837Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5533073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5533149Z layer_outputs = layer_module( 2025-09-07T07:09:43.5533370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5533473Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5533711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5533792Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5534035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5534117Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5534360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5534516Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5534519Z 2025-09-07T07:09:43.5534626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5534820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5534887Z return mod(**inputs) 2025-09-07T07:09:43.5535133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5535205Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5535455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5535553Z layer_outputs = layer_module( 2025-09-07T07:09:43.5535780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5535863Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5536093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5536178Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5536408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5536490Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5536740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5536891Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5536896Z 2025-09-07T07:09:43.5537002Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5537196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5537266Z return mod(**inputs) 2025-09-07T07:09:43.5537500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5537570Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5537825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5537897Z layer_outputs = layer_module( 2025-09-07T07:09:43.5538118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5538195Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5538433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5538520Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5538753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5538838Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5539070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5539226Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5539247Z 2025-09-07T07:09:43.5539349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5539550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5539626Z return mod(**inputs) 2025-09-07T07:09:43.5539866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5539948Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5540192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5540261Z layer_outputs = layer_module( 2025-09-07T07:09:43.5540493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5540573Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5540821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5540902Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5541149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5541231Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5541482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5541569Z value_states = self.v(current_states) 2025-09-07T07:09:43.5541573Z 2025-09-07T07:09:43.5541677Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5541891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5541956Z return mod(**inputs) 2025-09-07T07:09:43.5542213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5542295Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5542560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5542640Z layer_outputs = layer_module( 2025-09-07T07:09:43.5542867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5542948Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5543200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5543282Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5543567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5543651Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5543932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5544059Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5544065Z 2025-09-07T07:09:43.5544174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5544402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5544473Z return mod(**inputs) 2025-09-07T07:09:43.5544748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5544826Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5545088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5545172Z layer_outputs = layer_module( 2025-09-07T07:09:43.5545415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5545528Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5545861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5545961Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5546220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5546310Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5546581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5546698Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5546703Z 2025-09-07T07:09:43.5546821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5547039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5547112Z return mod(**inputs) 2025-09-07T07:09:43.5547385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5547462Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5547733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5547831Z layer_outputs = layer_module( 2025-09-07T07:09:43.5548069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5548162Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5548422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5548516Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5548778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5548873Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5549147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5549265Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5549271Z 2025-09-07T07:09:43.5549388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5549603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5549681Z return mod(**inputs) 2025-09-07T07:09:43.5549940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5550017Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5550308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5550386Z layer_outputs = layer_module( 2025-09-07T07:09:43.5550634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5550718Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5550986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5551073Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5551332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5551427Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5551687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5551778Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5551782Z 2025-09-07T07:09:43.5551871Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5551998Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5552220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5552290Z return mod(**inputs) 2025-09-07T07:09:43.5552557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5552635Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5552896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5552979Z layer_outputs = layer_module( 2025-09-07T07:09:43.5553216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5553310Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5553567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5553674Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5553933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5554037Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5554320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5554412Z return self.weight * hidden_states 2025-09-07T07:09:43.5554415Z 2025-09-07T07:09:43.5554523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5554720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5554785Z return mod(**inputs) 2025-09-07T07:09:43.5555048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5555119Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5555376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5555445Z layer_outputs = layer_module( 2025-09-07T07:09:43.5555656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5555742Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5555968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5556064Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5556296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5556418Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5556676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5556779Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5556783Z 2025-09-07T07:09:43.5556891Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5557087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5557164Z return mod(**inputs) 2025-09-07T07:09:43.5557406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5557477Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5557726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5557796Z layer_outputs = layer_module( 2025-09-07T07:09:43.5558023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5558136Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5558374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5558460Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5558694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5558818Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5559052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5559135Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5559139Z 2025-09-07T07:09:43.5559250Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5559444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5559516Z return mod(**inputs) 2025-09-07T07:09:43.5559752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5559828Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5560065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5560178Z layer_outputs = layer_module( 2025-09-07T07:09:43.5560403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5560480Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5560718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5560803Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5561045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5561156Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5561403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5561499Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5561504Z 2025-09-07T07:09:43.5561602Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5561801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5561864Z return mod(**inputs) 2025-09-07T07:09:43.5562099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5562176Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5562431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5562509Z layer_outputs = layer_module( 2025-09-07T07:09:43.5562726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5562808Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5563038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5563125Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5563362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5563470Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5563742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5563823Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5563827Z 2025-09-07T07:09:43.5563923Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5564034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5564231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5564302Z return mod(**inputs) 2025-09-07T07:09:43.5564541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5564621Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5564869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5564940Z layer_outputs = layer_module( 2025-09-07T07:09:43.5565157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5565243Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5565482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5565585Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5565815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5565945Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5566177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5566253Z return self.weight * hidden_states 2025-09-07T07:09:43.5566257Z 2025-09-07T07:09:43.5566362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5566558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5566628Z return mod(**inputs) 2025-09-07T07:09:43.5566873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5566945Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5567211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5567283Z layer_outputs = layer_module( 2025-09-07T07:09:43.5567516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5567597Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5567857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5567939Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5568183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5568293Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5568542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5568639Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5568642Z 2025-09-07T07:09:43.5568742Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5568934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5569006Z return mod(**inputs) 2025-09-07T07:09:43.5569243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5569319Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5569559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5569627Z layer_outputs = layer_module( 2025-09-07T07:09:43.5569850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5569945Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5570184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5570264Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5570507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5570589Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5570822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5570904Z key_states = self.k(current_states) 2025-09-07T07:09:43.5570907Z 2025-09-07T07:09:43.5571008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5571214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5571279Z return mod(**inputs) 2025-09-07T07:09:43.5571517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5571596Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5571835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5571929Z layer_outputs = layer_module( 2025-09-07T07:09:43.5572148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5572230Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5572465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5572542Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5572790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5572871Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5573137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5573269Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5573274Z 2025-09-07T07:09:43.5573374Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5573577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5573641Z return mod(**inputs) 2025-09-07T07:09:43.5573886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5573955Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5574216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5574295Z layer_outputs = layer_module( 2025-09-07T07:09:43.5574514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5574602Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5574840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5574926Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5575162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5575243Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5575494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5575654Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5575658Z 2025-09-07T07:09:43.5575788Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5575998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5576065Z return mod(**inputs) 2025-09-07T07:09:43.5576315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5576388Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5576644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5576716Z layer_outputs = layer_module( 2025-09-07T07:09:43.5576946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5577026Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5577275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5577366Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5577602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5577688Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5577940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5578090Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5578094Z 2025-09-07T07:09:43.5578204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5578398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5578471Z return mod(**inputs) 2025-09-07T07:09:43.5578712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5578784Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5579045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5579117Z layer_outputs = layer_module( 2025-09-07T07:09:43.5579342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5579423Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5579671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5579754Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5579996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5580084Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5580351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5580516Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5580519Z 2025-09-07T07:09:43.5580624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5580828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5580903Z return mod(**inputs) 2025-09-07T07:09:43.5581150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5581228Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5581482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5581561Z layer_outputs = layer_module( 2025-09-07T07:09:43.5581788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5581887Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5582140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5582222Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5582475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5582561Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5582819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5582910Z value_states = self.v(current_states) 2025-09-07T07:09:43.5582914Z 2025-09-07T07:09:43.5583020Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5583241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5583314Z return mod(**inputs) 2025-09-07T07:09:43.5583583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5583661Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5583919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5584022Z layer_outputs = layer_module( 2025-09-07T07:09:43.5584263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5584354Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5584616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5584702Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5584970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5585056Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5585341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5585459Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5585464Z 2025-09-07T07:09:43.5585572Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5585874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5585949Z return mod(**inputs) 2025-09-07T07:09:43.5586221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5586298Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5586585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5586664Z layer_outputs = layer_module( 2025-09-07T07:09:43.5586903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5586994Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5587252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5587348Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5587614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5587694Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5587939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5588049Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5588052Z 2025-09-07T07:09:43.5588182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5588381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5588456Z return mod(**inputs) 2025-09-07T07:09:43.5588696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5588769Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5589015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5589085Z layer_outputs = layer_module( 2025-09-07T07:09:43.5589310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5589387Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5589625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5589715Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5589955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5590044Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5590280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5590404Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5590416Z 2025-09-07T07:09:43.5590516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5590712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5590785Z return mod(**inputs) 2025-09-07T07:09:43.5591023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5591099Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5591365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5591438Z layer_outputs = layer_module( 2025-09-07T07:09:43.5591663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5591743Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5591987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5592067Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5592303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5592388Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5592638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5592724Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5592728Z 2025-09-07T07:09:43.5592829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5593034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5593100Z return mod(**inputs) 2025-09-07T07:09:43.5593339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5593418Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5593664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5593742Z layer_outputs = layer_module( 2025-09-07T07:09:43.5593966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5594046Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5594323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5594404Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5594650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-09-07T07:09:43.5594788Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:09:43.5594791Z 2025-09-07T07:09:43.5594873Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5594982Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5595182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5595253Z return mod(**inputs) 2025-09-07T07:09:43.5595506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5595584Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5595827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5595896Z layer_outputs = layer_module( 2025-09-07T07:09:43.5596120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5596216Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5596457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5596549Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5596794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5596901Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5597146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5597234Z return self.weight * hidden_states 2025-09-07T07:09:43.5597253Z 2025-09-07T07:09:43.5597360Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5597561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5597636Z return mod(**inputs) 2025-09-07T07:09:43.5597881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5597959Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5598212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5598291Z layer_outputs = layer_module( 2025-09-07T07:09:43.5598556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5598640Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5598892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5598982Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5599235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5599353Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5599598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5599704Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5599708Z 2025-09-07T07:09:43.5599811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5600021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5600105Z return mod(**inputs) 2025-09-07T07:09:43.5600360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5600433Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5600679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5600758Z layer_outputs = layer_module( 2025-09-07T07:09:43.5600981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5601065Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5601307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5601397Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5601649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5601766Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5602018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5602100Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5602119Z 2025-09-07T07:09:43.5602224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5602433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5602498Z return mod(**inputs) 2025-09-07T07:09:43.5602750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5602822Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5603076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5603149Z layer_outputs = layer_module( 2025-09-07T07:09:43.5603396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5603488Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5603732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5603832Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5604077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5604192Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5604445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5604550Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5604555Z 2025-09-07T07:09:43.5604670Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5604874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5604948Z return mod(**inputs) 2025-09-07T07:09:43.5605197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5605273Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5605526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5605598Z layer_outputs = layer_module( 2025-09-07T07:09:43.5605830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5605910Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5606155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5606269Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5606512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5606634Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5606878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5606957Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5606968Z 2025-09-07T07:09:43.5607050Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5607154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5607361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5607427Z return mod(**inputs) 2025-09-07T07:09:43.5607681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5607756Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5608001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5608080Z layer_outputs = layer_module( 2025-09-07T07:09:43.5608321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5608407Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5608650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5608731Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5608980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5609087Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5609336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5609428Z return self.weight * hidden_states 2025-09-07T07:09:43.5609432Z 2025-09-07T07:09:43.5609543Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5609752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5609817Z return mod(**inputs) 2025-09-07T07:09:43.5610063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5610133Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5610385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5610454Z layer_outputs = layer_module( 2025-09-07T07:09:43.5611104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5611193Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5611431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5611520Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5611756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5611838Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5612083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5612159Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5612162Z 2025-09-07T07:09:43.5612272Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5612468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5612560Z return mod(**inputs) 2025-09-07T07:09:43.5612803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5612875Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5613125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5613197Z layer_outputs = layer_module( 2025-09-07T07:09:43.5613423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5613500Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5613739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5613827Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5614065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5614154Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5614391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5614465Z key_states = self.k(current_states) 2025-09-07T07:09:43.5614493Z 2025-09-07T07:09:43.5614595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5614791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5614862Z return mod(**inputs) 2025-09-07T07:09:43.5615102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5615178Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5615419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5615491Z layer_outputs = layer_module( 2025-09-07T07:09:43.5615734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5615812Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5616056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5616136Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5616371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5616458Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5616693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5616828Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5616849Z 2025-09-07T07:09:43.5616951Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5617158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5617222Z return mod(**inputs) 2025-09-07T07:09:43.5617462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5617540Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5617780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5617856Z layer_outputs = layer_module( 2025-09-07T07:09:43.5618079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5618158Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5618403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5618501Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5618751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5618834Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5619076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5619251Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5619254Z 2025-09-07T07:09:43.5619355Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5619709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5619786Z return mod(**inputs) 2025-09-07T07:09:43.5620046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5620123Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5620369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5620448Z layer_outputs = layer_module( 2025-09-07T07:09:43.5620672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5620813Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5621055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5621135Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5621384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5621466Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5621717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5621901Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5621906Z 2025-09-07T07:09:43.5622019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5622221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5622293Z return mod(**inputs) 2025-09-07T07:09:43.5622564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5622641Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5622916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5622991Z layer_outputs = layer_module( 2025-09-07T07:09:43.5623257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5623351Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5623616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5623709Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5623964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5624052Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5624313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5624473Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5624477Z 2025-09-07T07:09:43.5624593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5624808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5624911Z return mod(**inputs) 2025-09-07T07:09:43.5625175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5625254Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5625527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5625796Z layer_outputs = layer_module( 2025-09-07T07:09:43.5626056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5626142Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5626407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5626503Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5626776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5626868Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5627118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5627204Z value_states = self.v(current_states) 2025-09-07T07:09:43.5627246Z 2025-09-07T07:09:43.5627350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5627546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5627622Z return mod(**inputs) 2025-09-07T07:09:43.5627868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5627952Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5628216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5628293Z layer_outputs = layer_module( 2025-09-07T07:09:43.5628558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5628646Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5628911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5628998Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5629254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5629350Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5629610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5629736Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5629757Z 2025-09-07T07:09:43.5629868Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5630090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5630160Z return mod(**inputs) 2025-09-07T07:09:43.5630420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5630504Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5630764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5630847Z layer_outputs = layer_module( 2025-09-07T07:09:43.5631083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5631167Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5631438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5631544Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5631808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5631894Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5632156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5632273Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5632277Z 2025-09-07T07:09:43.5632384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5632602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5632671Z return mod(**inputs) 2025-09-07T07:09:43.5632941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5633017Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5633282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5633364Z layer_outputs = layer_module( 2025-09-07T07:09:43.5633603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5633712Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5633967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5634052Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5634313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5634399Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5634663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5634779Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5634783Z 2025-09-07T07:09:43.5634913Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5635128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5635201Z return mod(**inputs) 2025-09-07T07:09:43.5635474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5635551Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5635818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5635901Z layer_outputs = layer_module( 2025-09-07T07:09:43.5636142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5636232Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5636475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5636562Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5636805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5636896Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5637137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5637216Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5637219Z 2025-09-07T07:09:43.5637311Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5637414Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5637624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5637710Z return mod(**inputs) 2025-09-07T07:09:43.5637960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5638042Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5638286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5638365Z layer_outputs = layer_module( 2025-09-07T07:09:43.5638589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5638668Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5638918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5639011Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5639265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5639365Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5639614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5639694Z return self.weight * hidden_states 2025-09-07T07:09:43.5639715Z 2025-09-07T07:09:43.5639819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5640030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5640096Z return mod(**inputs) 2025-09-07T07:09:43.5640349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5640422Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5640668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5640751Z layer_outputs = layer_module( 2025-09-07T07:09:43.5640993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5641083Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5641324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5641419Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5641667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5641784Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5642034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5642150Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5642154Z 2025-09-07T07:09:43.5642269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5642472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5642538Z return mod(**inputs) 2025-09-07T07:09:43.5642789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5642864Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5643120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5643192Z layer_outputs = layer_module( 2025-09-07T07:09:43.5643417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5643504Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5643752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5643872Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5644118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5644241Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5644488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5644568Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5644571Z 2025-09-07T07:09:43.5644682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5644885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5644958Z return mod(**inputs) 2025-09-07T07:09:43.5645214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5645286Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5645543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5645615Z layer_outputs = layer_module( 2025-09-07T07:09:43.5645844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5645944Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5646186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5646281Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5646524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5646646Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5646890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5646985Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5647010Z 2025-09-07T07:09:43.5647116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5647316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5647391Z return mod(**inputs) 2025-09-07T07:09:43.5647639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5647718Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5647965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5648037Z layer_outputs = layer_module( 2025-09-07T07:09:43.5648286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5648369Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5648620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5648708Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5648963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5649085Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5649342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5649435Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5649439Z 2025-09-07T07:09:43.5649525Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5649642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5649857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5649955Z return mod(**inputs) 2025-09-07T07:09:43.5650229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5650305Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5650573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5650648Z layer_outputs = layer_module( 2025-09-07T07:09:43.5650889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5650979Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5651222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5651314Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5651556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5651677Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5651921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5652018Z return self.weight * hidden_states 2025-09-07T07:09:43.5652021Z 2025-09-07T07:09:43.5652135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5652341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5652415Z return mod(**inputs) 2025-09-07T07:09:43.5652666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5652737Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5652996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5653070Z layer_outputs = layer_module( 2025-09-07T07:09:43.5653324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5653405Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5653659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5653741Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5653982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5654077Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5654325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5654427Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5654431Z 2025-09-07T07:09:43.5654539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5654752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5654826Z return mod(**inputs) 2025-09-07T07:09:43.5655072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5655151Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5655389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5655460Z layer_outputs = layer_module( 2025-09-07T07:09:43.5655685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5655763Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5656006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5656128Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5656370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5656451Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5656683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5656767Z key_states = self.k(current_states) 2025-09-07T07:09:43.5656771Z 2025-09-07T07:09:43.5656872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5657072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5657135Z return mod(**inputs) 2025-09-07T07:09:43.5657371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5657451Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5657690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5657768Z layer_outputs = layer_module( 2025-09-07T07:09:43.5657984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5658079Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5658321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5658399Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5658641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5658720Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5658960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5659092Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5659113Z 2025-09-07T07:09:43.5659215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5659416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5659483Z return mod(**inputs) 2025-09-07T07:09:43.5659741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5659812Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5660054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5660132Z layer_outputs = layer_module( 2025-09-07T07:09:43.5660371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5660460Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5660703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5660790Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5661033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5661116Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5661369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5661523Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5661526Z 2025-09-07T07:09:43.5661636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5661835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5661920Z return mod(**inputs) 2025-09-07T07:09:43.5662180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5662254Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5662513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5662587Z layer_outputs = layer_module( 2025-09-07T07:09:43.5662836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5662920Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5663181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5663277Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5663542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5663639Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5663901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5664068Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5664089Z 2025-09-07T07:09:43.5664208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5664423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5664499Z return mod(**inputs) 2025-09-07T07:09:43.5664763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5664839Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5665111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5665188Z layer_outputs = layer_module( 2025-09-07T07:09:43.5665453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5665538Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5665886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5665986Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5666251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5666349Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5666620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5666814Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5666820Z 2025-09-07T07:09:43.5666933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5667149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5667230Z return mod(**inputs) 2025-09-07T07:09:43.5667493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5667583Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5667843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5667929Z layer_outputs = layer_module( 2025-09-07T07:09:43.5668170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5668257Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5668526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5668632Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5668898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5668985Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5669242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5669332Z value_states = self.v(current_states) 2025-09-07T07:09:43.5669335Z 2025-09-07T07:09:43.5669444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5669663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5669734Z return mod(**inputs) 2025-09-07T07:09:43.5669993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5670081Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5670341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5670423Z layer_outputs = layer_module( 2025-09-07T07:09:43.5670660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5670773Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5671032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5671119Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5671385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5671472Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5671747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5671871Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5671895Z 2025-09-07T07:09:43.5672012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5672237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5672311Z return mod(**inputs) 2025-09-07T07:09:43.5672587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5672664Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5672938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5673015Z layer_outputs = layer_module( 2025-09-07T07:09:43.5673285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5673385Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5673649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5673745Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5674008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5674098Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5674369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5674487Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5674490Z 2025-09-07T07:09:43.5674612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5674834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5674908Z return mod(**inputs) 2025-09-07T07:09:43.5675202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5675281Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5675554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5675633Z layer_outputs = layer_module( 2025-09-07T07:09:43.5675882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5675969Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5676241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5676333Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5676592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5676686Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5676944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5677058Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5677081Z 2025-09-07T07:09:43.5677198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5677409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5677485Z return mod(**inputs) 2025-09-07T07:09:43.5677749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5677833Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5678096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5678175Z layer_outputs = layer_module( 2025-09-07T07:09:43.5678447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5678537Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5678809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5678900Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5679164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5679261Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5679535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5679623Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5679627Z 2025-09-07T07:09:43.5679752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5679972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5680051Z return mod(**inputs) 2025-09-07T07:09:43.5680317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5680402Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5680669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5680760Z layer_outputs = layer_module( 2025-09-07T07:09:43.5680981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5681060Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5681309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5681389Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5681651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-09-07T07:09:43.5681785Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:09:43.5681788Z 2025-09-07T07:09:43.5681871Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5681993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5682205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5682283Z return mod(**inputs) 2025-09-07T07:09:43.5682543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5682618Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5682894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5682970Z layer_outputs = layer_module( 2025-09-07T07:09:43.5683215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5683301Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5683573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5683692Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5683963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5684075Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5684330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5684420Z return self.weight * hidden_states 2025-09-07T07:09:43.5684425Z 2025-09-07T07:09:43.5684535Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5684748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5684840Z return mod(**inputs) 2025-09-07T07:09:43.5685103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5685188Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5685449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5685531Z layer_outputs = layer_module( 2025-09-07T07:09:43.5685770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5685855Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5686141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5686240Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5686520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5686647Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5686915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5687029Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5687033Z 2025-09-07T07:09:43.5687140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5687365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5687434Z return mod(**inputs) 2025-09-07T07:09:43.5687707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5687790Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5688082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5688165Z layer_outputs = layer_module( 2025-09-07T07:09:43.5688399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5688494Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5688761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5688857Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5689137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5689260Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5689529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5689616Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5689619Z 2025-09-07T07:09:43.5689729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5689947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5690036Z return mod(**inputs) 2025-09-07T07:09:43.5690309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5690386Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5690651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5690727Z layer_outputs = layer_module( 2025-09-07T07:09:43.5690968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5691062Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5691382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5691486Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5691756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5691879Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5692154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5692246Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5692249Z 2025-09-07T07:09:43.5692366Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5692611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5692690Z return mod(**inputs) 2025-09-07T07:09:43.5692960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5693040Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5693320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5693398Z layer_outputs = layer_module( 2025-09-07T07:09:43.5693656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5693740Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5694003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5694105Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5694374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5694525Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5694796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5694883Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5694895Z 2025-09-07T07:09:43.5694983Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5695093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5695322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5695394Z return mod(**inputs) 2025-09-07T07:09:43.5695665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-09-07T07:09:43.5695741Z encoder_outputs = self.encoder( 2025-09-07T07:09:43.5696006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-09-07T07:09:43.5696129Z hidden_states = self.final_layer_norm(hidden_states) 2025-09-07T07:09:43.5696389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5696478Z return self.weight * hidden_states 2025-09-07T07:09:43.5696501Z 2025-09-07T07:09:43.5696610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5696823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5696901Z return mod(**inputs) 2025-09-07T07:09:43.5697160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5697243Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5697509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5697586Z layer_outputs = layer_module( 2025-09-07T07:09:43.5697862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5697950Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5698216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5698305Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5698574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5698668Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5698933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5699027Z key_states = self.k(current_states) 2025-09-07T07:09:43.5699049Z 2025-09-07T07:09:43.5699165Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5699393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5699466Z return mod(**inputs) 2025-09-07T07:09:43.5699734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5699825Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5700093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5700179Z layer_outputs = layer_module( 2025-09-07T07:09:43.5700424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5700517Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5700780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5700890Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5701164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5701256Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5701530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5701678Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5701682Z 2025-09-07T07:09:43.5701793Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5702021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5702091Z return mod(**inputs) 2025-09-07T07:09:43.5702373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5702452Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5702723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5702810Z layer_outputs = layer_module( 2025-09-07T07:09:43.5703053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5703168Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5703438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5703535Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5703805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5703897Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5704171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5704343Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5704347Z 2025-09-07T07:09:43.5704482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5704703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5704776Z return mod(**inputs) 2025-09-07T07:09:43.5705056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5705135Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5705419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5705497Z layer_outputs = layer_module( 2025-09-07T07:09:43.5705859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5705956Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5706226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5706323Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5706590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5706704Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5706961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5707046Z value_states = self.v(current_states) 2025-09-07T07:09:43.5707050Z 2025-09-07T07:09:43.5707170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5707384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5707465Z return mod(**inputs) 2025-09-07T07:09:43.5707745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5707824Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5708093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5708172Z layer_outputs = layer_module( 2025-09-07T07:09:43.5708419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5708502Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5708766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5708852Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5709109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5709208Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5709464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5709587Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5709591Z 2025-09-07T07:09:43.5709725Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5709941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5710019Z return mod(**inputs) 2025-09-07T07:09:43.5710281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5710366Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5710630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5710715Z layer_outputs = layer_module( 2025-09-07T07:09:43.5710958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5711063Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5711331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5711419Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5711681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5711771Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5712026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5712149Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5712153Z 2025-09-07T07:09:43.5712279Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5712505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5712576Z return mod(**inputs) 2025-09-07T07:09:43.5712838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5712926Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5713192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5713276Z layer_outputs = layer_module( 2025-09-07T07:09:43.5713518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5713610Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5713874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5713962Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5714256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5714346Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5714623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5714743Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5714747Z 2025-09-07T07:09:43.5714870Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5715094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5715163Z return mod(**inputs) 2025-09-07T07:09:43.5715435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5715513Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5715784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5715861Z layer_outputs = layer_module( 2025-09-07T07:09:43.5716103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5716215Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5716475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5716566Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5716823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5716910Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5717178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5717261Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5717265Z 2025-09-07T07:09:43.5717359Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5717485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5717703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5717781Z return mod(**inputs) 2025-09-07T07:09:43.5718045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5718129Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5718391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5718475Z layer_outputs = layer_module( 2025-09-07T07:09:43.5718736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5718824Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5719095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5719191Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5719460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5719756Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5720023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5720117Z return self.weight * hidden_states 2025-09-07T07:09:43.5720121Z 2025-09-07T07:09:43.5720232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5720458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5720528Z return mod(**inputs) 2025-09-07T07:09:43.5720845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5720933Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5721192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5721278Z layer_outputs = layer_module( 2025-09-07T07:09:43.5721517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5721606Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5721860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5721954Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5722220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5722346Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5722608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5722712Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5722758Z 2025-09-07T07:09:43.5722868Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5723091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5723159Z return mod(**inputs) 2025-09-07T07:09:43.5723427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5723514Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5723771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5723842Z layer_outputs = layer_module( 2025-09-07T07:09:43.5724101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5724191Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5724435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5724532Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5724777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5724894Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5725147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5725230Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5725258Z 2025-09-07T07:09:43.5725372Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5725576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5725647Z return mod(**inputs) 2025-09-07T07:09:43.5725901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5725975Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5726228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5726302Z layer_outputs = layer_module( 2025-09-07T07:09:43.5726536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5726616Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5726860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5726974Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5727219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5727344Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5727586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5727678Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5727688Z 2025-09-07T07:09:43.5727792Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5727996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5728070Z return mod(**inputs) 2025-09-07T07:09:43.5728321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5728399Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5728646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5728717Z layer_outputs = layer_module( 2025-09-07T07:09:43.5728949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5729047Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5729303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5729391Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5729639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5729762Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5730014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5730104Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5730107Z 2025-09-07T07:09:43.5730227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5730437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5730505Z return mod(**inputs) 2025-09-07T07:09:43.5730750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5730832Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5731078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5731159Z layer_outputs = layer_module( 2025-09-07T07:09:43.5731402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5731486Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5731742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5731825Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5732076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5732186Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5732428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5732514Z return self.weight * hidden_states 2025-09-07T07:09:43.5732518Z 2025-09-07T07:09:43.5732620Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5732830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5732896Z return mod(**inputs) 2025-09-07T07:09:43.5733169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5733243Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5733488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5733571Z layer_outputs = layer_module( 2025-09-07T07:09:43.5733794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5733882Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5734124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5734206Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5734456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5734542Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5734794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5734871Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5734875Z 2025-09-07T07:09:43.5734983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5735206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5735274Z return mod(**inputs) 2025-09-07T07:09:43.5735529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5735601Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5735857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5735929Z layer_outputs = layer_module( 2025-09-07T07:09:43.5736155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5736258Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5736509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5736599Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5736857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5736940Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5737192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5737267Z key_states = self.k(current_states) 2025-09-07T07:09:43.5737271Z 2025-09-07T07:09:43.5737394Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5737590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5737662Z return mod(**inputs) 2025-09-07T07:09:43.5737904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5737974Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5738222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5738291Z layer_outputs = layer_module( 2025-09-07T07:09:43.5738520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5738598Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5738837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5738928Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5739191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5739283Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5739526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5739661Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5739671Z 2025-09-07T07:09:43.5739776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5739975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5740048Z return mod(**inputs) 2025-09-07T07:09:43.5740292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5740370Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5740614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5740686Z layer_outputs = layer_module( 2025-09-07T07:09:43.5740919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5741000Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5741269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5741352Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5741594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5741684Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5741929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5742097Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5742102Z 2025-09-07T07:09:43.5742222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5742433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5742500Z return mod(**inputs) 2025-09-07T07:09:43.5742755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5742834Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5743084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5743163Z layer_outputs = layer_module( 2025-09-07T07:09:43.5743404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5743504Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5743781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5743868Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5744130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5744220Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5744476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5744567Z value_states = self.v(current_states) 2025-09-07T07:09:43.5744570Z 2025-09-07T07:09:43.5744679Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5744902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5744971Z return mod(**inputs) 2025-09-07T07:09:43.5745237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5745340Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5745599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5745750Z layer_outputs = layer_module( 2025-09-07T07:09:43.5746000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5746093Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5746356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5746443Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5746712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5746803Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5747081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5747198Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5747202Z 2025-09-07T07:09:43.5747313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5747542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5747620Z return mod(**inputs) 2025-09-07T07:09:43.5747875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5747946Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5748196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5748267Z layer_outputs = layer_module( 2025-09-07T07:09:43.5748489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5748577Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5748837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5748928Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5749165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5749245Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5749546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5749654Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5749658Z 2025-09-07T07:09:43.5749766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5749981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5750057Z return mod(**inputs) 2025-09-07T07:09:43.5750305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5750375Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5750624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5750696Z layer_outputs = layer_module( 2025-09-07T07:09:43.5750926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5751002Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5751245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5751333Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5751571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5751681Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5751918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5752032Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5752035Z 2025-09-07T07:09:43.5752135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5752330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5752402Z return mod(**inputs) 2025-09-07T07:09:43.5752643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5752720Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5753018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5753093Z layer_outputs = layer_module( 2025-09-07T07:09:43.5753329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5753411Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5753678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5753761Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5754004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5754095Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5754339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5754424Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5754428Z 2025-09-07T07:09:43.5754513Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5754624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5754848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5754916Z return mod(**inputs) 2025-09-07T07:09:43.5755177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5755248Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5755498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5755569Z layer_outputs = layer_module( 2025-09-07T07:09:43.5755798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5755926Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5756171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5756266Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5756509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-09-07T07:09:43.5756620Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5756873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5756950Z return self.weight * hidden_states 2025-09-07T07:09:43.5756953Z 2025-09-07T07:09:43.5757069Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5757314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5757388Z return mod(**inputs) 2025-09-07T07:09:43.5757637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5757729Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5757986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5758058Z layer_outputs = layer_module( 2025-09-07T07:09:43.5758290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5758373Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5758618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5758709Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5758953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5759048Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5759293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5759375Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5759385Z 2025-09-07T07:09:43.5759489Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5759691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5759783Z return mod(**inputs) 2025-09-07T07:09:43.5760030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5760110Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5760356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5760428Z layer_outputs = layer_module( 2025-09-07T07:09:43.5760664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5760745Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5761011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5761094Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5761339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5761431Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5761675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5761760Z key_states = self.k(current_states) 2025-09-07T07:09:43.5761764Z 2025-09-07T07:09:43.5761868Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5762094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5762163Z return mod(**inputs) 2025-09-07T07:09:43.5762414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5762497Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5762745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5762825Z layer_outputs = layer_module( 2025-09-07T07:09:43.5763051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5763131Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5763384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5763466Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5763715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5763820Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5764077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5764226Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5764231Z 2025-09-07T07:09:43.5764339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5764562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5764626Z return mod(**inputs) 2025-09-07T07:09:43.5764878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5764949Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5765196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5765277Z layer_outputs = layer_module( 2025-09-07T07:09:43.5765501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5765587Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5765830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5765934Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5766184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5766269Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5766521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5766680Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5766685Z 2025-09-07T07:09:43.5766795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5767009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5767076Z return mod(**inputs) 2025-09-07T07:09:43.5767332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5767417Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5767665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5767735Z layer_outputs = layer_module( 2025-09-07T07:09:43.5767952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5768036Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5768296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5768388Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5768624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5768709Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5768961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5769039Z value_states = self.v(current_states) 2025-09-07T07:09:43.5769043Z 2025-09-07T07:09:43.5769153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5769353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5769425Z return mod(**inputs) 2025-09-07T07:09:43.5769672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5769765Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5770019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5770090Z layer_outputs = layer_module( 2025-09-07T07:09:43.5770326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5770408Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5770650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5770741Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5770981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5771082Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5771320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5771436Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5771441Z 2025-09-07T07:09:43.5771541Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5771734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5771825Z return mod(**inputs) 2025-09-07T07:09:43.5772069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5772147Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5772390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5772459Z layer_outputs = layer_module( 2025-09-07T07:09:43.5772692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5772769Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5773038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5773121Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5773360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5773453Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5773698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5773814Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5773817Z 2025-09-07T07:09:43.5773920Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5774144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5774214Z return mod(**inputs) 2025-09-07T07:09:43.5774465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5774546Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5774798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5774877Z layer_outputs = layer_module( 2025-09-07T07:09:43.5775094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5775171Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5775423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5775505Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5775755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5775856Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5776102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5776210Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5776214Z 2025-09-07T07:09:43.5776317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5776528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5776594Z return mod(**inputs) 2025-09-07T07:09:43.5776848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5776921Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5777169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5777250Z layer_outputs = layer_module( 2025-09-07T07:09:43.5777472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5777557Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5777808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5777918Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5778179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5778265Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5778525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5778607Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5778611Z 2025-09-07T07:09:43.5778706Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5778818Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5779051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5779131Z return mod(**inputs) 2025-09-07T07:09:43.5779392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5779478Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5779737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5779814Z layer_outputs = layer_module( 2025-09-07T07:09:43.5780059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5780141Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5780424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5780524Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5780779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5780889Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5781147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5781237Z return self.weight * hidden_states 2025-09-07T07:09:43.5781241Z 2025-09-07T07:09:43.5781349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5781570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5781640Z return mod(**inputs) 2025-09-07T07:09:43.5781903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5782006Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5782268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5782353Z layer_outputs = layer_module( 2025-09-07T07:09:43.5782592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5782678Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5782946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5783042Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5783308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5783436Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5783703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5783813Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5783817Z 2025-09-07T07:09:43.5783926Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5784149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5784238Z return mod(**inputs) 2025-09-07T07:09:43.5784510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5784588Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5784847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5784934Z layer_outputs = layer_module( 2025-09-07T07:09:43.5785175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5785268Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5785542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5785714Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5785991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5786116Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5786382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5786471Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5786475Z 2025-09-07T07:09:43.5786598Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5786845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5786924Z return mod(**inputs) 2025-09-07T07:09:43.5787204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5787287Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5787576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5787654Z layer_outputs = layer_module( 2025-09-07T07:09:43.5787896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5787991Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5788248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5788353Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5788615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5788765Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5789024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5789118Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5789123Z 2025-09-07T07:09:43.5789244Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5789458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5789534Z return mod(**inputs) 2025-09-07T07:09:43.5789797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5789874Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5790142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5790222Z layer_outputs = layer_module( 2025-09-07T07:09:43.5790468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5790550Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5790809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5790932Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5791189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5791317Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5791573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5791669Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5791674Z 2025-09-07T07:09:43.5791761Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5791872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5792122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5792194Z return mod(**inputs) 2025-09-07T07:09:43.5792469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5792545Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5792813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5792897Z layer_outputs = layer_module( 2025-09-07T07:09:43.5793137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5793250Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5793511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5793607Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5793863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5793978Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5794242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5794325Z return self.weight * hidden_states 2025-09-07T07:09:43.5794329Z 2025-09-07T07:09:43.5794446Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5794659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5794728Z return mod(**inputs) 2025-09-07T07:09:43.5794995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5795090Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5795364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5795438Z layer_outputs = layer_module( 2025-09-07T07:09:43.5795679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5795771Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5796032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5796124Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5796383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5796481Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5796743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5796827Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5796831Z 2025-09-07T07:09:43.5796949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5797166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5797270Z return mod(**inputs) 2025-09-07T07:09:43.5797531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5797608Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5797878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5797956Z layer_outputs = layer_module( 2025-09-07T07:09:43.5798213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5798295Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5798555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5798647Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5798894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5798991Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5799248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5799338Z key_states = self.k(current_states) 2025-09-07T07:09:43.5799341Z 2025-09-07T07:09:43.5799451Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5799690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5799774Z return mod(**inputs) 2025-09-07T07:09:43.5800037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5800121Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5800384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5800463Z layer_outputs = layer_module( 2025-09-07T07:09:43.5800709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5800794Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5801057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5801145Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5801411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5801516Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5801775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5801924Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5801929Z 2025-09-07T07:09:43.5802039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5802260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5802328Z return mod(**inputs) 2025-09-07T07:09:43.5802589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5802673Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5802930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5803015Z layer_outputs = layer_module( 2025-09-07T07:09:43.5803258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5803345Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5803609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5803716Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5803980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5804067Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5804331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5804499Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5804505Z 2025-09-07T07:09:43.5804616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5804855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5804925Z return mod(**inputs) 2025-09-07T07:09:43.5805195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5805274Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5805535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5805620Z layer_outputs = layer_module( 2025-09-07T07:09:43.5805855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5805947Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5806219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5806315Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5806575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5806663Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5806937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5807022Z value_states = self.v(current_states) 2025-09-07T07:09:43.5807025Z 2025-09-07T07:09:43.5807151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5807357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5807423Z return mod(**inputs) 2025-09-07T07:09:43.5807682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5807773Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5808033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5808105Z layer_outputs = layer_module( 2025-09-07T07:09:43.5808335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5808429Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5808689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5808780Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5809041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5809133Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5809395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5809514Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5809519Z 2025-09-07T07:09:43.5809636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5809854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5809961Z return mod(**inputs) 2025-09-07T07:09:43.5810206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5810280Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5810530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5810600Z layer_outputs = layer_module( 2025-09-07T07:09:43.5810844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5810929Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5811215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5811303Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5811567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5811663Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5811926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5812046Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5812050Z 2025-09-07T07:09:43.5812159Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5812389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5812469Z return mod(**inputs) 2025-09-07T07:09:43.5812732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5812816Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5813076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5813154Z layer_outputs = layer_module( 2025-09-07T07:09:43.5813407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5813493Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5813757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5813844Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5814111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5814217Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5814475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5814598Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5814603Z 2025-09-07T07:09:43.5814712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5814933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5815002Z return mod(**inputs) 2025-09-07T07:09:43.5815261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5815346Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5815607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5815690Z layer_outputs = layer_module( 2025-09-07T07:09:43.5815932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5816025Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5816282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5816398Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5816663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5816750Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5817012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5817094Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5817098Z 2025-09-07T07:09:43.5817209Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5817435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5817519Z return mod(**inputs) 2025-09-07T07:09:43.5817796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5817875Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5818142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5818226Z layer_outputs = layer_module( 2025-09-07T07:09:43.5818467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5818560Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5818842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5818938Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5819198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-09-07T07:09:43.5819343Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:09:43.5819347Z 2025-09-07T07:09:43.5819442Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5819689Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5819924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5819996Z return mod(**inputs) 2025-09-07T07:09:43.5820266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5820355Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5820628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5820761Z layer_outputs = layer_module( 2025-09-07T07:09:43.5821013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5821100Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5821389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5821477Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5821746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-09-07T07:09:43.5821862Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5822133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5822217Z return self.weight * hidden_states 2025-09-07T07:09:43.5822222Z 2025-09-07T07:09:43.5822333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5822567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5822638Z return mod(**inputs) 2025-09-07T07:09:43.5822914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5823020Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5823281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5823365Z layer_outputs = layer_module( 2025-09-07T07:09:43.5823602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5823694Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5823952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5824046Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5824327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5824419Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5824682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5824766Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5824770Z 2025-09-07T07:09:43.5824886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5825099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5825166Z return mod(**inputs) 2025-09-07T07:09:43.5825440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5825541Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5825871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5825958Z layer_outputs = layer_module( 2025-09-07T07:09:43.5826203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5826297Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5826564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5826660Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5826935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5827031Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5827290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5827392Z key_states = self.k(current_states) 2025-09-07T07:09:43.5827396Z 2025-09-07T07:09:43.5827515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5827730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5827809Z return mod(**inputs) 2025-09-07T07:09:43.5828073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5828149Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5828418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5828495Z layer_outputs = layer_module( 2025-09-07T07:09:43.5828740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5828826Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5829094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5829183Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5829443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5829560Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5829815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5829962Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5829965Z 2025-09-07T07:09:43.5830075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5830286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5830365Z return mod(**inputs) 2025-09-07T07:09:43.5830626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5830735Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5830998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5831077Z layer_outputs = layer_module( 2025-09-07T07:09:43.5831324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5831407Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5831676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5831773Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5832034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5832118Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5832360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5832522Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5832525Z 2025-09-07T07:09:43.5832629Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5832833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5832898Z return mod(**inputs) 2025-09-07T07:09:43.5833142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5833219Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5833459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5833539Z layer_outputs = layer_module( 2025-09-07T07:09:43.5833772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5833858Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5834093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5834176Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5834424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5834508Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5834813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5834906Z value_states = self.v(current_states) 2025-09-07T07:09:43.5834910Z 2025-09-07T07:09:43.5835016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5835223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5835290Z return mod(**inputs) 2025-09-07T07:09:43.5835540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5835613Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5835868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5835945Z layer_outputs = layer_module( 2025-09-07T07:09:43.5836161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5836246Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5836483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5836570Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5836808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5836906Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5837153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5837263Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5837266Z 2025-09-07T07:09:43.5837375Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5837579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5837643Z return mod(**inputs) 2025-09-07T07:09:43.5837892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5837982Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5838237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5838311Z layer_outputs = layer_module( 2025-09-07T07:09:43.5838548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5838624Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5838866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5838957Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5839197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5839288Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5839529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5839637Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5839668Z 2025-09-07T07:09:43.5839777Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5839974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5840046Z return mod(**inputs) 2025-09-07T07:09:43.5840287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5840358Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5840606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5840676Z layer_outputs = layer_module( 2025-09-07T07:09:43.5840900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5840988Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5841234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5841316Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5841561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5841670Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5841910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5842021Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5842024Z 2025-09-07T07:09:43.5842123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5842320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5842392Z return mod(**inputs) 2025-09-07T07:09:43.5842639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5842718Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5842978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5843057Z layer_outputs = layer_module( 2025-09-07T07:09:43.5843284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5843362Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5843615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5843695Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5843942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5844043Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5844290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5844378Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5844382Z 2025-09-07T07:09:43.5844465Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5844581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5844793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5844857Z return mod(**inputs) 2025-09-07T07:09:43.5845108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5845179Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5845429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5845500Z layer_outputs = layer_module( 2025-09-07T07:09:43.5845746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5845825Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5846065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5846166Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5846409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5846509Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5846746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5846823Z return self.weight * hidden_states 2025-09-07T07:09:43.5846826Z 2025-09-07T07:09:43.5846934Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5847135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5847208Z return mod(**inputs) 2025-09-07T07:09:43.5847453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5847523Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5847786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5847855Z layer_outputs = layer_module( 2025-09-07T07:09:43.5848080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5848158Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5848402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5848493Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5848746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5848873Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5849109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5849216Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5849219Z 2025-09-07T07:09:43.5849318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5849515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5849588Z return mod(**inputs) 2025-09-07T07:09:43.5849832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5849925Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5850174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5850255Z layer_outputs = layer_module( 2025-09-07T07:09:43.5850484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5850563Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5850801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5850888Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5851126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5851237Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5851470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5851571Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5851574Z 2025-09-07T07:09:43.5851673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5851871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5851938Z return mod(**inputs) 2025-09-07T07:09:43.5852169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5852245Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5852478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5852554Z layer_outputs = layer_module( 2025-09-07T07:09:43.5852765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5852851Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5853087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5853176Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5853420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5853553Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5853804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5853892Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5853896Z 2025-09-07T07:09:43.5853998Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5854207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5854274Z return mod(**inputs) 2025-09-07T07:09:43.5854528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5855564Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5855820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5855894Z layer_outputs = layer_module( 2025-09-07T07:09:43.5856110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5856199Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5856442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5856537Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5856802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5856920Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5857178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5857264Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5857267Z 2025-09-07T07:09:43.5857357Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5857459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5857657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5857731Z return mod(**inputs) 2025-09-07T07:09:43.5857976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5858058Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5858300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5858394Z layer_outputs = layer_module( 2025-09-07T07:09:43.5858624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5858700Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5858939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5859018Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5859261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5859368Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5859610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5859694Z return self.weight * hidden_states 2025-09-07T07:09:43.5859698Z 2025-09-07T07:09:43.5859800Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5860011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5860077Z return mod(**inputs) 2025-09-07T07:09:43.5860327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5860420Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5860668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5860749Z layer_outputs = layer_module( 2025-09-07T07:09:43.5860967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5861055Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5861296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5861377Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5861643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5861732Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5861988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5862070Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5862074Z 2025-09-07T07:09:43.5862180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5862395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5862462Z return mod(**inputs) 2025-09-07T07:09:43.5862718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5862818Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5863088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5863165Z layer_outputs = layer_module( 2025-09-07T07:09:43.5863410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5863504Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5863763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5863857Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5864111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5864202Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5864468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5864568Z key_states = self.k(current_states) 2025-09-07T07:09:43.5864572Z 2025-09-07T07:09:43.5864690Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5864904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5864974Z return mod(**inputs) 2025-09-07T07:09:43.5865245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5865320Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5865591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5865746Z layer_outputs = layer_module( 2025-09-07T07:09:43.5866007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5866098Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5866367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5866467Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5866744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5866863Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5867118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5867262Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5867274Z 2025-09-07T07:09:43.5867388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5867608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5867692Z return mod(**inputs) 2025-09-07T07:09:43.5867963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5868070Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5868341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5868421Z layer_outputs = layer_module( 2025-09-07T07:09:43.5868673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5868760Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5869033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5869122Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5869405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5869503Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5869777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5869957Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5869961Z 2025-09-07T07:09:43.5870075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5870301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5870373Z return mod(**inputs) 2025-09-07T07:09:43.5870640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5870727Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5870996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5871085Z layer_outputs = layer_module( 2025-09-07T07:09:43.5871350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5871436Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5871706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5871796Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5872065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5872154Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5872416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5872507Z value_states = self.v(current_states) 2025-09-07T07:09:43.5872511Z 2025-09-07T07:09:43.5872624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5872847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5872922Z return mod(**inputs) 2025-09-07T07:09:43.5873196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5873276Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5873566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5873652Z layer_outputs = layer_module( 2025-09-07T07:09:43.5873896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5874001Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5874260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5874347Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5874634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5874740Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5875013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5875134Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5875138Z 2025-09-07T07:09:43.5875254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5875487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5875556Z return mod(**inputs) 2025-09-07T07:09:43.5875828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5875921Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5876204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5876282Z layer_outputs = layer_module( 2025-09-07T07:09:43.5876537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5876632Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5876891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5876985Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5877255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5877343Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5877621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5877738Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5877761Z 2025-09-07T07:09:43.5877878Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5878113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5878190Z return mod(**inputs) 2025-09-07T07:09:43.5878453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5878528Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5878795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5878871Z layer_outputs = layer_module( 2025-09-07T07:09:43.5879117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5879204Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5879462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5879561Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5879818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5879935Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5880199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5880312Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5880325Z 2025-09-07T07:09:43.5880432Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5880663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5880741Z return mod(**inputs) 2025-09-07T07:09:43.5881004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5881088Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5881383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5881460Z layer_outputs = layer_module( 2025-09-07T07:09:43.5881709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5881794Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5882066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5882154Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5882417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5882531Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5882791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5882884Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5882888Z 2025-09-07T07:09:43.5882977Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5883094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5883308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5883379Z return mod(**inputs) 2025-09-07T07:09:43.5883656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5883728Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5883980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5884053Z layer_outputs = layer_module( 2025-09-07T07:09:43.5884277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5884389Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5884629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5884723Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5884968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-09-07T07:09:43.5885086Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5885328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5885405Z return self.weight * hidden_states 2025-09-07T07:09:43.5885408Z 2025-09-07T07:09:43.5885517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5885713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5885784Z return mod(**inputs) 2025-09-07T07:09:43.5886026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5886096Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5886358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5886429Z layer_outputs = layer_module( 2025-09-07T07:09:43.5886651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5886727Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5886972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5887061Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5887297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5887412Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5887654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5887732Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5887743Z 2025-09-07T07:09:43.5887844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5888043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5888115Z return mod(**inputs) 2025-09-07T07:09:43.5888360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5888437Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5888697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5888771Z layer_outputs = layer_module( 2025-09-07T07:09:43.5889004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5889083Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5889332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5889415Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5889655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5889748Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5889986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5890072Z key_states = self.k(current_states) 2025-09-07T07:09:43.5890106Z 2025-09-07T07:09:43.5890207Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5890414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5890479Z return mod(**inputs) 2025-09-07T07:09:43.5890718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5890800Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5891039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5891116Z layer_outputs = layer_module( 2025-09-07T07:09:43.5891334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5891411Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5891654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5891735Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5891977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5892060Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5892311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5892447Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5892450Z 2025-09-07T07:09:43.5892551Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5892754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5892818Z return mod(**inputs) 2025-09-07T07:09:43.5893071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5893145Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5893412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5893499Z layer_outputs = layer_module( 2025-09-07T07:09:43.5893735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5893827Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5894084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5894169Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5894433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5894543Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5894812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5894980Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5894984Z 2025-09-07T07:09:43.5895100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5895314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5895383Z return mod(**inputs) 2025-09-07T07:09:43.5895650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5895726Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5895991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5896067Z layer_outputs = layer_module( 2025-09-07T07:09:43.5896305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5896415Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5896682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5896772Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5897018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5897102Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5897351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5897430Z value_states = self.v(current_states) 2025-09-07T07:09:43.5897433Z 2025-09-07T07:09:43.5897544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5897754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5897830Z return mod(**inputs) 2025-09-07T07:09:43.5898076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5898149Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5898404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5898494Z layer_outputs = layer_module( 2025-09-07T07:09:43.5898724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5898801Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5899045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5899134Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5899380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5899475Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5899735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5899844Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5899855Z 2025-09-07T07:09:43.5899958Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5900173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5900252Z return mod(**inputs) 2025-09-07T07:09:43.5900519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5900602Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5900888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5900967Z layer_outputs = layer_module( 2025-09-07T07:09:43.5901214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5901297Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5901565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5901652Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5901915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5902010Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5902270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5902395Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5902414Z 2025-09-07T07:09:43.5902522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5902743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5902813Z return mod(**inputs) 2025-09-07T07:09:43.5903072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5903160Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5903419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5903502Z layer_outputs = layer_module( 2025-09-07T07:09:43.5903740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5903822Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5904088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5904175Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5904438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5904527Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5904801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5904926Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5904930Z 2025-09-07T07:09:43.5905039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5905267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5905338Z return mod(**inputs) 2025-09-07T07:09:43.5905684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5905775Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5906065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5906155Z layer_outputs = layer_module( 2025-09-07T07:09:43.5906402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5906501Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5906770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5906861Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5907141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5907231Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5907519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5907616Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5907622Z 2025-09-07T07:09:43.5907735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5907944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5908013Z return mod(**inputs) 2025-09-07T07:09:43.5908269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5908342Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5908593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5908668Z layer_outputs = layer_module( 2025-09-07T07:09:43.5908898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5909006Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5909263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5909355Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5909610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-09-07T07:09:43.5909755Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:09:43.5909767Z 2025-09-07T07:09:43.5909855Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5909966Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5910185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5910254Z return mod(**inputs) 2025-09-07T07:09:43.5910524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5910603Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5910869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5910954Z layer_outputs = layer_module( 2025-09-07T07:09:43.5911192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5911313Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5911571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5911667Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5911931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5912037Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5912302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5912406Z return self.weight * hidden_states 2025-09-07T07:09:43.5912410Z 2025-09-07T07:09:43.5912529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5912742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5912813Z return mod(**inputs) 2025-09-07T07:09:43.5913080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5913156Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5913425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5913499Z layer_outputs = layer_module( 2025-09-07T07:09:43.5913757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5913853Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5914111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5914215Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5914476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5914601Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5914863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5914969Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5914973Z 2025-09-07T07:09:43.5915087Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5915305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5915400Z return mod(**inputs) 2025-09-07T07:09:43.5915665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5915743Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5916016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5916094Z layer_outputs = layer_module( 2025-09-07T07:09:43.5916343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5916428Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5916688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5916793Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5917056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5917191Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5917454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5917546Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5917568Z 2025-09-07T07:09:43.5917678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5917891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5917969Z return mod(**inputs) 2025-09-07T07:09:43.5918241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5918324Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5918586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5918667Z layer_outputs = layer_module( 2025-09-07T07:09:43.5918926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5919010Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5919273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5919370Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5919754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5919895Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5920156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5920304Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5920310Z 2025-09-07T07:09:43.5920423Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5920648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5920721Z return mod(**inputs) 2025-09-07T07:09:43.5920986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5921072Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5921331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5921415Z layer_outputs = layer_module( 2025-09-07T07:09:43.5921653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5921738Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5922012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5922140Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5922411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5922533Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5922801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5922889Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5922893Z 2025-09-07T07:09:43.5922980Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5923099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5923313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5923390Z return mod(**inputs) 2025-09-07T07:09:43.5923653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5923732Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5923999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5924076Z layer_outputs = layer_module( 2025-09-07T07:09:43.5924358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5924442Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5924699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5924792Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5925049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5925170Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5925456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5925549Z return self.weight * hidden_states 2025-09-07T07:09:43.5925553Z 2025-09-07T07:09:43.5925663Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5925876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5925954Z return mod(**inputs) 2025-09-07T07:09:43.5926213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5926298Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5926559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5926634Z layer_outputs = layer_module( 2025-09-07T07:09:43.5926910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5927000Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5927274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5927358Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5927603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5927698Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5927946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5928033Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5928036Z 2025-09-07T07:09:43.5928140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5928352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5928438Z return mod(**inputs) 2025-09-07T07:09:43.5928686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5928765Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5929013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5929095Z layer_outputs = layer_module( 2025-09-07T07:09:43.5929319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5929397Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5929649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5929732Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5929978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5930064Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5930313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5930390Z key_states = self.k(current_states) 2025-09-07T07:09:43.5930412Z 2025-09-07T07:09:43.5930516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5930725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5930790Z return mod(**inputs) 2025-09-07T07:09:43.5931044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5931115Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5931360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5931440Z layer_outputs = layer_module( 2025-09-07T07:09:43.5931680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5931771Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5932013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5932095Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5932341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5932423Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5932670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5932818Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5932823Z 2025-09-07T07:09:43.5932934Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5933136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5933202Z return mod(**inputs) 2025-09-07T07:09:43.5933455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5933529Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5933781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5933853Z layer_outputs = layer_module( 2025-09-07T07:09:43.5934074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5934162Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5934405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5934520Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5934767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5934857Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5935104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5935260Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5935264Z 2025-09-07T07:09:43.5935378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5935583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5935655Z return mod(**inputs) 2025-09-07T07:09:43.5935908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5935982Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5936241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5936313Z layer_outputs = layer_module( 2025-09-07T07:09:43.5936548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5936648Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5936890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5936980Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5937225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5937317Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5937561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5937666Z value_states = self.v(current_states) 2025-09-07T07:09:43.5937670Z 2025-09-07T07:09:43.5937776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5937988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5938064Z return mod(**inputs) 2025-09-07T07:09:43.5938306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5938383Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5938624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5938693Z layer_outputs = layer_module( 2025-09-07T07:09:43.5938934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5939015Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5939260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5939339Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5939582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5939665Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5939901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5940016Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5940019Z 2025-09-07T07:09:43.5940121Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5940326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5940408Z return mod(**inputs) 2025-09-07T07:09:43.5940653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5940732Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5940972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5941052Z layer_outputs = layer_module( 2025-09-07T07:09:43.5941272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5941350Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5941601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5941681Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5941929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5942012Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5942263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5942375Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5942397Z 2025-09-07T07:09:43.5942502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5942711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5942778Z return mod(**inputs) 2025-09-07T07:09:43.5943034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5943105Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5943352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5943434Z layer_outputs = layer_module( 2025-09-07T07:09:43.5943674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5943766Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5944009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5944103Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5944360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5944448Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5944715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5944828Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5944850Z 2025-09-07T07:09:43.5944967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5945181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5945249Z return mod(**inputs) 2025-09-07T07:09:43.5945518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5945594Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5945942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5946024Z layer_outputs = layer_module( 2025-09-07T07:09:43.5946278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5946374Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5946651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5946770Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5947049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5947147Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5947411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5947488Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5947492Z 2025-09-07T07:09:43.5947582Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5947685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5947888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5947953Z return mod(**inputs) 2025-09-07T07:09:43.5948193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5948276Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5948519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5948596Z layer_outputs = layer_module( 2025-09-07T07:09:43.5948814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5948912Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5949156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5949238Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5949484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-09-07T07:09:43.5949588Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5949839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5949916Z return self.weight * hidden_states 2025-09-07T07:09:43.5949938Z 2025-09-07T07:09:43.5950042Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5950247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5950314Z return mod(**inputs) 2025-09-07T07:09:43.5950561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5950630Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5950865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5950944Z layer_outputs = layer_module( 2025-09-07T07:09:43.5951185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5951272Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5951509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5951596Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5951832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5951916Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5952160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5952236Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5952240Z 2025-09-07T07:09:43.5952348Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5952545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5952610Z return mod(**inputs) 2025-09-07T07:09:43.5952879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5952950Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5953202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5953276Z layer_outputs = layer_module( 2025-09-07T07:09:43.5953504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5953586Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5953816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5953900Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5954130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5954221Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5954456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5954533Z key_states = self.k(current_states) 2025-09-07T07:09:43.5954536Z 2025-09-07T07:09:43.5954686Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5954887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5954959Z return mod(**inputs) 2025-09-07T07:09:43.5955201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5955271Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5955522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5955592Z layer_outputs = layer_module( 2025-09-07T07:09:43.5955820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5955915Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5956161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5956246Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5956475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5956565Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5956800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5956931Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5956934Z 2025-09-07T07:09:43.5957058Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5957255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5957327Z return mod(**inputs) 2025-09-07T07:09:43.5957558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5957636Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5957871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5957938Z layer_outputs = layer_module( 2025-09-07T07:09:43.5958158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5958232Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5958470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5958548Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5958801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5958881Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5959109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5959266Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5959270Z 2025-09-07T07:09:43.5959369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5959564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5959627Z return mod(**inputs) 2025-09-07T07:09:43.5959860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5959935Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5960174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5960253Z layer_outputs = layer_module( 2025-09-07T07:09:43.5960483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5960578Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5960827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5960909Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5961178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5961270Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5961543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.5961639Z value_states = self.v(current_states) 2025-09-07T07:09:43.5961643Z 2025-09-07T07:09:43.5961769Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5961991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5962063Z return mod(**inputs) 2025-09-07T07:09:43.5962329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5962415Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5962658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5962738Z layer_outputs = layer_module( 2025-09-07T07:09:43.5962962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5963073Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5963316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5963405Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5963643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5963728Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5963973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5964079Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5964083Z 2025-09-07T07:09:43.5964191Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5964392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5964464Z return mod(**inputs) 2025-09-07T07:09:43.5964753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5964832Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5965101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5965181Z layer_outputs = layer_module( 2025-09-07T07:09:43.5965417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5965506Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5965763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5965858Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5966117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5966215Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5966472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.5966586Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.5966590Z 2025-09-07T07:09:43.5966707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5966940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5967026Z return mod(**inputs) 2025-09-07T07:09:43.5967259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5967327Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5967571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5967642Z layer_outputs = layer_module( 2025-09-07T07:09:43.5967864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5967952Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5968192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5968274Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5968510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5968599Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5968835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.5968946Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.5968949Z 2025-09-07T07:09:43.5969067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5969267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5969339Z return mod(**inputs) 2025-09-07T07:09:43.5969582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5969660Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5969902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5969972Z layer_outputs = layer_module( 2025-09-07T07:09:43.5970209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5970283Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5970527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.5970607Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.5970857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.5970938Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.5971172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.5971256Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.5971259Z 2025-09-07T07:09:43.5971336Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5971441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5971633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5971695Z return mod(**inputs) 2025-09-07T07:09:43.5971941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5972011Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5972259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5972328Z layer_outputs = layer_module( 2025-09-07T07:09:43.5972545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5972647Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5972882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5972981Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5973215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.5973317Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5973553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5973630Z return self.weight * hidden_states 2025-09-07T07:09:43.5973634Z 2025-09-07T07:09:43.5973756Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5973952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5974025Z return mod(**inputs) 2025-09-07T07:09:43.5974264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5974337Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5974589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5974659Z layer_outputs = layer_module( 2025-09-07T07:09:43.5974913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5974992Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5975238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5975328Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5975563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5975687Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5975922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.5976026Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.5976030Z 2025-09-07T07:09:43.5976131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5976327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5976402Z return mod(**inputs) 2025-09-07T07:09:43.5976661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5976742Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5976984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5977055Z layer_outputs = layer_module( 2025-09-07T07:09:43.5977283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5977360Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5977603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5977692Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5977936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5978054Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5978292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.5978378Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.5978381Z 2025-09-07T07:09:43.5978502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5978707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5978773Z return mod(**inputs) 2025-09-07T07:09:43.5979014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5979093Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5979334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5979411Z layer_outputs = layer_module( 2025-09-07T07:09:43.5979628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5979736Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5979973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5980071Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5980337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5980463Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5980743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.5980840Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.5980844Z 2025-09-07T07:09:43.5980974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5981205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5981278Z return mod(**inputs) 2025-09-07T07:09:43.5981563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5981644Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5981924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5981999Z layer_outputs = layer_module( 2025-09-07T07:09:43.5982240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5982332Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5982597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5982717Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5982977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.5983102Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.5983371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.5983459Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.5983463Z 2025-09-07T07:09:43.5983580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5983794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5983869Z return mod(**inputs) 2025-09-07T07:09:43.5984130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5984207Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5984477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5984558Z layer_outputs = layer_module( 2025-09-07T07:09:43.5984811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5984915Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5985191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.5985297Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.5985572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-09-07T07:09:43.5985807Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-09-07T07:09:43.5985814Z 2025-09-07T07:09:43.5985911Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.5986028Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5986297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5986371Z return mod(**inputs) 2025-09-07T07:09:43.5986650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5986730Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5987005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5987083Z layer_outputs = layer_module( 2025-09-07T07:09:43.5987326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5987422Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5987717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5987827Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5988081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.5988194Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.5988455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.5988544Z return self.weight * hidden_states 2025-09-07T07:09:43.5988548Z 2025-09-07T07:09:43.5988673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5988912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5988986Z return mod(**inputs) 2025-09-07T07:09:43.5989266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5989362Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5989635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5989714Z layer_outputs = layer_module( 2025-09-07T07:09:43.5989964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5990051Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5990311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5990407Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5990672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5990770Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5991034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.5991121Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.5991125Z 2025-09-07T07:09:43.5991246Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5991466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5991564Z return mod(**inputs) 2025-09-07T07:09:43.5991834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5991918Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5992186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5992264Z layer_outputs = layer_module( 2025-09-07T07:09:43.5992519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5992606Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5992906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5992996Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5993267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5993370Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5993636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.5993728Z key_states = self.k(current_states) 2025-09-07T07:09:43.5993732Z 2025-09-07T07:09:43.5993843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5994062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5994162Z return mod(**inputs) 2025-09-07T07:09:43.5994434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5994523Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5994791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5994879Z layer_outputs = layer_module( 2025-09-07T07:09:43.5995121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5995206Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5995483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5995570Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5995843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5995952Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5996218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.5996372Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.5996376Z 2025-09-07T07:09:43.5996490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5996714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5996784Z return mod(**inputs) 2025-09-07T07:09:43.5997059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5997138Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5997404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5997490Z layer_outputs = layer_module( 2025-09-07T07:09:43.5997738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.5997826Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.5998065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.5998165Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.5998416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.5998499Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.5998747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.5998905Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.5998909Z 2025-09-07T07:09:43.5999020Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.5999222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.5999304Z return mod(**inputs) 2025-09-07T07:09:43.5999558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.5999630Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.5999885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.5999959Z layer_outputs = layer_module( 2025-09-07T07:09:43.6000180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6000268Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6000545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6000638Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6000881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6000965Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6001219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.6001300Z value_states = self.v(current_states) 2025-09-07T07:09:43.6001303Z 2025-09-07T07:09:43.6001413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6001613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6001687Z return mod(**inputs) 2025-09-07T07:09:43.6001931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6002003Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6002254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6002346Z layer_outputs = layer_module( 2025-09-07T07:09:43.6002582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6002667Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6002916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6003004Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6003249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6003338Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6003581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6003693Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6003706Z 2025-09-07T07:09:43.6003818Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6004032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6004107Z return mod(**inputs) 2025-09-07T07:09:43.6004393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6004475Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6004731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6004807Z layer_outputs = layer_module( 2025-09-07T07:09:43.6005050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6005137Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6005402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6005507Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6005768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6005865Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6006121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6006254Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6006257Z 2025-09-07T07:09:43.6006361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6006571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6006637Z return mod(**inputs) 2025-09-07T07:09:43.6006902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6006985Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6007230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6007310Z layer_outputs = layer_module( 2025-09-07T07:09:43.6007533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6007613Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6007869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6007950Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6008197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6008281Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6008540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.6008659Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.6008663Z 2025-09-07T07:09:43.6008767Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6008980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6009045Z return mod(**inputs) 2025-09-07T07:09:43.6009299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6009371Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6009616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6009697Z layer_outputs = layer_module( 2025-09-07T07:09:43.6009922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6010013Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6010258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6010338Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6010612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6010695Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6010945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.6011023Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.6011027Z 2025-09-07T07:09:43.6011116Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6011222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6011425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6011499Z return mod(**inputs) 2025-09-07T07:09:43.6011762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6011845Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6012095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6012169Z layer_outputs = layer_module( 2025-09-07T07:09:43.6012400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6012479Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6012730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6012827Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6013074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-09-07T07:09:43.6013189Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6013435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6013527Z return self.weight * hidden_states 2025-09-07T07:09:43.6013530Z 2025-09-07T07:09:43.6013641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6013860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6013929Z return mod(**inputs) 2025-09-07T07:09:43.6014191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6014276Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6014535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6014640Z layer_outputs = layer_module( 2025-09-07T07:09:43.6014877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6014960Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6015225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6015312Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6015577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6015662Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6015904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.6015991Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.6015996Z 2025-09-07T07:09:43.6016102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6016311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6016377Z return mod(**inputs) 2025-09-07T07:09:43.6016651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6016728Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6016993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6017077Z layer_outputs = layer_module( 2025-09-07T07:09:43.6017317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6017409Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6017669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6017774Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6018040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6018131Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6018395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.6018478Z key_states = self.k(current_states) 2025-09-07T07:09:43.6018482Z 2025-09-07T07:09:43.6018599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6018810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6018879Z return mod(**inputs) 2025-09-07T07:09:43.6019164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6019245Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6019513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6019708Z layer_outputs = layer_module( 2025-09-07T07:09:43.6019961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6020056Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6020320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6020414Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6020674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6020766Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6021034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.6021227Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.6021232Z 2025-09-07T07:09:43.6021350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6021568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6021644Z return mod(**inputs) 2025-09-07T07:09:43.6021908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6021984Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6022254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6022330Z layer_outputs = layer_module( 2025-09-07T07:09:43.6022580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6022667Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6022947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6023047Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6023396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6023495Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6023755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.6023921Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.6023933Z 2025-09-07T07:09:43.6024042Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6024258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6024338Z return mod(**inputs) 2025-09-07T07:09:43.6024621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6024709Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6024998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6025076Z layer_outputs = layer_module( 2025-09-07T07:09:43.6025329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6025415Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6025741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6025870Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6026151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6026253Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6026539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.6026635Z value_states = self.v(current_states) 2025-09-07T07:09:43.6026640Z 2025-09-07T07:09:43.6026753Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6026992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6027063Z return mod(**inputs) 2025-09-07T07:09:43.6027324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6027408Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6027674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6027795Z layer_outputs = layer_module( 2025-09-07T07:09:43.6028034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6028117Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6028387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6028473Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6028733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6028815Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6029051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6029167Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6029172Z 2025-09-07T07:09:43.6029272Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6029477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6029543Z return mod(**inputs) 2025-09-07T07:09:43.6029798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6029883Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6030118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6030194Z layer_outputs = layer_module( 2025-09-07T07:09:43.6030406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6030488Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6030724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6030803Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6031058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6031140Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6031381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6031485Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6031488Z 2025-09-07T07:09:43.6031594Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6031784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6031847Z return mod(**inputs) 2025-09-07T07:09:43.6032107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6032180Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6032423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6032492Z layer_outputs = layer_module( 2025-09-07T07:09:43.6032703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6032788Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6033022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6033107Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6033341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6033425Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6033667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.6033791Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.6033794Z 2025-09-07T07:09:43.6033903Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6034097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6034170Z return mod(**inputs) 2025-09-07T07:09:43.6034408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6034480Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6034724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6034794Z layer_outputs = layer_module( 2025-09-07T07:09:43.6035023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6035105Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6035363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6035459Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6035742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6035836Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6036076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.6036161Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.6036165Z 2025-09-07T07:09:43.6036247Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6036350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6036560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6036626Z return mod(**inputs) 2025-09-07T07:09:43.6036894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6036968Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6037219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6037300Z layer_outputs = layer_module( 2025-09-07T07:09:43.6037517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6037602Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6037844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6037950Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6038198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.6038297Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6038543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6038622Z return self.weight * hidden_states 2025-09-07T07:09:43.6038625Z 2025-09-07T07:09:43.6038734Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6038939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6039003Z return mod(**inputs) 2025-09-07T07:09:43.6039247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6039316Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6039561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6039648Z layer_outputs = layer_module( 2025-09-07T07:09:43.6039869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6039953Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6040195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6040292Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6040529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6040644Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6040889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.6040988Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.6040994Z 2025-09-07T07:09:43.6041104Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6041313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6041386Z return mod(**inputs) 2025-09-07T07:09:43.6041619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6041707Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6041954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6042023Z layer_outputs = layer_module( 2025-09-07T07:09:43.6042243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6042320Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6042561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6042658Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6042912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6043039Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6043289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.6043376Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.6043379Z 2025-09-07T07:09:43.6043482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6043683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6043759Z return mod(**inputs) 2025-09-07T07:09:43.6044027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6044112Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6044375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6044450Z layer_outputs = layer_module( 2025-09-07T07:09:43.6044696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6044780Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6045039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6045127Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6045368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6045495Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6045758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.6045856Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.6045860Z 2025-09-07T07:09:43.6045964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6046186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6046249Z return mod(**inputs) 2025-09-07T07:09:43.6046489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6046568Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6046814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6046889Z layer_outputs = layer_module( 2025-09-07T07:09:43.6047111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6047190Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6047438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6047523Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6047786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6047899Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6048145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.6048225Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.6048228Z 2025-09-07T07:09:43.6048307Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6048417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6048614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6048706Z return mod(**inputs) 2025-09-07T07:09:43.6048951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6049023Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6049275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6049345Z layer_outputs = layer_module( 2025-09-07T07:09:43.6049572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6049649Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6049915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6050004Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6050241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.6050357Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6050600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6050687Z return self.weight * hidden_states 2025-09-07T07:09:43.6050691Z 2025-09-07T07:09:43.6050796Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6050996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6051069Z return mod(**inputs) 2025-09-07T07:09:43.6051316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6051398Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6051651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6051761Z layer_outputs = layer_module( 2025-09-07T07:09:43.6051986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6052066Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6052315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6052394Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6052641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6052724Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6052962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.6053047Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.6053052Z 2025-09-07T07:09:43.6053155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6053360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6053426Z return mod(**inputs) 2025-09-07T07:09:43.6053695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6053777Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6054023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6054102Z layer_outputs = layer_module( 2025-09-07T07:09:43.6054325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6054407Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6054655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6054755Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6055007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6055094Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6055351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.6055426Z key_states = self.k(current_states) 2025-09-07T07:09:43.6055429Z 2025-09-07T07:09:43.6055530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6055738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6055801Z return mod(**inputs) 2025-09-07T07:09:43.6056065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6056139Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6056381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6056461Z layer_outputs = layer_module( 2025-09-07T07:09:43.6056688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6056774Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6057023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6057101Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6057343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6057425Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6057669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.6057818Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.6057822Z 2025-09-07T07:09:43.6057930Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6058125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6058189Z return mod(**inputs) 2025-09-07T07:09:43.6058436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6058507Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6058751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6058822Z layer_outputs = layer_module( 2025-09-07T07:09:43.6059047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6059137Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6059379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6059469Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6059728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6059819Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6060060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.6060219Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.6060223Z 2025-09-07T07:09:43.6060335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6060536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6060610Z return mod(**inputs) 2025-09-07T07:09:43.6060873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6060947Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6061206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6061277Z layer_outputs = layer_module( 2025-09-07T07:09:43.6061510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6061589Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6061834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6061941Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6062185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6062278Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6062521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.6062606Z value_states = self.v(current_states) 2025-09-07T07:09:43.6062609Z 2025-09-07T07:09:43.6062712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6062915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6062991Z return mod(**inputs) 2025-09-07T07:09:43.6063247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6063330Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6063591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6063686Z layer_outputs = layer_module( 2025-09-07T07:09:43.6063932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6064017Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6064284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6064371Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6064632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6064717Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6064975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6065100Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6065105Z 2025-09-07T07:09:43.6065216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6065438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6065507Z return mod(**inputs) 2025-09-07T07:09:43.6065858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6065973Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6066236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6066322Z layer_outputs = layer_module( 2025-09-07T07:09:43.6066558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6066642Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6066912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6067000Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6067283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6067375Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6067642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6067759Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6067763Z 2025-09-07T07:09:43.6067874Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6068102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6068168Z return mod(**inputs) 2025-09-07T07:09:43.6068444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6068521Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6068771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6068853Z layer_outputs = layer_module( 2025-09-07T07:09:43.6069079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6069170Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6069417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6069505Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6069748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6069832Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6070082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.6070211Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.6070215Z 2025-09-07T07:09:43.6070328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6070535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6070601Z return mod(**inputs) 2025-09-07T07:09:43.6070862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6070933Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6071186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6071257Z layer_outputs = layer_module( 2025-09-07T07:09:43.6071481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6071570Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6071814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6071903Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6072146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6072255Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6072498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.6072576Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.6072580Z 2025-09-07T07:09:43.6072692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6072896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6072969Z return mod(**inputs) 2025-09-07T07:09:43.6073242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6073317Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6073580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6073658Z layer_outputs = layer_module( 2025-09-07T07:09:43.6073902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6073987Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6074250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6074337Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6074612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-09-07T07:09:43.6074770Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:09:43.6074774Z 2025-09-07T07:09:43.6074862Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6074980Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6075382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6075449Z return mod(**inputs) 2025-09-07T07:09:43.6075705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6075776Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6076025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6076095Z layer_outputs = layer_module( 2025-09-07T07:09:43.6076327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6076427Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6076671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6076764Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6077007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-09-07T07:09:43.6077125Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6077380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6077462Z return self.weight * hidden_states 2025-09-07T07:09:43.6077466Z 2025-09-07T07:09:43.6077585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6077800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6077878Z return mod(**inputs) 2025-09-07T07:09:43.6078137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6078213Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6078476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6078573Z layer_outputs = layer_module( 2025-09-07T07:09:43.6078821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6078903Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6079164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6079246Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6079488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6079584Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6079842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.6079930Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.6079935Z 2025-09-07T07:09:43.6080040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6080253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6080332Z return mod(**inputs) 2025-09-07T07:09:43.6080592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6080672Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6080952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6081032Z layer_outputs = layer_module( 2025-09-07T07:09:43.6081279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6081364Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6081630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6081719Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6081981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6082072Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6082330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.6082421Z key_states = self.k(current_states) 2025-09-07T07:09:43.6082425Z 2025-09-07T07:09:43.6082536Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6082775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6082848Z return mod(**inputs) 2025-09-07T07:09:43.6083109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6083195Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6083459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6083542Z layer_outputs = layer_module( 2025-09-07T07:09:43.6083781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6083873Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6084137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6084224Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6084492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6084581Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6084851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.6085010Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.6085014Z 2025-09-07T07:09:43.6085122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6085344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6085413Z return mod(**inputs) 2025-09-07T07:09:43.6085681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6085758Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6086038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6086121Z layer_outputs = layer_module( 2025-09-07T07:09:43.6086357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6086449Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6086707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6086800Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6087056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6087145Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6087423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.6087595Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.6087599Z 2025-09-07T07:09:43.6087717Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6087929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6088001Z return mod(**inputs) 2025-09-07T07:09:43.6088272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6088348Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6088616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6088692Z layer_outputs = layer_module( 2025-09-07T07:09:43.6088941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6089041Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6089299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6089394Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6089652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6089750Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6090003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.6090088Z value_states = self.v(current_states) 2025-09-07T07:09:43.6090091Z 2025-09-07T07:09:43.6090208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6090422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6090499Z return mod(**inputs) 2025-09-07T07:09:43.6090773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6090855Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6091128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6091222Z layer_outputs = layer_module( 2025-09-07T07:09:43.6091463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6091546Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6091809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6091896Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6092152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6092250Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6092546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6092671Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6092675Z 2025-09-07T07:09:43.6092786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6093000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6093076Z return mod(**inputs) 2025-09-07T07:09:43.6093345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6093427Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6093705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6093788Z layer_outputs = layer_module( 2025-09-07T07:09:43.6094026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6094110Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6094374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6094463Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6094725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6094813Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6095066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6095189Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6095193Z 2025-09-07T07:09:43.6095303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6095552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6095618Z return mod(**inputs) 2025-09-07T07:09:43.6095866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6095947Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6096195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6096274Z layer_outputs = layer_module( 2025-09-07T07:09:43.6096500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6096587Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6096839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6096926Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6097195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6097284Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6097550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.6097968Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.6097972Z 2025-09-07T07:09:43.6098082Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6098303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6098375Z return mod(**inputs) 2025-09-07T07:09:43.6098647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6098726Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6098999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6099098Z layer_outputs = layer_module( 2025-09-07T07:09:43.6099338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6099434Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6099693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6099791Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6100045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6100134Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6100418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.6100505Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.6100509Z 2025-09-07T07:09:43.6100602Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6100714Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6100934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6101013Z return mod(**inputs) 2025-09-07T07:09:43.6101279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6101366Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6101633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6101715Z layer_outputs = layer_module( 2025-09-07T07:09:43.6101959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6102044Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6102327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6102427Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6102701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.6102808Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6103070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6103162Z return self.weight * hidden_states 2025-09-07T07:09:43.6103166Z 2025-09-07T07:09:43.6103275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6103502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6103586Z return mod(**inputs) 2025-09-07T07:09:43.6103848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6103935Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6104195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6104299Z layer_outputs = layer_module( 2025-09-07T07:09:43.6104537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6104626Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6104886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6104982Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6105254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6105384Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6105774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.6105895Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.6105901Z 2025-09-07T07:09:43.6106012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6106258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6106329Z return mod(**inputs) 2025-09-07T07:09:43.6106604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6106683Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6106990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6107071Z layer_outputs = layer_module( 2025-09-07T07:09:43.6107319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6107416Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6107687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6107790Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6108047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6108177Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6108425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.6108507Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.6108512Z 2025-09-07T07:09:43.6108622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6108844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6108917Z return mod(**inputs) 2025-09-07T07:09:43.6109163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6109235Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6109488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6109559Z layer_outputs = layer_module( 2025-09-07T07:09:43.6109791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6109871Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6110129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6110235Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6110496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6110626Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6110882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.6110998Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.6111010Z 2025-09-07T07:09:43.6111119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6111333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6111405Z return mod(**inputs) 2025-09-07T07:09:43.6111715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6111801Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6112082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6112160Z layer_outputs = layer_module( 2025-09-07T07:09:43.6112407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6112493Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6112758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6112852Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6113108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6113239Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6113515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.6113611Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.6113614Z 2025-09-07T07:09:43.6113702Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6113819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6114031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6114103Z return mod(**inputs) 2025-09-07T07:09:43.6114370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6114447Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6114726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6114803Z layer_outputs = layer_module( 2025-09-07T07:09:43.6115057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6115170Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6115426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6115519Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6115776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-09-07T07:09:43.6115889Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6116151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6116234Z return self.weight * hidden_states 2025-09-07T07:09:43.6116237Z 2025-09-07T07:09:43.6116355Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6116568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6116645Z return mod(**inputs) 2025-09-07T07:09:43.6116906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6116982Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6117248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6117344Z layer_outputs = layer_module( 2025-09-07T07:09:43.6117586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6117669Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6117923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6118017Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6118274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6118371Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6118641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.6118724Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.6118737Z 2025-09-07T07:09:43.6118848Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6119060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6119140Z return mod(**inputs) 2025-09-07T07:09:43.6119400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6119483Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6119935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6120018Z layer_outputs = layer_module( 2025-09-07T07:09:43.6120266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6120351Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6120615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6120702Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6120959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6121058Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6121315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.6121406Z key_states = self.k(current_states) 2025-09-07T07:09:43.6121410Z 2025-09-07T07:09:43.6121523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6121781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6121853Z return mod(**inputs) 2025-09-07T07:09:43.6122116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6122203Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6122459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6122543Z layer_outputs = layer_module( 2025-09-07T07:09:43.6122781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6122863Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6123126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6123214Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6123476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6123564Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6123818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.6123995Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.6123999Z 2025-09-07T07:09:43.6124109Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6124332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6124403Z return mod(**inputs) 2025-09-07T07:09:43.6124670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6124747Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6125006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6125114Z layer_outputs = layer_module( 2025-09-07T07:09:43.6125350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6125442Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6125699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6125785Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6126047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6126136Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6126416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.6126589Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.6126593Z 2025-09-07T07:09:43.6126714Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6126934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6127006Z return mod(**inputs) 2025-09-07T07:09:43.6127282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6127355Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6127618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6127692Z layer_outputs = layer_module( 2025-09-07T07:09:43.6127922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6128013Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6128276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6128365Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6128607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6128692Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6128940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.6129018Z value_states = self.v(current_states) 2025-09-07T07:09:43.6129022Z 2025-09-07T07:09:43.6129134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6129336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6129410Z return mod(**inputs) 2025-09-07T07:09:43.6129658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6129731Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6129982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6130072Z layer_outputs = layer_module( 2025-09-07T07:09:43.6130301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6130381Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6130619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6130707Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6130949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6131038Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6131295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6131408Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6131419Z 2025-09-07T07:09:43.6131525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6131730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6131803Z return mod(**inputs) 2025-09-07T07:09:43.6132051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6132130Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6132374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6132459Z layer_outputs = layer_module( 2025-09-07T07:09:43.6132693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6132773Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6133024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6133108Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6133350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6133440Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6133680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6133793Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6133797Z 2025-09-07T07:09:43.6133900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6134126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6134194Z return mod(**inputs) 2025-09-07T07:09:43.6134442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6134521Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6134769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6134846Z layer_outputs = layer_module( 2025-09-07T07:09:43.6135068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6135146Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6135393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6135477Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6135730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6135810Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6136054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.6136192Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.6136195Z 2025-09-07T07:09:43.6136299Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6136510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6136576Z return mod(**inputs) 2025-09-07T07:09:43.6136832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6136906Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6137149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6137250Z layer_outputs = layer_module( 2025-09-07T07:09:43.6137474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6137561Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6137800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-09-07T07:09:43.6137881Z self_attention_outputs = self.layer[0]( 2025-09-07T07:09:43.6138129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-09-07T07:09:43.6138209Z attention_output = self.SelfAttention( 2025-09-07T07:09:43.6138477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.6138556Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.6138561Z 2025-09-07T07:09:43.6138650Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6138754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6138958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6139034Z return mod(**inputs) 2025-09-07T07:09:43.6139285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6139363Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6139612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6139684Z layer_outputs = layer_module( 2025-09-07T07:09:43.6139920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6139999Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6140268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6140350Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6140589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-09-07T07:09:43.6140704Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6140944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6141031Z return self.weight * hidden_states 2025-09-07T07:09:43.6141035Z 2025-09-07T07:09:43.6141144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6141361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6141433Z return mod(**inputs) 2025-09-07T07:09:43.6141696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6141783Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6142043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6142146Z layer_outputs = layer_module( 2025-09-07T07:09:43.6142383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6142467Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6142730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6142816Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6143081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6143173Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6143445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-09-07T07:09:43.6143537Z query_states = self.q(hidden_states) 2025-09-07T07:09:43.6143541Z 2025-09-07T07:09:43.6143653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6143875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6143944Z return mod(**inputs) 2025-09-07T07:09:43.6144216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6144293Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6144552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6144664Z layer_outputs = layer_module( 2025-09-07T07:09:43.6144903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6144999Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6145257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6145347Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6145692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6145794Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6146071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-09-07T07:09:43.6146156Z key_states = self.k(current_states) 2025-09-07T07:09:43.6146160Z 2025-09-07T07:09:43.6146284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6146506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6146601Z return mod(**inputs) 2025-09-07T07:09:43.6146891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6146972Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6147256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6147346Z layer_outputs = layer_module( 2025-09-07T07:09:43.6147588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6147684Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6147946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6148044Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6148313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6148404Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6148678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-09-07T07:09:43.6148844Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:09:43.6148848Z 2025-09-07T07:09:43.6148964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6149180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6149258Z return mod(**inputs) 2025-09-07T07:09:43.6149520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6149597Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6149866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6149943Z layer_outputs = layer_module( 2025-09-07T07:09:43.6150214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6150298Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6150555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6150649Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6150906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6151004Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6151309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-09-07T07:09:43.6151486Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:09:43.6151492Z 2025-09-07T07:09:43.6151605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6151819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6151897Z return mod(**inputs) 2025-09-07T07:09:43.6152162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6152246Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6152508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6152584Z layer_outputs = layer_module( 2025-09-07T07:09:43.6152833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6152917Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6153202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6153291Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6153547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6153644Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6153898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-09-07T07:09:43.6153985Z value_states = self.v(current_states) 2025-09-07T07:09:43.6153989Z 2025-09-07T07:09:43.6154096Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6154313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6154382Z return mod(**inputs) 2025-09-07T07:09:43.6154641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6154728Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6154990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6155072Z layer_outputs = layer_module( 2025-09-07T07:09:43.6155332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6155415Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6155681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6155767Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6156032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6156122Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6156382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6156522Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6156526Z 2025-09-07T07:09:43.6156638Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6156864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6156934Z return mod(**inputs) 2025-09-07T07:09:43.6157202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6157279Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6157538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6157621Z layer_outputs = layer_module( 2025-09-07T07:09:43.6157876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6157969Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6158228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6158318Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6158593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6158678Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6158927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-09-07T07:09:43.6159034Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:09:43.6159038Z 2025-09-07T07:09:43.6159148Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6159352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6159439Z return mod(**inputs) 2025-09-07T07:09:43.6159698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6159772Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6160028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6160102Z layer_outputs = layer_module( 2025-09-07T07:09:43.6160324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6160410Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6160654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6160747Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6160989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6161077Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6161327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-09-07T07:09:43.6161457Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:09:43.6161460Z 2025-09-07T07:09:43.6161571Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6161771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6161851Z return mod(**inputs) 2025-09-07T07:09:43.6162096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6162169Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6162422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6162494Z layer_outputs = layer_module( 2025-09-07T07:09:43.6162737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6162817Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6163065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6163155Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6163402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-09-07T07:09:43.6163499Z attention_output = self.EncDecAttention( 2025-09-07T07:09:43.6163758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-09-07T07:09:43.6163865Z attn_output = self.o(attn_output) 2025-09-07T07:09:43.6163871Z 2025-09-07T07:09:43.6163984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6164199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6164277Z return mod(**inputs) 2025-09-07T07:09:43.6164540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6164628Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6164889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6164965Z layer_outputs = layer_module( 2025-09-07T07:09:43.6165210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6165293Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6165566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-09-07T07:09:43.6165666Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:09:43.6165912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-09-07T07:09:43.6166053Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:09:43.6166058Z 2025-09-07T07:09:43.6166142Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6166251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6166450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6166528Z return mod(**inputs) 2025-09-07T07:09:43.6166773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6166844Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6167096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6167168Z layer_outputs = layer_module( 2025-09-07T07:09:43.6167398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6167476Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6167734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6167834Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6168076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-09-07T07:09:43.6168181Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:09:43.6168426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6168511Z return self.weight * hidden_states 2025-09-07T07:09:43.6168516Z 2025-09-07T07:09:43.6168617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6168841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6168916Z return mod(**inputs) 2025-09-07T07:09:43.6169163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6169243Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6169490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6169561Z layer_outputs = layer_module( 2025-09-07T07:09:43.6169790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6169869Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6170134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6170228Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6170474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6170602Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6170846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-09-07T07:09:43.6170954Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-09-07T07:09:43.6170958Z 2025-09-07T07:09:43.6171060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6171267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6171333Z return mod(**inputs) 2025-09-07T07:09:43.6171588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6171682Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6171931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6172007Z layer_outputs = layer_module( 2025-09-07T07:09:43.6172236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6172314Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6172575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6172667Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6172942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6173069Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6173357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-09-07T07:09:43.6173448Z hidden_linear = self.wi_1(hidden_states) 2025-09-07T07:09:43.6173451Z 2025-09-07T07:09:43.6173560Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6173803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6173873Z return mod(**inputs) 2025-09-07T07:09:43.6174151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6174227Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6174498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6174582Z layer_outputs = layer_module( 2025-09-07T07:09:43.6174835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6174927Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6175212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6175317Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6175573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6175689Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6175953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-09-07T07:09:43.6176040Z hidden_states = hidden_gelu * hidden_linear 2025-09-07T07:09:43.6176043Z 2025-09-07T07:09:43.6176151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6176364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6176432Z return mod(**inputs) 2025-09-07T07:09:43.6176679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6176750Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6176998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-09-07T07:09:43.6177071Z layer_outputs = layer_module( 2025-09-07T07:09:43.6177297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:09:43.6177383Z return super().__call__(*args, **kwargs) 2025-09-07T07:09:43.6177625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-09-07T07:09:43.6177728Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:09:43.6177987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-09-07T07:09:43.6178137Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:09:43.6178396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-09-07T07:09:43.6178483Z hidden_states = self.wo(hidden_states) 2025-09-07T07:09:43.6178487Z 2025-09-07T07:09:43.6178581Z cudagraph partition due to non gpu ops 2025-09-07T07:09:43.6178692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6178911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6178979Z return mod(**inputs) 2025-09-07T07:09:43.6179240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-09-07T07:09:43.6179326Z decoder_outputs = self.decoder( 2025-09-07T07:09:43.6179587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-09-07T07:09:43.6179707Z hidden_states = self.final_layer_norm(hidden_states) 2025-09-07T07:09:43.6179964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-09-07T07:09:43.6180065Z return self.weight * hidden_states 2025-09-07T07:09:43.6180077Z 2025-09-07T07:09:43.6180188Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6180401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6180478Z return mod(**inputs) 2025-09-07T07:09:43.6180738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1816, in forward 2025-09-07T07:09:43.6180839Z lm_logits = self.lm_head(sequence_output) 2025-09-07T07:09:43.6180844Z 2025-09-07T07:09:43.6180953Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6181187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6181266Z return mod(**inputs) 2025-09-07T07:09:43.6181532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-09-07T07:09:43.6181695Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-09-07T07:09:43.6181699Z 2025-09-07T07:09:43.6181810Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6182021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6182098Z return mod(**inputs) 2025-09-07T07:09:43.6182367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-09-07T07:09:43.6182537Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-09-07T07:09:43.6182543Z 2025-09-07T07:09:43.6182651Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:09:43.6182871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:09:43.6182939Z return mod(**inputs) 2025-09-07T07:09:43.6183201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-09-07T07:09:43.6183356Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-09-07T07:09:43.6183360Z 2025-09-07T07:09:57.5540051Z Compilation time (from dynamo_timed): 25.462242104 2025-09-07T07:09:57.5785002Z pass 2025-09-07T07:09:57.5785444Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:09:57.5786678Z TIMING: _recursive_pre_grad_passes:0.01578 _recursive_joint_graph_passes:0.75858 _recursive_post_grad_passes:0.26244 async_compile.wait:0.79946 code_gen:12.79808 inductor_compile:15.59243 backend_compile:20.65384 gc:0.00097 entire_frame_compile:25.46224 total_wall_time:25.46224 2025-09-07T07:09:57.5788227Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:29413 | FakeTensor.__torch_dispatch__:8057 | ProxyTorchDispatchMode.__torch_dispatch__:10618 2025-09-07T07:09:57.5788813Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-09-07T07:10:00.3253807Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:10:00.3255570Z import pynvml # type: ignore[import] 2025-09-07T07:10:03.1333181Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:10:03.1336095Z from pkg_resources import resource_filename 2025-09-07T07:10:03.8430681Z 2025-09-07T07:10:03.8558194Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-09-07T07:10:03.8566691Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-09-07T07:10:07.0212781Z 2025-09-07T07:10:07.0213793Z loading model: 0it [00:03, ?it/s] 2025-09-07T07:10:07.0250046Z cpu eval MegatronBertForCausalLM 2025-09-07T07:10:08.8139555Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:10:09.5054435Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:10:10.1944225Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:10:24.6586585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6587403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6587915Z return mod(**inputs) 2025-09-07T07:10:24.6588520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6589037Z outputs = self.bert( 2025-09-07T07:10:24.6589495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6589965Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6590519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6590999Z layer_outputs = layer_module( 2025-09-07T07:10:24.6591396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6591807Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6592279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6592804Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6593285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6593754Z self_outputs = self.self( 2025-09-07T07:10:24.6594160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6594571Z return func(*args, **kwargs) 2025-09-07T07:10:24.6595032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.6595564Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.6595724Z 2025-09-07T07:10:24.6595852Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6596256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6596671Z return mod(**inputs) 2025-09-07T07:10:24.6597121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6597600Z outputs = self.bert( 2025-09-07T07:10:24.6598038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6598581Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6599039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6599508Z layer_outputs = layer_module( 2025-09-07T07:10:24.6599891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6600293Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6600816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6601363Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6601834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6602295Z self_outputs = self.self( 2025-09-07T07:10:24.6602693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6603103Z return func(*args, **kwargs) 2025-09-07T07:10:24.6603561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.6604052Z key_layer = self.key(current_states) 2025-09-07T07:10:24.6604201Z 2025-09-07T07:10:24.6604330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6604731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6605079Z return mod(**inputs) 2025-09-07T07:10:24.6605494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6605946Z outputs = self.bert( 2025-09-07T07:10:24.6606403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6606908Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6607348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6607784Z layer_outputs = layer_module( 2025-09-07T07:10:24.6608144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6608515Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6608954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6609401Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6609839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6610268Z self_outputs = self.self( 2025-09-07T07:10:24.6610639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6611048Z return func(*args, **kwargs) 2025-09-07T07:10:24.6611491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.6611959Z value_layer = self.value(current_states) 2025-09-07T07:10:24.6612099Z 2025-09-07T07:10:24.6612188Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6612425Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6612683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6613079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6613432Z return mod(**inputs) 2025-09-07T07:10:24.6613862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6614315Z outputs = self.bert( 2025-09-07T07:10:24.6614744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6615205Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6615650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6616157Z layer_outputs = layer_module( 2025-09-07T07:10:24.6616545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6616963Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6617428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6617901Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6618355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.6618849Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.6619353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.6620078Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6620231Z 2025-09-07T07:10:24.6620341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6620716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6621049Z return mod(**inputs) 2025-09-07T07:10:24.6621462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6621899Z outputs = self.bert( 2025-09-07T07:10:24.6622393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6622858Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6623323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6623807Z layer_outputs = layer_module( 2025-09-07T07:10:24.6624181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6624589Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6625067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6625559Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6626192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6626642Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6627190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6627742Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6628251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.6628739Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6628895Z 2025-09-07T07:10:24.6629013Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6629422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6629788Z return mod(**inputs) 2025-09-07T07:10:24.6630235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6630698Z outputs = self.bert( 2025-09-07T07:10:24.6631135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6631603Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6632065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6632563Z layer_outputs = layer_module( 2025-09-07T07:10:24.6632949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6633357Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6633828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6634335Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6634794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6635238Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6635764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6636293Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6636788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.6637297Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.6638359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.6638834Z return self.act(input) 2025-09-07T07:10:24.6638985Z 2025-09-07T07:10:24.6639150Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6639735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6640221Z return mod(**inputs) 2025-09-07T07:10:24.6640733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6641254Z outputs = self.bert( 2025-09-07T07:10:24.6641778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6642315Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6642886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6643413Z layer_outputs = layer_module( 2025-09-07T07:10:24.6643847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6644274Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6644814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6645332Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6645831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6646306Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6646823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6647422Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6647978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.6648522Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6648689Z 2025-09-07T07:10:24.6648867Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6649280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6649734Z return mod(**inputs) 2025-09-07T07:10:24.6650213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6650794Z outputs = self.bert( 2025-09-07T07:10:24.6651302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6651806Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6652344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6652887Z layer_outputs = layer_module( 2025-09-07T07:10:24.6653329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6653792Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6654272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6654833Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6655345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6655859Z self_outputs = self.self( 2025-09-07T07:10:24.6656323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6656797Z return func(*args, **kwargs) 2025-09-07T07:10:24.6657350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.6657892Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.6658064Z 2025-09-07T07:10:24.6658251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6658697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6659073Z return mod(**inputs) 2025-09-07T07:10:24.6659580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6660125Z outputs = self.bert( 2025-09-07T07:10:24.6660617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6661188Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6661686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6662261Z layer_outputs = layer_module( 2025-09-07T07:10:24.6662727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6663210Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6663756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6664306Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6664883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6665452Z self_outputs = self.self( 2025-09-07T07:10:24.6666149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6666646Z return func(*args, **kwargs) 2025-09-07T07:10:24.6667170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.6667733Z key_layer = self.key(current_states) 2025-09-07T07:10:24.6667941Z 2025-09-07T07:10:24.6668080Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6668575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6669042Z return mod(**inputs) 2025-09-07T07:10:24.6669521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6670086Z outputs = self.bert( 2025-09-07T07:10:24.6670599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6671121Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6671709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6672257Z layer_outputs = layer_module( 2025-09-07T07:10:24.6672745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6673225Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6673804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6674342Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6674831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6675322Z self_outputs = self.self( 2025-09-07T07:10:24.6675811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6676265Z return func(*args, **kwargs) 2025-09-07T07:10:24.6676783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.6677276Z value_layer = self.value(current_states) 2025-09-07T07:10:24.6677466Z 2025-09-07T07:10:24.6677573Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6677872Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6678195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6678606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6679011Z return mod(**inputs) 2025-09-07T07:10:24.6679511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6680159Z outputs = self.bert( 2025-09-07T07:10:24.6680641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6681209Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6681768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6682277Z layer_outputs = layer_module( 2025-09-07T07:10:24.6682728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6683288Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6683815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6684369Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6684922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.6685548Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.6686172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.6686670Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6686852Z 2025-09-07T07:10:24.6687040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6687521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6687922Z return mod(**inputs) 2025-09-07T07:10:24.6688430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6688898Z outputs = self.bert( 2025-09-07T07:10:24.6689371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6689900Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6690417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6690897Z layer_outputs = layer_module( 2025-09-07T07:10:24.6691330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6691768Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6692257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6692870Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6693419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6693920Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6711793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6712557Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6713076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.6713560Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6713728Z 2025-09-07T07:10:24.6713850Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6714271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6714622Z return mod(**inputs) 2025-09-07T07:10:24.6715052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6715488Z outputs = self.bert( 2025-09-07T07:10:24.6715890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6716425Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6716867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6717307Z layer_outputs = layer_module( 2025-09-07T07:10:24.6717696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6718101Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6718582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6719061Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6719503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6720100Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6720575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6721088Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6721666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.6722167Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.6722593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.6722974Z return self.act(input) 2025-09-07T07:10:24.6723102Z 2025-09-07T07:10:24.6723223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6723634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6724013Z return mod(**inputs) 2025-09-07T07:10:24.6724511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6724984Z outputs = self.bert( 2025-09-07T07:10:24.6725390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6725852Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6726312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6726773Z layer_outputs = layer_module( 2025-09-07T07:10:24.6727160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6727583Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6728054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6728527Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6728966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6729401Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6729893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6730447Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6730970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.6731440Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6731591Z 2025-09-07T07:10:24.6731741Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6732131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6732487Z return mod(**inputs) 2025-09-07T07:10:24.6732930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6733388Z outputs = self.bert( 2025-09-07T07:10:24.6733810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6734268Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6734722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6735177Z layer_outputs = layer_module( 2025-09-07T07:10:24.6735558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6735948Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6736410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6736873Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6737352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6737762Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6738216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6738735Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6739223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.6739661Z return input_tensor + hidden_states 2025-09-07T07:10:24.6739799Z 2025-09-07T07:10:24.6739934Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6740301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6740653Z return mod(**inputs) 2025-09-07T07:10:24.6741091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6741544Z outputs = self.bert( 2025-09-07T07:10:24.6741966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6742422Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6742892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6743357Z layer_outputs = layer_module( 2025-09-07T07:10:24.6743738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6744124Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6744588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6745062Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6745527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6746052Z self_outputs = self.self( 2025-09-07T07:10:24.6746451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6746880Z return func(*args, **kwargs) 2025-09-07T07:10:24.6747342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.6747791Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.6747931Z 2025-09-07T07:10:24.6748035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6748400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6748730Z return mod(**inputs) 2025-09-07T07:10:24.6749128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6749546Z outputs = self.bert( 2025-09-07T07:10:24.6749960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6750382Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6750799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6751217Z layer_outputs = layer_module( 2025-09-07T07:10:24.6751568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6751930Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6752407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6752878Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6753347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6753797Z self_outputs = self.self( 2025-09-07T07:10:24.6754200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6754583Z return func(*args, **kwargs) 2025-09-07T07:10:24.6755025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.6755469Z key_layer = self.key(current_states) 2025-09-07T07:10:24.6755605Z 2025-09-07T07:10:24.6755712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6756082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6756412Z return mod(**inputs) 2025-09-07T07:10:24.6756817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6757241Z outputs = self.bert( 2025-09-07T07:10:24.6757636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6758081Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6758516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6758944Z layer_outputs = layer_module( 2025-09-07T07:10:24.6759293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6759663Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6760099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6760554Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6760996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6761419Z self_outputs = self.self( 2025-09-07T07:10:24.6761791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6762193Z return func(*args, **kwargs) 2025-09-07T07:10:24.6762614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.6763051Z value_layer = self.value(current_states) 2025-09-07T07:10:24.6763190Z 2025-09-07T07:10:24.6763276Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6763499Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6763748Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6764122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6764445Z return mod(**inputs) 2025-09-07T07:10:24.6764854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6765291Z outputs = self.bert( 2025-09-07T07:10:24.6765690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6766112Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6766520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6766954Z layer_outputs = layer_module( 2025-09-07T07:10:24.6767298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6767655Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6768567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6768987Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6769415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.6769945Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.6770507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.6770992Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6771141Z 2025-09-07T07:10:24.6771257Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6771659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6771981Z return mod(**inputs) 2025-09-07T07:10:24.6772374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6772779Z outputs = self.bert( 2025-09-07T07:10:24.6773188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6773611Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6774028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6774449Z layer_outputs = layer_module( 2025-09-07T07:10:24.6774791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6775149Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6775572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6776011Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6776408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6776814Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6777268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6777756Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6778210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.6778656Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6778806Z 2025-09-07T07:10:24.6778910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6779270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6779595Z return mod(**inputs) 2025-09-07T07:10:24.6779999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6780410Z outputs = self.bert( 2025-09-07T07:10:24.6780820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6781260Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6781715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6782190Z layer_outputs = layer_module( 2025-09-07T07:10:24.6782559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6782927Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6783359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6783812Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6784240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6784686Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6785173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6785785Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6786261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.6786739Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.6787160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.6787531Z return self.act(input) 2025-09-07T07:10:24.6787647Z 2025-09-07T07:10:24.6787785Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6788164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6788494Z return mod(**inputs) 2025-09-07T07:10:24.6788912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6789343Z outputs = self.bert( 2025-09-07T07:10:24.6789755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6790194Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6790621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6791061Z layer_outputs = layer_module( 2025-09-07T07:10:24.6791428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6791816Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6792242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6792687Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6793095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6793503Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6793966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6794479Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6794968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.6795413Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6795559Z 2025-09-07T07:10:24.6795673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6796046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6796373Z return mod(**inputs) 2025-09-07T07:10:24.6796806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6797219Z outputs = self.bert( 2025-09-07T07:10:24.6797610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6798029Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6798435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6798851Z layer_outputs = layer_module( 2025-09-07T07:10:24.6799194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6799571Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6799985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6800415Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6800841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6801258Z self_outputs = self.self( 2025-09-07T07:10:24.6801627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6802003Z return func(*args, **kwargs) 2025-09-07T07:10:24.6802444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.6802891Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.6803032Z 2025-09-07T07:10:24.6803150Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6803530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6803851Z return mod(**inputs) 2025-09-07T07:10:24.6804251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6804662Z outputs = self.bert( 2025-09-07T07:10:24.6805057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6805469Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6805889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6806322Z layer_outputs = layer_module( 2025-09-07T07:10:24.6806670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6807028Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6807443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6807872Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6808298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6808713Z self_outputs = self.self( 2025-09-07T07:10:24.6809076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6809450Z return func(*args, **kwargs) 2025-09-07T07:10:24.6809868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.6810305Z key_layer = self.key(current_states) 2025-09-07T07:10:24.6810442Z 2025-09-07T07:10:24.6810563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6810942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6811258Z return mod(**inputs) 2025-09-07T07:10:24.6811665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6812090Z outputs = self.bert( 2025-09-07T07:10:24.6812497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6812928Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6813364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6813822Z layer_outputs = layer_module( 2025-09-07T07:10:24.6814175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6814542Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6814961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6815394Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6815826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6816252Z self_outputs = self.self( 2025-09-07T07:10:24.6816636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6817010Z return func(*args, **kwargs) 2025-09-07T07:10:24.6817420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.6817848Z value_layer = self.value(current_states) 2025-09-07T07:10:24.6817983Z 2025-09-07T07:10:24.6818076Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6818288Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6818527Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6818894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6819217Z return mod(**inputs) 2025-09-07T07:10:24.6819843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6820269Z outputs = self.bert( 2025-09-07T07:10:24.6820678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6821168Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6821595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6822021Z layer_outputs = layer_module( 2025-09-07T07:10:24.6822399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6822789Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6823252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6823720Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6824178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.6824687Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.6825180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.6825623Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6825854Z 2025-09-07T07:10:24.6825972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6826351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6826705Z return mod(**inputs) 2025-09-07T07:10:24.6827123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6827539Z outputs = self.bert( 2025-09-07T07:10:24.6827942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6828361Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6828816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6829249Z layer_outputs = layer_module( 2025-09-07T07:10:24.6829587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6829935Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6830355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6830779Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6831173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6831590Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6832026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6832498Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6832944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.6833379Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6833521Z 2025-09-07T07:10:24.6833632Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6834002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6834323Z return mod(**inputs) 2025-09-07T07:10:24.6834721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6835139Z outputs = self.bert( 2025-09-07T07:10:24.6836425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6836830Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6837237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6837675Z layer_outputs = layer_module( 2025-09-07T07:10:24.6838024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6838383Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6838802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6839219Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6839610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6839994Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6840469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6840951Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6841439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.6841914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.6842313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.6842654Z return self.act(input) 2025-09-07T07:10:24.6842773Z 2025-09-07T07:10:24.6842877Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6843242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6843572Z return mod(**inputs) 2025-09-07T07:10:24.6843993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6844409Z outputs = self.bert( 2025-09-07T07:10:24.6844809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6845234Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6845653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6846070Z layer_outputs = layer_module( 2025-09-07T07:10:24.6846419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6846807Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6847239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6847704Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6848112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6848508Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6848957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6849459Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6849934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.6850371Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6850537Z 2025-09-07T07:10:24.6850640Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6851007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6851335Z return mod(**inputs) 2025-09-07T07:10:24.6851724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6852148Z outputs = self.bert( 2025-09-07T07:10:24.6852545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6852965Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6853372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6853791Z layer_outputs = layer_module( 2025-09-07T07:10:24.6854137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6854496Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6854913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6855362Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6855762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6856154Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6856604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6857108Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6857575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.6858003Z return input_tensor + hidden_states 2025-09-07T07:10:24.6858142Z 2025-09-07T07:10:24.6858269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6858633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6858960Z return mod(**inputs) 2025-09-07T07:10:24.6859354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6859770Z outputs = self.bert( 2025-09-07T07:10:24.6860169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6860599Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6861040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6861470Z layer_outputs = layer_module( 2025-09-07T07:10:24.6861825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6862197Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6862644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6863106Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6863561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6863990Z self_outputs = self.self( 2025-09-07T07:10:24.6864371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6864789Z return func(*args, **kwargs) 2025-09-07T07:10:24.6865271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.6865814Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.6865979Z 2025-09-07T07:10:24.6866098Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6866494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6866840Z return mod(**inputs) 2025-09-07T07:10:24.6867278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6867736Z outputs = self.bert( 2025-09-07T07:10:24.6868175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6868641Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6869093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6869557Z layer_outputs = layer_module( 2025-09-07T07:10:24.6869933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6870357Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6870818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6871281Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6871748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6872200Z self_outputs = self.self( 2025-09-07T07:10:24.6872592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6872995Z return func(*args, **kwargs) 2025-09-07T07:10:24.6873461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.6873903Z key_layer = self.key(current_states) 2025-09-07T07:10:24.6874038Z 2025-09-07T07:10:24.6874153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6874520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6874845Z return mod(**inputs) 2025-09-07T07:10:24.6875263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6875695Z outputs = self.bert( 2025-09-07T07:10:24.6876114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6876544Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6876966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6877392Z layer_outputs = layer_module( 2025-09-07T07:10:24.6877742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6878116Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6878548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6878982Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6879424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6879854Z self_outputs = self.self( 2025-09-07T07:10:24.6880226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6880633Z return func(*args, **kwargs) 2025-09-07T07:10:24.6881060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.6881507Z value_layer = self.value(current_states) 2025-09-07T07:10:24.6881645Z 2025-09-07T07:10:24.6881737Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6881961Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6882195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6882568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6882910Z return mod(**inputs) 2025-09-07T07:10:24.6883327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6883752Z outputs = self.bert( 2025-09-07T07:10:24.6884167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6884597Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6885024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6885478Z layer_outputs = layer_module( 2025-09-07T07:10:24.6885827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6886205Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6886642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6887085Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6887527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.6888036Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.6888531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.6888974Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6889119Z 2025-09-07T07:10:24.6889237Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6889613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6889941Z return mod(**inputs) 2025-09-07T07:10:24.6890363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6890797Z outputs = self.bert( 2025-09-07T07:10:24.6891228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6891656Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6892083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6892508Z layer_outputs = layer_module( 2025-09-07T07:10:24.6892858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6893232Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6893658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6894106Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6894522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6894958Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6895406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6895895Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6896357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.6896794Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6896934Z 2025-09-07T07:10:24.6897047Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6897425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6897739Z return mod(**inputs) 2025-09-07T07:10:24.6898138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6898555Z outputs = self.bert( 2025-09-07T07:10:24.6898951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6899368Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6899786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6900229Z layer_outputs = layer_module( 2025-09-07T07:10:24.6900579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6900938Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6901354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6901798Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6902213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6902631Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6903092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6903579Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6904042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.6904510Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.6904897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.6905244Z return self.act(input) 2025-09-07T07:10:24.6905380Z 2025-09-07T07:10:24.6905490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6905948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6906318Z return mod(**inputs) 2025-09-07T07:10:24.6906754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6907199Z outputs = self.bert( 2025-09-07T07:10:24.6907604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6908031Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6908451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6908874Z layer_outputs = layer_module( 2025-09-07T07:10:24.6909224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6909611Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6910039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6910474Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6910878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6911264Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6911711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6912212Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6912700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.6913134Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6913272Z 2025-09-07T07:10:24.6913376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6913742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6914063Z return mod(**inputs) 2025-09-07T07:10:24.6914481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6914889Z outputs = self.bert( 2025-09-07T07:10:24.6915287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6915710Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6916130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6916553Z layer_outputs = layer_module( 2025-09-07T07:10:24.6916903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6917259Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6917680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6918106Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6918527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6918934Z self_outputs = self.self( 2025-09-07T07:10:24.6919299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6919871Z return func(*args, **kwargs) 2025-09-07T07:10:24.6920328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.6920746Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.6920890Z 2025-09-07T07:10:24.6920992Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6921353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6921677Z return mod(**inputs) 2025-09-07T07:10:24.6922073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6922494Z outputs = self.bert( 2025-09-07T07:10:24.6922901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6923326Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6923748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6924198Z layer_outputs = layer_module( 2025-09-07T07:10:24.6924536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6924905Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6925368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6925840Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6926297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6926755Z self_outputs = self.self( 2025-09-07T07:10:24.6927147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6927537Z return func(*args, **kwargs) 2025-09-07T07:10:24.6927963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.6928391Z key_layer = self.key(current_states) 2025-09-07T07:10:24.6928536Z 2025-09-07T07:10:24.6928642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6929040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6929373Z return mod(**inputs) 2025-09-07T07:10:24.6929781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6930212Z outputs = self.bert( 2025-09-07T07:10:24.6930607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6931024Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6931438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6931884Z layer_outputs = layer_module( 2025-09-07T07:10:24.6932226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6932589Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6933012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6933452Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6933874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6934294Z self_outputs = self.self( 2025-09-07T07:10:24.6934685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6935068Z return func(*args, **kwargs) 2025-09-07T07:10:24.6935490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.6935917Z value_layer = self.value(current_states) 2025-09-07T07:10:24.6936064Z 2025-09-07T07:10:24.6936149Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6936376Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6936623Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6936987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6937322Z return mod(**inputs) 2025-09-07T07:10:24.6937733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6938163Z outputs = self.bert( 2025-09-07T07:10:24.6938570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6939007Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6939428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6939852Z layer_outputs = layer_module( 2025-09-07T07:10:24.6940198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6940556Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6940974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6941461Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6941894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.6942374Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.6942855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.6943305Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6943473Z 2025-09-07T07:10:24.6943581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6943948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6944288Z return mod(**inputs) 2025-09-07T07:10:24.6944698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6945135Z outputs = self.bert( 2025-09-07T07:10:24.6945552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6946054Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6946536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6946988Z layer_outputs = layer_module( 2025-09-07T07:10:24.6947369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6947758Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6948187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6948625Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6949043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6949445Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6949903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6950388Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6950836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.6951267Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6951410Z 2025-09-07T07:10:24.6951516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6951876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6952204Z return mod(**inputs) 2025-09-07T07:10:24.6952606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6953039Z outputs = self.bert( 2025-09-07T07:10:24.6953469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6953900Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6954330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6954741Z layer_outputs = layer_module( 2025-09-07T07:10:24.6955089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6955460Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6955899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6956342Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6956750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6957157Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6957617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.6958132Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.6958587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.6959063Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.6959456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.6959811Z return self.act(input) 2025-09-07T07:10:24.6959925Z 2025-09-07T07:10:24.6960037Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6960398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6960732Z return mod(**inputs) 2025-09-07T07:10:24.6961156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6961589Z outputs = self.bert( 2025-09-07T07:10:24.6961995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6962421Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6962885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6963317Z layer_outputs = layer_module( 2025-09-07T07:10:24.6963695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6964062Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6964505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6964950Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6965365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6965772Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6966230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6966757Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6967246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.6967707Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.6967867Z 2025-09-07T07:10:24.6967977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6968330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6968655Z return mod(**inputs) 2025-09-07T07:10:24.6969052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6969466Z outputs = self.bert( 2025-09-07T07:10:24.6969863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6970275Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6970691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6971109Z layer_outputs = layer_module( 2025-09-07T07:10:24.6971454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6971816Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6972231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.6972698Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.6973114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.6973527Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.6973983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.6974487Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.6974969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.6975402Z return input_tensor + hidden_states 2025-09-07T07:10:24.6975555Z 2025-09-07T07:10:24.6975673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6976032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6976373Z return mod(**inputs) 2025-09-07T07:10:24.6976794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6977216Z outputs = self.bert( 2025-09-07T07:10:24.6977626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6978047Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6978487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6978906Z layer_outputs = layer_module( 2025-09-07T07:10:24.6979254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6979615Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6980048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6980494Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6980931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6981360Z self_outputs = self.self( 2025-09-07T07:10:24.6981745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6982166Z return func(*args, **kwargs) 2025-09-07T07:10:24.6982615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.6983059Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.6983199Z 2025-09-07T07:10:24.6983313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6983698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6984052Z return mod(**inputs) 2025-09-07T07:10:24.6984485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6984936Z outputs = self.bert( 2025-09-07T07:10:24.6985367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6985892Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6986355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6986818Z layer_outputs = layer_module( 2025-09-07T07:10:24.6987199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6987597Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6988032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6988475Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6988913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6989346Z self_outputs = self.self( 2025-09-07T07:10:24.6989715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6990104Z return func(*args, **kwargs) 2025-09-07T07:10:24.6990562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.6991007Z key_layer = self.key(current_states) 2025-09-07T07:10:24.6991147Z 2025-09-07T07:10:24.6991261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6991624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6991955Z return mod(**inputs) 2025-09-07T07:10:24.6992365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.6992798Z outputs = self.bert( 2025-09-07T07:10:24.6993217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.6993650Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.6994081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.6994511Z layer_outputs = layer_module( 2025-09-07T07:10:24.6994868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.6995233Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.6995668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.6996106Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.6996545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.6996981Z self_outputs = self.self( 2025-09-07T07:10:24.6997369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.6997756Z return func(*args, **kwargs) 2025-09-07T07:10:24.6998177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.6998618Z value_layer = self.value(current_states) 2025-09-07T07:10:24.6998757Z 2025-09-07T07:10:24.6998849Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6999076Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.6999314Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.6999675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.6999999Z return mod(**inputs) 2025-09-07T07:10:24.7000391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7000811Z outputs = self.bert( 2025-09-07T07:10:24.7001211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7001637Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7002056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7002495Z layer_outputs = layer_module( 2025-09-07T07:10:24.7002844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7003203Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7003630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7004051Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7004479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7004973Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7005466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7005914Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7006056Z 2025-09-07T07:10:24.7006161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7006533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7006854Z return mod(**inputs) 2025-09-07T07:10:24.7007252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7007682Z outputs = self.bert( 2025-09-07T07:10:24.7008071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7008491Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7008908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7009324Z layer_outputs = layer_module( 2025-09-07T07:10:24.7009671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7010025Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7010447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7010877Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7011283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7011687Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7012137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7012618Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7013070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7013496Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7013632Z 2025-09-07T07:10:24.7013737Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7014096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7014416Z return mod(**inputs) 2025-09-07T07:10:24.7014813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7015226Z outputs = self.bert( 2025-09-07T07:10:24.7015613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7016032Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7016470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7016887Z layer_outputs = layer_module( 2025-09-07T07:10:24.7017227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7017587Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7018016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7018450Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7018873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7019258Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7019840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7020347Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7020817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7021281Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7021670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7022084Z return self.act(input) 2025-09-07T07:10:24.7022215Z 2025-09-07T07:10:24.7022331Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7022733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7023083Z return mod(**inputs) 2025-09-07T07:10:24.7023510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7023968Z outputs = self.bert( 2025-09-07T07:10:24.7024404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7024869Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7025321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7025841Z layer_outputs = layer_module( 2025-09-07T07:10:24.7026242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7026693Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7027157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7027595Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7028012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7028418Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7028878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7029415Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7029901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7030352Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7030502Z 2025-09-07T07:10:24.7030610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7030988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7031351Z return mod(**inputs) 2025-09-07T07:10:24.7031753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7032185Z outputs = self.bert( 2025-09-07T07:10:24.7032594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7033029Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7033451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7033885Z layer_outputs = layer_module( 2025-09-07T07:10:24.7034263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7034636Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7035068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7035499Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7035941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7036458Z self_outputs = self.self( 2025-09-07T07:10:24.7036823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7037212Z return func(*args, **kwargs) 2025-09-07T07:10:24.7037618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7038057Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7038200Z 2025-09-07T07:10:24.7038303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7038665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7038989Z return mod(**inputs) 2025-09-07T07:10:24.7039385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7039459Z outputs = self.bert( 2025-09-07T07:10:24.7039745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7039826Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7040116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7040209Z layer_outputs = layer_module( 2025-09-07T07:10:24.7040441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7040520Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7040814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7040895Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7041193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7041269Z self_outputs = self.self( 2025-09-07T07:10:24.7041514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7041593Z return func(*args, **kwargs) 2025-09-07T07:10:24.7041880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7041964Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7041968Z 2025-09-07T07:10:24.7042068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7042293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7042365Z return mod(**inputs) 2025-09-07T07:10:24.7042654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7042725Z outputs = self.bert( 2025-09-07T07:10:24.7043011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7043084Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7043396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7043469Z layer_outputs = layer_module( 2025-09-07T07:10:24.7043705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7043783Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7044068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7044157Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7044445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7044522Z self_outputs = self.self( 2025-09-07T07:10:24.7044783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7044863Z return func(*args, **kwargs) 2025-09-07T07:10:24.7045151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7045232Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7045237Z 2025-09-07T07:10:24.7045325Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7045404Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7045515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7045712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7045778Z return mod(**inputs) 2025-09-07T07:10:24.7046080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7046146Z outputs = self.bert( 2025-09-07T07:10:24.7046464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7046538Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7046840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7046913Z layer_outputs = layer_module( 2025-09-07T07:10:24.7047139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7047237Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7047522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7047610Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7047897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7048030Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7048327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7048429Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7048433Z 2025-09-07T07:10:24.7048542Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7048742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7048813Z return mod(**inputs) 2025-09-07T07:10:24.7049100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7049164Z outputs = self.bert( 2025-09-07T07:10:24.7049458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7049531Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7049839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7049914Z layer_outputs = layer_module( 2025-09-07T07:10:24.7050135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7050220Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7050505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7050595Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7050874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7050955Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7051289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7051394Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7051690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7051775Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7051779Z 2025-09-07T07:10:24.7051890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7052092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7052160Z return mod(**inputs) 2025-09-07T07:10:24.7052464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7052548Z outputs = self.bert( 2025-09-07T07:10:24.7052848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7052923Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7053223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7053298Z layer_outputs = layer_module( 2025-09-07T07:10:24.7053525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7053614Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7053912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7054005Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7054273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7054354Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7054692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7054817Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7055117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7055234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7055459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7055529Z return self.act(input) 2025-09-07T07:10:24.7055532Z 2025-09-07T07:10:24.7055637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7055849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7055932Z return mod(**inputs) 2025-09-07T07:10:24.7056237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7056305Z outputs = self.bert( 2025-09-07T07:10:24.7056597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7056679Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7056973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7057050Z layer_outputs = layer_module( 2025-09-07T07:10:24.7057293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7057375Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7057674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7057757Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7058025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7058102Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7058433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7058568Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7058864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7058975Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7058979Z 2025-09-07T07:10:24.7059083Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7059295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7059362Z return mod(**inputs) 2025-09-07T07:10:24.7059666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7059733Z outputs = self.bert( 2025-09-07T07:10:24.7060028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7060110Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7060405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7060485Z layer_outputs = layer_module( 2025-09-07T07:10:24.7060712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7060794Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7061096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7061199Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7061470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7061547Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7061871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7062013Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7062325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7062443Z return input_tensor + hidden_states 2025-09-07T07:10:24.7062448Z 2025-09-07T07:10:24.7062561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7062786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7062859Z return mod(**inputs) 2025-09-07T07:10:24.7063176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7063257Z outputs = self.bert( 2025-09-07T07:10:24.7063574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7063663Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7063997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7064083Z layer_outputs = layer_module( 2025-09-07T07:10:24.7064323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7064408Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7064727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7064814Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7065130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7065207Z self_outputs = self.self( 2025-09-07T07:10:24.7065474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7065584Z return func(*args, **kwargs) 2025-09-07T07:10:24.7065980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7066084Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7066089Z 2025-09-07T07:10:24.7066200Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7066430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7066510Z return mod(**inputs) 2025-09-07T07:10:24.7066845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7066928Z outputs = self.bert( 2025-09-07T07:10:24.7067254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7067338Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7067648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7067721Z layer_outputs = layer_module( 2025-09-07T07:10:24.7067961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7068064Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7068362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7068445Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7068739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7068819Z self_outputs = self.self( 2025-09-07T07:10:24.7069069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7069161Z return func(*args, **kwargs) 2025-09-07T07:10:24.7069468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7069557Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7069562Z 2025-09-07T07:10:24.7069666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7069873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7069945Z return mod(**inputs) 2025-09-07T07:10:24.7070225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7070295Z outputs = self.bert( 2025-09-07T07:10:24.7070594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7070668Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7070958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7071026Z layer_outputs = layer_module( 2025-09-07T07:10:24.7071252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7071330Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7071620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7071699Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7071986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7072063Z self_outputs = self.self( 2025-09-07T07:10:24.7072320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7072398Z return func(*args, **kwargs) 2025-09-07T07:10:24.7072687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7072768Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7072772Z 2025-09-07T07:10:24.7072864Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7072943Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7073051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7073252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7073319Z return mod(**inputs) 2025-09-07T07:10:24.7073627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7073694Z outputs = self.bert( 2025-09-07T07:10:24.7074002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7074073Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7074381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7074450Z layer_outputs = layer_module( 2025-09-07T07:10:24.7074671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7074754Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7075040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7075127Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7075428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7075559Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7075861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7075947Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7075950Z 2025-09-07T07:10:24.7076062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7076267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7076340Z return mod(**inputs) 2025-09-07T07:10:24.7076679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7076748Z outputs = self.bert( 2025-09-07T07:10:24.7077054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7077129Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7077435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7077504Z layer_outputs = layer_module( 2025-09-07T07:10:24.7077718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7077798Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7078073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7078161Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7078413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7078514Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7078824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7078931Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7079218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7079298Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7079301Z 2025-09-07T07:10:24.7079408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7079603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7079668Z return mod(**inputs) 2025-09-07T07:10:24.7079962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7080029Z outputs = self.bert( 2025-09-07T07:10:24.7080317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7080403Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7080691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7080759Z layer_outputs = layer_module( 2025-09-07T07:10:24.7080971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7081055Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7081333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7081423Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7081698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7081775Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7082099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7082204Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7082497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7082613Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7082856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7082950Z return self.act(input) 2025-09-07T07:10:24.7082956Z 2025-09-07T07:10:24.7083070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7083312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7083383Z return mod(**inputs) 2025-09-07T07:10:24.7083708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7083782Z outputs = self.bert( 2025-09-07T07:10:24.7084093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7084178Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7084492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7084576Z layer_outputs = layer_module( 2025-09-07T07:10:24.7084814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7084925Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7085208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7085290Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7085567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7085644Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7085976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7086108Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7086404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7086495Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7086500Z 2025-09-07T07:10:24.7086603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7086814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7086902Z return mod(**inputs) 2025-09-07T07:10:24.7087223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7087290Z outputs = self.bert( 2025-09-07T07:10:24.7087583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7087664Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7087963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7088040Z layer_outputs = layer_module( 2025-09-07T07:10:24.7088284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7088364Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7088667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7088751Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7089056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7089126Z self_outputs = self.self( 2025-09-07T07:10:24.7089381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7089482Z return func(*args, **kwargs) 2025-09-07T07:10:24.7089780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7089868Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7089872Z 2025-09-07T07:10:24.7089975Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7090182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7090248Z return mod(**inputs) 2025-09-07T07:10:24.7090539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7090614Z outputs = self.bert( 2025-09-07T07:10:24.7090904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7090985Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7091276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7091372Z layer_outputs = layer_module( 2025-09-07T07:10:24.7091597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7091676Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7091970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7092052Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7092346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7092418Z self_outputs = self.self( 2025-09-07T07:10:24.7092667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7092748Z return func(*args, **kwargs) 2025-09-07T07:10:24.7093044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7093130Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7093134Z 2025-09-07T07:10:24.7093258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7093464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7093531Z return mod(**inputs) 2025-09-07T07:10:24.7093831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7093906Z outputs = self.bert( 2025-09-07T07:10:24.7094201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7094283Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7094593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7094667Z layer_outputs = layer_module( 2025-09-07T07:10:24.7094902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7094983Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7095286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7095368Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7095662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7095757Z self_outputs = self.self( 2025-09-07T07:10:24.7096006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7096087Z return func(*args, **kwargs) 2025-09-07T07:10:24.7096383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7096473Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7096477Z 2025-09-07T07:10:24.7096560Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7096642Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7096755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7096955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7097031Z return mod(**inputs) 2025-09-07T07:10:24.7097328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7097416Z outputs = self.bert( 2025-09-07T07:10:24.7097719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7097793Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7098098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7098172Z layer_outputs = layer_module( 2025-09-07T07:10:24.7098400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7098481Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7098758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7098847Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7099133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7099270Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7099554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7099657Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7099660Z 2025-09-07T07:10:24.7099769Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7099965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7100035Z return mod(**inputs) 2025-09-07T07:10:24.7100323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7100392Z outputs = self.bert( 2025-09-07T07:10:24.7100682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7100771Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7101065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7101138Z layer_outputs = layer_module( 2025-09-07T07:10:24.7101364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7101443Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7101735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7101828Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7102111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7102200Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7102530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7102637Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7102938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7103021Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7103024Z 2025-09-07T07:10:24.7103135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7103335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7103408Z return mod(**inputs) 2025-09-07T07:10:24.7103707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7103798Z outputs = self.bert( 2025-09-07T07:10:24.7104110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7104185Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7104492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7104564Z layer_outputs = layer_module( 2025-09-07T07:10:24.7104793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7104878Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7105179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7105273Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7105546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7105631Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7106036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7106175Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7106486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7106608Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7106844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7106921Z return self.act(input) 2025-09-07T07:10:24.7106927Z 2025-09-07T07:10:24.7107047Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7107272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7107356Z return mod(**inputs) 2025-09-07T07:10:24.7107659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7107728Z outputs = self.bert( 2025-09-07T07:10:24.7108032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7108107Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7108390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7108469Z layer_outputs = layer_module( 2025-09-07T07:10:24.7108704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7108793Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7109106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7109191Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7109460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7109537Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7109861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7109995Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7110302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7110400Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7110404Z 2025-09-07T07:10:24.7110504Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7110702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7110769Z return mod(**inputs) 2025-09-07T07:10:24.7111056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7111121Z outputs = self.bert( 2025-09-07T07:10:24.7111397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7111477Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7111754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7111832Z layer_outputs = layer_module( 2025-09-07T07:10:24.7112046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7112133Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7112419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7112519Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7112781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7112857Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7113176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7113307Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7113598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7113709Z return input_tensor + hidden_states 2025-09-07T07:10:24.7113713Z 2025-09-07T07:10:24.7113818Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7114040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7114103Z return mod(**inputs) 2025-09-07T07:10:24.7114399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7114464Z outputs = self.bert( 2025-09-07T07:10:24.7114751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7114845Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7115131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7115209Z layer_outputs = layer_module( 2025-09-07T07:10:24.7115426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7115505Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7115798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7115879Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7116172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7116242Z self_outputs = self.self( 2025-09-07T07:10:24.7116493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7116583Z return func(*args, **kwargs) 2025-09-07T07:10:24.7116868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7116955Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7116960Z 2025-09-07T07:10:24.7117060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7117261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7117327Z return mod(**inputs) 2025-09-07T07:10:24.7117612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7117684Z outputs = self.bert( 2025-09-07T07:10:24.7117975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7118056Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7118343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7118420Z layer_outputs = layer_module( 2025-09-07T07:10:24.7118636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7118731Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7119023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7119104Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7119397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7119468Z self_outputs = self.self( 2025-09-07T07:10:24.7119871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7120001Z return func(*args, **kwargs) 2025-09-07T07:10:24.7120292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7120381Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7120385Z 2025-09-07T07:10:24.7120489Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7120693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7120758Z return mod(**inputs) 2025-09-07T07:10:24.7121043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7121117Z outputs = self.bert( 2025-09-07T07:10:24.7121440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7121520Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7121808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7121878Z layer_outputs = layer_module( 2025-09-07T07:10:24.7122110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7122187Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7122486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7122567Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7122865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7122962Z self_outputs = self.self( 2025-09-07T07:10:24.7123202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7123279Z return func(*args, **kwargs) 2025-09-07T07:10:24.7123569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7123657Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7123660Z 2025-09-07T07:10:24.7123743Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7123821Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7123935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7124132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7124204Z return mod(**inputs) 2025-09-07T07:10:24.7124493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7124560Z outputs = self.bert( 2025-09-07T07:10:24.7124856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7124928Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7125248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7125317Z layer_outputs = layer_module( 2025-09-07T07:10:24.7125542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7125620Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7125920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7126011Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7126314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7126453Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7126744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7126829Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7126832Z 2025-09-07T07:10:24.7126941Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7127136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7127209Z return mod(**inputs) 2025-09-07T07:10:24.7127514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7127589Z outputs = self.bert( 2025-09-07T07:10:24.7127880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7127952Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7128247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7128319Z layer_outputs = layer_module( 2025-09-07T07:10:24.7128547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7128625Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7128916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7129009Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7129269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7129370Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7129686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7129804Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7130090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7130171Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7130174Z 2025-09-07T07:10:24.7130281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7130480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7130553Z return mod(**inputs) 2025-09-07T07:10:24.7130842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7130911Z outputs = self.bert( 2025-09-07T07:10:24.7131209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7131299Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7131593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7131662Z layer_outputs = layer_module( 2025-09-07T07:10:24.7131889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7131967Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7132253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7132346Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7132623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7132708Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7133026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7133130Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7133426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7133539Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7133777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7133847Z return self.act(input) 2025-09-07T07:10:24.7133852Z 2025-09-07T07:10:24.7133962Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7134163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7134228Z return mod(**inputs) 2025-09-07T07:10:24.7134528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7134596Z outputs = self.bert( 2025-09-07T07:10:24.7134888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7134959Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7135248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7135328Z layer_outputs = layer_module( 2025-09-07T07:10:24.7135561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7135647Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7135927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7136017Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7136278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7136353Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7136682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7136824Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7137118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7137201Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7137205Z 2025-09-07T07:10:24.7137305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7137507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7137594Z return mod(**inputs) 2025-09-07T07:10:24.7137886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7137951Z outputs = self.bert( 2025-09-07T07:10:24.7138243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7138316Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7138605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7138684Z layer_outputs = layer_module( 2025-09-07T07:10:24.7138918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7139005Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7139292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7139373Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7139669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7139738Z self_outputs = self.self( 2025-09-07T07:10:24.7139999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7140070Z return func(*args, **kwargs) 2025-09-07T07:10:24.7140371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7140456Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7140460Z 2025-09-07T07:10:24.7140559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7140759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7140823Z return mod(**inputs) 2025-09-07T07:10:24.7141110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7141173Z outputs = self.bert( 2025-09-07T07:10:24.7141458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7141538Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7141839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7141919Z layer_outputs = layer_module( 2025-09-07T07:10:24.7142137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7142217Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7142511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7142595Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7142895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7142966Z self_outputs = self.self( 2025-09-07T07:10:24.7143221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7143294Z return func(*args, **kwargs) 2025-09-07T07:10:24.7143587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7143676Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7143698Z 2025-09-07T07:10:24.7143814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7144016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7144080Z return mod(**inputs) 2025-09-07T07:10:24.7144370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7144442Z outputs = self.bert( 2025-09-07T07:10:24.7144733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7144815Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7145127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7145209Z layer_outputs = layer_module( 2025-09-07T07:10:24.7145435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7145516Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7145878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7145968Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7146280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7146381Z self_outputs = self.self( 2025-09-07T07:10:24.7146649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7146736Z return func(*args, **kwargs) 2025-09-07T07:10:24.7147049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7147138Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7147142Z 2025-09-07T07:10:24.7147223Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7147311Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7147415Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7147611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7147686Z return mod(**inputs) 2025-09-07T07:10:24.7147976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7148070Z outputs = self.bert( 2025-09-07T07:10:24.7148366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7148439Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7148748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7148823Z layer_outputs = layer_module( 2025-09-07T07:10:24.7149054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7149135Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7149438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7149526Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7149816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7149951Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7150243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7150349Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7150353Z 2025-09-07T07:10:24.7150451Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7150647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7150719Z return mod(**inputs) 2025-09-07T07:10:24.7151000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7151072Z outputs = self.bert( 2025-09-07T07:10:24.7151353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7151439Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7151736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7151807Z layer_outputs = layer_module( 2025-09-07T07:10:24.7152032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7152108Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7152404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7152484Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7152757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7152841Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7153158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7153266Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7153554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7153635Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7153646Z 2025-09-07T07:10:24.7153744Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7153939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7154009Z return mod(**inputs) 2025-09-07T07:10:24.7154292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7154382Z outputs = self.bert( 2025-09-07T07:10:24.7154662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7154734Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7155022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7155091Z layer_outputs = layer_module( 2025-09-07T07:10:24.7155312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7155387Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7155668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7155758Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7156015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7156096Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7156406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7156533Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7156811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7156922Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7157138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7157210Z return self.act(input) 2025-09-07T07:10:24.7157213Z 2025-09-07T07:10:24.7157327Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7157541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7157617Z return mod(**inputs) 2025-09-07T07:10:24.7157907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7157973Z outputs = self.bert( 2025-09-07T07:10:24.7158256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7158327Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7158614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7158684Z layer_outputs = layer_module( 2025-09-07T07:10:24.7158912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7158999Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7159279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7159365Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7159616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7159688Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7160003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7160130Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7160418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7160521Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7160524Z 2025-09-07T07:10:24.7160631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7160823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7160888Z return mod(**inputs) 2025-09-07T07:10:24.7161178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7161241Z outputs = self.bert( 2025-09-07T07:10:24.7161526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7161596Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7161879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7161955Z layer_outputs = layer_module( 2025-09-07T07:10:24.7162171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7162254Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7162531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7162635Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7162886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7162959Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7163277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7163405Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7163714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7163792Z return input_tensor + hidden_states 2025-09-07T07:10:24.7163795Z 2025-09-07T07:10:24.7163894Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7164091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7164154Z return mod(**inputs) 2025-09-07T07:10:24.7164442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7164507Z outputs = self.bert( 2025-09-07T07:10:24.7164798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7164885Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7165169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7165245Z layer_outputs = layer_module( 2025-09-07T07:10:24.7165461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7165548Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7165827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7165905Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7166190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7166258Z self_outputs = self.self( 2025-09-07T07:10:24.7166502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7166632Z return func(*args, **kwargs) 2025-09-07T07:10:24.7166927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7167007Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7167012Z 2025-09-07T07:10:24.7167123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7167319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7167381Z return mod(**inputs) 2025-09-07T07:10:24.7167671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7167734Z outputs = self.bert( 2025-09-07T07:10:24.7168027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7168104Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7168385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7168459Z layer_outputs = layer_module( 2025-09-07T07:10:24.7168687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7168768Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7169048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7169126Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7169411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7169481Z self_outputs = self.self( 2025-09-07T07:10:24.7169727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7169825Z return func(*args, **kwargs) 2025-09-07T07:10:24.7170107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7170193Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7170197Z 2025-09-07T07:10:24.7170295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7170493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7170556Z return mod(**inputs) 2025-09-07T07:10:24.7170838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7170910Z outputs = self.bert( 2025-09-07T07:10:24.7171205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7171289Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7171568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7171646Z layer_outputs = layer_module( 2025-09-07T07:10:24.7171858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7171932Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7172218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7172296Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7172584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7172668Z self_outputs = self.self( 2025-09-07T07:10:24.7172905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7172982Z return func(*args, **kwargs) 2025-09-07T07:10:24.7173261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7173349Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7173352Z 2025-09-07T07:10:24.7173433Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7173519Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7173619Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7173811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7173883Z return mod(**inputs) 2025-09-07T07:10:24.7174168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7174242Z outputs = self.bert( 2025-09-07T07:10:24.7174523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7174611Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7174898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7174967Z layer_outputs = layer_module( 2025-09-07T07:10:24.7175186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7175261Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7175596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7175683Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7175977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7176112Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7176391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7176478Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7176481Z 2025-09-07T07:10:24.7176582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7176772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7176844Z return mod(**inputs) 2025-09-07T07:10:24.7177140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7177214Z outputs = self.bert( 2025-09-07T07:10:24.7177500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7177571Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7177855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7177925Z layer_outputs = layer_module( 2025-09-07T07:10:24.7178149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7178225Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7178516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7178598Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7178866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7178949Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7179255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7179363Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7179652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7179737Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7179740Z 2025-09-07T07:10:24.7179837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7180022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7180095Z return mod(**inputs) 2025-09-07T07:10:24.7180369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7180438Z outputs = self.bert( 2025-09-07T07:10:24.7180707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7180794Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7181073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7181139Z layer_outputs = layer_module( 2025-09-07T07:10:24.7181357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7181429Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7181711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7181799Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7182065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7182149Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7182456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7182561Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7182840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7182948Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7183187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7183258Z return self.act(input) 2025-09-07T07:10:24.7183261Z 2025-09-07T07:10:24.7183378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7183565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7183628Z return mod(**inputs) 2025-09-07T07:10:24.7183914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7183977Z outputs = self.bert( 2025-09-07T07:10:24.7184260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7184330Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7184612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7184681Z layer_outputs = layer_module( 2025-09-07T07:10:24.7184910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7184998Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7185269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7185358Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7185607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7185886Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7186251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7186397Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7186721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7186812Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7186816Z 2025-09-07T07:10:24.7186937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7187181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7187248Z return mod(**inputs) 2025-09-07T07:10:24.7187547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7187615Z outputs = self.bert( 2025-09-07T07:10:24.7187910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7187985Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7188272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7188369Z layer_outputs = layer_module( 2025-09-07T07:10:24.7188598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7188683Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7188963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7189052Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7189327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7189395Z self_outputs = self.self( 2025-09-07T07:10:24.7189655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7189728Z return func(*args, **kwargs) 2025-09-07T07:10:24.7190013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7190092Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7190096Z 2025-09-07T07:10:24.7190195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7190391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7190455Z return mod(**inputs) 2025-09-07T07:10:24.7190743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7190807Z outputs = self.bert( 2025-09-07T07:10:24.7191093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7191163Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7191464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7191543Z layer_outputs = layer_module( 2025-09-07T07:10:24.7191760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7191854Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7192143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7192222Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7192504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7192573Z self_outputs = self.self( 2025-09-07T07:10:24.7192819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7192891Z return func(*args, **kwargs) 2025-09-07T07:10:24.7193193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7193268Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7193286Z 2025-09-07T07:10:24.7193385Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7193586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7193649Z return mod(**inputs) 2025-09-07T07:10:24.7193939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7194004Z outputs = self.bert( 2025-09-07T07:10:24.7194286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7194367Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7194663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7194742Z layer_outputs = layer_module( 2025-09-07T07:10:24.7194963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7195046Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7195323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7195401Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7195713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7195781Z self_outputs = self.self( 2025-09-07T07:10:24.7196033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7196103Z return func(*args, **kwargs) 2025-09-07T07:10:24.7196387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7196476Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7196480Z 2025-09-07T07:10:24.7196561Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7196647Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7196749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7196945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7197016Z return mod(**inputs) 2025-09-07T07:10:24.7197312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7197411Z outputs = self.bert( 2025-09-07T07:10:24.7197694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7197770Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7198052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7198119Z layer_outputs = layer_module( 2025-09-07T07:10:24.7198343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7198417Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7198710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7198788Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7199074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7199206Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7199487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7199591Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7199595Z 2025-09-07T07:10:24.7199692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7199891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7199955Z return mod(**inputs) 2025-09-07T07:10:24.7200237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7200309Z outputs = self.bert( 2025-09-07T07:10:24.7200601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7200681Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7200965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7201034Z layer_outputs = layer_module( 2025-09-07T07:10:24.7201256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7201332Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7201622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7201704Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7201979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7202057Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7202364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7202474Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7202752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7202837Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7202841Z 2025-09-07T07:10:24.7202941Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7203133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7203203Z return mod(**inputs) 2025-09-07T07:10:24.7203486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7203579Z outputs = self.bert( 2025-09-07T07:10:24.7203861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7203942Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7204226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7204296Z layer_outputs = layer_module( 2025-09-07T07:10:24.7204524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7204600Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7204894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7204981Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7205245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7205327Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7205646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7205773Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7206055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7206176Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7206385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7206457Z return self.act(input) 2025-09-07T07:10:24.7206462Z 2025-09-07T07:10:24.7206570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7206786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7206860Z return mod(**inputs) 2025-09-07T07:10:24.7207156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7207223Z outputs = self.bert( 2025-09-07T07:10:24.7207515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7207587Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7207886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7207972Z layer_outputs = layer_module( 2025-09-07T07:10:24.7208198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7208276Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7208559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7208658Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7208909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7208989Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7209297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7209423Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7209709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7209806Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7209810Z 2025-09-07T07:10:24.7209917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7210108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7210183Z return mod(**inputs) 2025-09-07T07:10:24.7210463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7210526Z outputs = self.bert( 2025-09-07T07:10:24.7210809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7210880Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7211162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7211233Z layer_outputs = layer_module( 2025-09-07T07:10:24.7211446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7211528Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7211817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7211905Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7212157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7212237Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7212544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7212669Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7212968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7213045Z return input_tensor + hidden_states 2025-09-07T07:10:24.7213049Z 2025-09-07T07:10:24.7213155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7213348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7213412Z return mod(**inputs) 2025-09-07T07:10:24.7213700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7213763Z outputs = self.bert( 2025-09-07T07:10:24.7214064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7214135Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7214419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7214486Z layer_outputs = layer_module( 2025-09-07T07:10:24.7214699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7214781Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7215059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7215144Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7215417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7215484Z self_outputs = self.self( 2025-09-07T07:10:24.7215731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7215814Z return func(*args, **kwargs) 2025-09-07T07:10:24.7216102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7216182Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7216186Z 2025-09-07T07:10:24.7216291Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7216484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7216546Z return mod(**inputs) 2025-09-07T07:10:24.7216832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7216895Z outputs = self.bert( 2025-09-07T07:10:24.7217180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7217251Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7217528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7217602Z layer_outputs = layer_module( 2025-09-07T07:10:24.7217829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7217910Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7218190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7218269Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7218556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7218624Z self_outputs = self.self( 2025-09-07T07:10:24.7218883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7218954Z return func(*args, **kwargs) 2025-09-07T07:10:24.7219238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7219317Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7219321Z 2025-09-07T07:10:24.7219420Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7219761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7219834Z return mod(**inputs) 2025-09-07T07:10:24.7220133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7220246Z outputs = self.bert( 2025-09-07T07:10:24.7220542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7220632Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7220927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7221011Z layer_outputs = layer_module( 2025-09-07T07:10:24.7221240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7221327Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7221625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7221708Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7222012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7222112Z self_outputs = self.self( 2025-09-07T07:10:24.7222380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7222451Z return func(*args, **kwargs) 2025-09-07T07:10:24.7222738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7222827Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7222832Z 2025-09-07T07:10:24.7222917Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7223005Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7223109Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7223310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7223386Z return mod(**inputs) 2025-09-07T07:10:24.7223686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7223761Z outputs = self.bert( 2025-09-07T07:10:24.7224053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7224158Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7224453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7224524Z layer_outputs = layer_module( 2025-09-07T07:10:24.7224758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7224835Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7225139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7225222Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7225538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7225723Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7226027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7226121Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7226125Z 2025-09-07T07:10:24.7226229Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7226438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7226509Z return mod(**inputs) 2025-09-07T07:10:24.7226840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7226922Z outputs = self.bert( 2025-09-07T07:10:24.7227239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7227326Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7227649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7227734Z layer_outputs = layer_module( 2025-09-07T07:10:24.7227969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7228047Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7228356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7228442Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7228733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7228812Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7229140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7229256Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7229550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7229639Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7229642Z 2025-09-07T07:10:24.7229745Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7229958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7230027Z return mod(**inputs) 2025-09-07T07:10:24.7230330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7230407Z outputs = self.bert( 2025-09-07T07:10:24.7230703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7230816Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7231112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7231183Z layer_outputs = layer_module( 2025-09-07T07:10:24.7231418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7231498Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7231798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7231902Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7232168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7232253Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7232574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7232686Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7232979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7233101Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7233334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7233410Z return self.act(input) 2025-09-07T07:10:24.7233413Z 2025-09-07T07:10:24.7233529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7233735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7233812Z return mod(**inputs) 2025-09-07T07:10:24.7234112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7234178Z outputs = self.bert( 2025-09-07T07:10:24.7234479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7234552Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7234855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7234945Z layer_outputs = layer_module( 2025-09-07T07:10:24.7235178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7235259Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7235561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7235650Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7235901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7235985Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7236303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7236433Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7236724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7236804Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7236807Z 2025-09-07T07:10:24.7236913Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7237121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7237191Z return mod(**inputs) 2025-09-07T07:10:24.7237474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7237537Z outputs = self.bert( 2025-09-07T07:10:24.7237825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7237898Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7238185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7238269Z layer_outputs = layer_module( 2025-09-07T07:10:24.7238482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7238567Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7238840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7238927Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7239213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7239291Z self_outputs = self.self( 2025-09-07T07:10:24.7239552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7239629Z return func(*args, **kwargs) 2025-09-07T07:10:24.7239931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7240014Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7240020Z 2025-09-07T07:10:24.7240132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7240333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7240400Z return mod(**inputs) 2025-09-07T07:10:24.7240703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7240769Z outputs = self.bert( 2025-09-07T07:10:24.7241069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7241165Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7241450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7241518Z layer_outputs = layer_module( 2025-09-07T07:10:24.7241731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7241818Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7242097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7242184Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7242460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7242529Z self_outputs = self.self( 2025-09-07T07:10:24.7242775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7242846Z return func(*args, **kwargs) 2025-09-07T07:10:24.7243133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7243226Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7243230Z 2025-09-07T07:10:24.7243335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7243525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7243589Z return mod(**inputs) 2025-09-07T07:10:24.7243874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7243936Z outputs = self.bert( 2025-09-07T07:10:24.7244221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7244291Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7244586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7245037Z layer_outputs = layer_module( 2025-09-07T07:10:24.7245386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7245746Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7246164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7246583Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7247039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7247452Z self_outputs = self.self( 2025-09-07T07:10:24.7247807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7248169Z return func(*args, **kwargs) 2025-09-07T07:10:24.7248565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7248992Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7249142Z 2025-09-07T07:10:24.7249231Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7249444Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7249673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7250040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7250362Z return mod(**inputs) 2025-09-07T07:10:24.7250769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7251205Z outputs = self.bert( 2025-09-07T07:10:24.7251597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7252117Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7252592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7253007Z layer_outputs = layer_module( 2025-09-07T07:10:24.7253352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7253709Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7254125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7254537Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7254958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7255422Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7255895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7256347Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7256485Z 2025-09-07T07:10:24.7256588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7256939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7257260Z return mod(**inputs) 2025-09-07T07:10:24.7257652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7258071Z outputs = self.bert( 2025-09-07T07:10:24.7258483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7258911Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7259334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7259754Z layer_outputs = layer_module( 2025-09-07T07:10:24.7260103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7260456Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7260889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7261339Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7261760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7262170Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7262646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7263152Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7263627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7264078Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7264224Z 2025-09-07T07:10:24.7264333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7264717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7265063Z return mod(**inputs) 2025-09-07T07:10:24.7265506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7266017Z outputs = self.bert( 2025-09-07T07:10:24.7266427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7266890Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7267357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7267781Z layer_outputs = layer_module( 2025-09-07T07:10:24.7268135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7268489Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7268921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7269356Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7269763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7270156Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7270631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7271114Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7271564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7272023Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7272398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7272739Z return self.act(input) 2025-09-07T07:10:24.7272858Z 2025-09-07T07:10:24.7272965Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7273341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7273670Z return mod(**inputs) 2025-09-07T07:10:24.7274063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7274489Z outputs = self.bert( 2025-09-07T07:10:24.7274883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7275304Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7275723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7276160Z layer_outputs = layer_module( 2025-09-07T07:10:24.7276514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7276887Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7277312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7277735Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7278136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7278525Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7278971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7279468Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7279935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7280382Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7280523Z 2025-09-07T07:10:24.7280625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7280975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7281293Z return mod(**inputs) 2025-09-07T07:10:24.7281671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7282077Z outputs = self.bert( 2025-09-07T07:10:24.7282459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7282866Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7283274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7283674Z layer_outputs = layer_module( 2025-09-07T07:10:24.7284014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7284367Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7284802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7285216Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7285605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7285987Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7286427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7286931Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7287423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7287854Z return input_tensor + hidden_states 2025-09-07T07:10:24.7287996Z 2025-09-07T07:10:24.7288102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7288464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7288789Z return mod(**inputs) 2025-09-07T07:10:24.7289174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7289594Z outputs = self.bert( 2025-09-07T07:10:24.7290040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7290450Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7290861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7291264Z layer_outputs = layer_module( 2025-09-07T07:10:24.7291606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7291964Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7292382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7292797Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7293217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7293630Z self_outputs = self.self( 2025-09-07T07:10:24.7294010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7294375Z return func(*args, **kwargs) 2025-09-07T07:10:24.7294767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7295188Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7295328Z 2025-09-07T07:10:24.7295433Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7295792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7296113Z return mod(**inputs) 2025-09-07T07:10:24.7296505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7296920Z outputs = self.bert( 2025-09-07T07:10:24.7297368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7297779Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7298187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7298634Z layer_outputs = layer_module( 2025-09-07T07:10:24.7298981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7299345Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7299772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7300202Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7300641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7301066Z self_outputs = self.self( 2025-09-07T07:10:24.7301456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7301836Z return func(*args, **kwargs) 2025-09-07T07:10:24.7302244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7302675Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7302816Z 2025-09-07T07:10:24.7302921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7303282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7303601Z return mod(**inputs) 2025-09-07T07:10:24.7304034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7304467Z outputs = self.bert( 2025-09-07T07:10:24.7304877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7305317Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7305814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7306281Z layer_outputs = layer_module( 2025-09-07T07:10:24.7306649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7307027Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7307470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7307915Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7308367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7308832Z self_outputs = self.self( 2025-09-07T07:10:24.7309205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7309592Z return func(*args, **kwargs) 2025-09-07T07:10:24.7310006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7310435Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7310573Z 2025-09-07T07:10:24.7310665Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7310889Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7311119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7311488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7311819Z return mod(**inputs) 2025-09-07T07:10:24.7312223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7312641Z outputs = self.bert( 2025-09-07T07:10:24.7313029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7313492Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7313914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7314332Z layer_outputs = layer_module( 2025-09-07T07:10:24.7314672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7315030Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7315451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7315887Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7316351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7316837Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7317327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7317773Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7317917Z 2025-09-07T07:10:24.7318032Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7318420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7318739Z return mod(**inputs) 2025-09-07T07:10:24.7319179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7319909Z outputs = self.bert( 2025-09-07T07:10:24.7320340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7320791Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7321229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7321662Z layer_outputs = layer_module( 2025-09-07T07:10:24.7322034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7322410Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7322850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7323358Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7323771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7324181Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7324647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7325153Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7325617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7326066Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7326218Z 2025-09-07T07:10:24.7326328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7326707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7327047Z return mod(**inputs) 2025-09-07T07:10:24.7327466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7327889Z outputs = self.bert( 2025-09-07T07:10:24.7328297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7328763Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7329189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7329612Z layer_outputs = layer_module( 2025-09-07T07:10:24.7329982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7330356Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7330790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7331265Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7331673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7332096Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7332549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7333033Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7333482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7333939Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7334352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7334716Z return self.act(input) 2025-09-07T07:10:24.7334829Z 2025-09-07T07:10:24.7334949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7335324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7335657Z return mod(**inputs) 2025-09-07T07:10:24.7336071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7336518Z outputs = self.bert( 2025-09-07T07:10:24.7336932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7337363Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7337799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7338265Z layer_outputs = layer_module( 2025-09-07T07:10:24.7338623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7339001Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7339433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7339880Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7340303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7340715Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7341185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7341714Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7342224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7342706Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7342857Z 2025-09-07T07:10:24.7342989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7343385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7343720Z return mod(**inputs) 2025-09-07T07:10:24.7344134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7344562Z outputs = self.bert( 2025-09-07T07:10:24.7344991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7345446Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7346003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7346483Z layer_outputs = layer_module( 2025-09-07T07:10:24.7346870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7347250Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7347689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7348152Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7348589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7349031Z self_outputs = self.self( 2025-09-07T07:10:24.7349436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7349821Z return func(*args, **kwargs) 2025-09-07T07:10:24.7350253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7350699Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7350845Z 2025-09-07T07:10:24.7350961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7351324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7351660Z return mod(**inputs) 2025-09-07T07:10:24.7352073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7352505Z outputs = self.bert( 2025-09-07T07:10:24.7352910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7353365Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7353805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7354236Z layer_outputs = layer_module( 2025-09-07T07:10:24.7354602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7354973Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7355407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7355850Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7356293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7356724Z self_outputs = self.self( 2025-09-07T07:10:24.7357092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7357480Z return func(*args, **kwargs) 2025-09-07T07:10:24.7357899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7358365Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7358500Z 2025-09-07T07:10:24.7358612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7358972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7359311Z return mod(**inputs) 2025-09-07T07:10:24.7359700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7360109Z outputs = self.bert( 2025-09-07T07:10:24.7360493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7360918Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7361341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7361759Z layer_outputs = layer_module( 2025-09-07T07:10:24.7362112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7362466Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7362892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7363322Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7363771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7364193Z self_outputs = self.self( 2025-09-07T07:10:24.7364540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7364908Z return func(*args, **kwargs) 2025-09-07T07:10:24.7365305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7365724Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7365859Z 2025-09-07T07:10:24.7365946Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7366160Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7366395Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7366762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7367095Z return mod(**inputs) 2025-09-07T07:10:24.7367496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7367947Z outputs = self.bert( 2025-09-07T07:10:24.7368364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7368787Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7369206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7369616Z layer_outputs = layer_module( 2025-09-07T07:10:24.7369964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7370324Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7370751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7371186Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7371614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7372109Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7372619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7373058Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7373191Z 2025-09-07T07:10:24.7373298Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7373644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7373959Z return mod(**inputs) 2025-09-07T07:10:24.7374350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7374768Z outputs = self.bert( 2025-09-07T07:10:24.7375182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7375609Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7376038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7376473Z layer_outputs = layer_module( 2025-09-07T07:10:24.7376821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7377176Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7377660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7378102Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7378512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7378912Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7379364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7379852Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7380313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7380749Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7380888Z 2025-09-07T07:10:24.7381000Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7381362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7381699Z return mod(**inputs) 2025-09-07T07:10:24.7382150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7382601Z outputs = self.bert( 2025-09-07T07:10:24.7383025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7383481Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7383936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7384393Z layer_outputs = layer_module( 2025-09-07T07:10:24.7384765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7385143Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7385603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7386159Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7386619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7387070Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7387545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7388028Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7388481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7388953Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7389347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7389690Z return self.act(input) 2025-09-07T07:10:24.7389814Z 2025-09-07T07:10:24.7389940Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7390316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7390663Z return mod(**inputs) 2025-09-07T07:10:24.7391061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7391482Z outputs = self.bert( 2025-09-07T07:10:24.7391881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7392301Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7392771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7393220Z layer_outputs = layer_module( 2025-09-07T07:10:24.7393606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7394001Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7394464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7394953Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7395399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7395832Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7396381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7396941Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7397484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7397966Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7398128Z 2025-09-07T07:10:24.7398244Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7398649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7399008Z return mod(**inputs) 2025-09-07T07:10:24.7399442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7399901Z outputs = self.bert( 2025-09-07T07:10:24.7400346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7400817Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7401280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7401741Z layer_outputs = layer_module( 2025-09-07T07:10:24.7402130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7402551Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7403013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7403483Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7403909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7404338Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7404828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7405378Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7405906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7406373Z return input_tensor + hidden_states 2025-09-07T07:10:24.7406527Z 2025-09-07T07:10:24.7406645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7407006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7407333Z return mod(**inputs) 2025-09-07T07:10:24.7407734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7408164Z outputs = self.bert( 2025-09-07T07:10:24.7408580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7409005Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7409422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7409833Z layer_outputs = layer_module( 2025-09-07T07:10:24.7410186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7410549Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7410975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7411404Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7411826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7412247Z self_outputs = self.self( 2025-09-07T07:10:24.7412635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7413019Z return func(*args, **kwargs) 2025-09-07T07:10:24.7413442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7413870Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7414012Z 2025-09-07T07:10:24.7414114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7414473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7414793Z return mod(**inputs) 2025-09-07T07:10:24.7415185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7415602Z outputs = self.bert( 2025-09-07T07:10:24.7415995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7416421Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7416845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7417275Z layer_outputs = layer_module( 2025-09-07T07:10:24.7417620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7417979Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7418399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7418821Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7419247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7419852Z self_outputs = self.self( 2025-09-07T07:10:24.7420269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7420652Z return func(*args, **kwargs) 2025-09-07T07:10:24.7421086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7421527Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7421673Z 2025-09-07T07:10:24.7421778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7422154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7422485Z return mod(**inputs) 2025-09-07T07:10:24.7422930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7423392Z outputs = self.bert( 2025-09-07T07:10:24.7423828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7424287Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7424737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7425194Z layer_outputs = layer_module( 2025-09-07T07:10:24.7425571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7426022Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7426497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7426970Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7427442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7427915Z self_outputs = self.self( 2025-09-07T07:10:24.7428291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7428689Z return func(*args, **kwargs) 2025-09-07T07:10:24.7429109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7429553Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7429702Z 2025-09-07T07:10:24.7429788Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7430012Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7430251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7430631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7430970Z return mod(**inputs) 2025-09-07T07:10:24.7431386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7431814Z outputs = self.bert( 2025-09-07T07:10:24.7432215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7432692Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7433124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7433572Z layer_outputs = layer_module( 2025-09-07T07:10:24.7433932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7434298Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7434737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7435197Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7435633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7436122Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7436590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7437025Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7437159Z 2025-09-07T07:10:24.7437269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7437616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7437944Z return mod(**inputs) 2025-09-07T07:10:24.7438342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7438809Z outputs = self.bert( 2025-09-07T07:10:24.7439194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7439605Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7440007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7440439Z layer_outputs = layer_module( 2025-09-07T07:10:24.7440770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7441119Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7441565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7442005Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7442416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7442805Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7443251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7443716Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7444164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7444587Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7444719Z 2025-09-07T07:10:24.7444827Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7445179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7445492Z return mod(**inputs) 2025-09-07T07:10:24.7445889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7446301Z outputs = self.bert( 2025-09-07T07:10:24.7446718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7447137Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7447550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7447970Z layer_outputs = layer_module( 2025-09-07T07:10:24.7448201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7448285Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7448575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7448689Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7448955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7449043Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7449360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7449464Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7449757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7449887Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7450107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7450180Z return self.act(input) 2025-09-07T07:10:24.7450185Z 2025-09-07T07:10:24.7450299Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7450500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7450569Z return mod(**inputs) 2025-09-07T07:10:24.7450864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7450930Z outputs = self.bert( 2025-09-07T07:10:24.7451231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7451302Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7451581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7451682Z layer_outputs = layer_module( 2025-09-07T07:10:24.7451896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7451982Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7452262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7452351Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7452608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7452684Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7453013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7453143Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7453434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7453515Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7453518Z 2025-09-07T07:10:24.7453643Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7453837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7453900Z return mod(**inputs) 2025-09-07T07:10:24.7454193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7454259Z outputs = self.bert( 2025-09-07T07:10:24.7454554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7454629Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7454937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7455018Z layer_outputs = layer_module( 2025-09-07T07:10:24.7455239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7455329Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7455619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7455701Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7455994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7456079Z self_outputs = self.self( 2025-09-07T07:10:24.7456332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7456405Z return func(*args, **kwargs) 2025-09-07T07:10:24.7456698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7456780Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7456783Z 2025-09-07T07:10:24.7456887Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7457089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7457154Z return mod(**inputs) 2025-09-07T07:10:24.7457447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7457511Z outputs = self.bert( 2025-09-07T07:10:24.7457797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7457892Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7458176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7458252Z layer_outputs = layer_module( 2025-09-07T07:10:24.7458475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7458558Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7458843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7458923Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7459217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7459287Z self_outputs = self.self( 2025-09-07T07:10:24.7459537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7459608Z return func(*args, **kwargs) 2025-09-07T07:10:24.7459894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7459997Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7460001Z 2025-09-07T07:10:24.7460102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7460304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7460370Z return mod(**inputs) 2025-09-07T07:10:24.7460663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7460730Z outputs = self.bert( 2025-09-07T07:10:24.7461017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7461114Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7461400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7461481Z layer_outputs = layer_module( 2025-09-07T07:10:24.7461702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7461778Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7462073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7462155Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7462479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7462555Z self_outputs = self.self( 2025-09-07T07:10:24.7462807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7462887Z return func(*args, **kwargs) 2025-09-07T07:10:24.7463184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7463274Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7463277Z 2025-09-07T07:10:24.7463363Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7463453Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7463557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7463758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7463834Z return mod(**inputs) 2025-09-07T07:10:24.7464159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7464238Z outputs = self.bert( 2025-09-07T07:10:24.7464534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7464612Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7464917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7464992Z layer_outputs = layer_module( 2025-09-07T07:10:24.7465222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7465299Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7465627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7465793Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7466118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7466270Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7466609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7466720Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7466724Z 2025-09-07T07:10:24.7466843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7467046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7467124Z return mod(**inputs) 2025-09-07T07:10:24.7467462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7467541Z outputs = self.bert( 2025-09-07T07:10:24.7467852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7467935Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7468224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7468296Z layer_outputs = layer_module( 2025-09-07T07:10:24.7468526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7468602Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7468907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7468993Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7469257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7469340Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7469656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7469769Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7470056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7470144Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7470148Z 2025-09-07T07:10:24.7470251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7470449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7470544Z return mod(**inputs) 2025-09-07T07:10:24.7470845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7470918Z outputs = self.bert( 2025-09-07T07:10:24.7471210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7471283Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7471582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7471654Z layer_outputs = layer_module( 2025-09-07T07:10:24.7471885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7471960Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7472264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7472349Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7472614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7472715Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7473035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7473143Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7473434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7473548Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7473772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7473843Z return self.act(input) 2025-09-07T07:10:24.7473846Z 2025-09-07T07:10:24.7473971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7474172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7474247Z return mod(**inputs) 2025-09-07T07:10:24.7474543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7474611Z outputs = self.bert( 2025-09-07T07:10:24.7474910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7474984Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7475301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7475377Z layer_outputs = layer_module( 2025-09-07T07:10:24.7475612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7475699Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7476009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7476098Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7476363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7476437Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7476768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7476901Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7477209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7477292Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7477295Z 2025-09-07T07:10:24.7477403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7477601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7477665Z return mod(**inputs) 2025-09-07T07:10:24.7477956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7478022Z outputs = self.bert( 2025-09-07T07:10:24.7478312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7478386Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7478675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7478747Z layer_outputs = layer_module( 2025-09-07T07:10:24.7478965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7479072Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7479419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7479506Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7479758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7479833Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7480159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7480350Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7480645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7480724Z return input_tensor + hidden_states 2025-09-07T07:10:24.7480729Z 2025-09-07T07:10:24.7480838Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7481038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7481105Z return mod(**inputs) 2025-09-07T07:10:24.7481399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7481465Z outputs = self.bert( 2025-09-07T07:10:24.7481778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7481854Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7482145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7482223Z layer_outputs = layer_module( 2025-09-07T07:10:24.7482451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7482532Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7482828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7482917Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7483207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7483311Z self_outputs = self.self( 2025-09-07T07:10:24.7483565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7483636Z return func(*args, **kwargs) 2025-09-07T07:10:24.7483930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7484011Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7484015Z 2025-09-07T07:10:24.7484115Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7484320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7484385Z return mod(**inputs) 2025-09-07T07:10:24.7484681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7484747Z outputs = self.bert( 2025-09-07T07:10:24.7485043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7485126Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7485421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7485525Z layer_outputs = layer_module( 2025-09-07T07:10:24.7485767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7485859Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7486191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7486273Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7486580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7486656Z self_outputs = self.self( 2025-09-07T07:10:24.7486932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7487006Z return func(*args, **kwargs) 2025-09-07T07:10:24.7487303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7487390Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7487394Z 2025-09-07T07:10:24.7487498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7487711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7487776Z return mod(**inputs) 2025-09-07T07:10:24.7488103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7488173Z outputs = self.bert( 2025-09-07T07:10:24.7488527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7488610Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7488910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7488992Z layer_outputs = layer_module( 2025-09-07T07:10:24.7489216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7489296Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7489599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7489684Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7489996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7490066Z self_outputs = self.self( 2025-09-07T07:10:24.7490320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7490392Z return func(*args, **kwargs) 2025-09-07T07:10:24.7490686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7490773Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7490777Z 2025-09-07T07:10:24.7490859Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7490948Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7491053Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7491254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7491329Z return mod(**inputs) 2025-09-07T07:10:24.7491627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7491701Z outputs = self.bert( 2025-09-07T07:10:24.7491994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7492087Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7492389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7492462Z layer_outputs = layer_module( 2025-09-07T07:10:24.7492695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7492776Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7493078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7493179Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7493470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7493613Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7493910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7494002Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7494005Z 2025-09-07T07:10:24.7494108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7494310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7494404Z return mod(**inputs) 2025-09-07T07:10:24.7494700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7494776Z outputs = self.bert( 2025-09-07T07:10:24.7495068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7495151Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7495444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7495517Z layer_outputs = layer_module( 2025-09-07T07:10:24.7495751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7495828Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7496134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7496239Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7496509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7496594Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7496915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7497030Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7497326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7497415Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7497419Z 2025-09-07T07:10:24.7497523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7497729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7497807Z return mod(**inputs) 2025-09-07T07:10:24.7498107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7498182Z outputs = self.bert( 2025-09-07T07:10:24.7498492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7498566Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7498875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7498944Z layer_outputs = layer_module( 2025-09-07T07:10:24.7499174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7499251Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7499560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7499643Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7499906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7499991Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7500306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7500415Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7500699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7500830Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7501047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7501118Z return self.act(input) 2025-09-07T07:10:24.7501121Z 2025-09-07T07:10:24.7501230Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7501430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7501506Z return mod(**inputs) 2025-09-07T07:10:24.7501800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7501867Z outputs = self.bert( 2025-09-07T07:10:24.7502166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7502240Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7502540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7502633Z layer_outputs = layer_module( 2025-09-07T07:10:24.7502859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7502948Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7503244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7503336Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7503601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7503685Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7504011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7504145Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7504450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7504533Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7504554Z 2025-09-07T07:10:24.7504666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7504867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7504933Z return mod(**inputs) 2025-09-07T07:10:24.7505253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7505322Z outputs = self.bert( 2025-09-07T07:10:24.7505638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7505784Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7506138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7506217Z layer_outputs = layer_module( 2025-09-07T07:10:24.7506457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7506552Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7506870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7506959Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7507242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7507331Z self_outputs = self.self( 2025-09-07T07:10:24.7507586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7507657Z return func(*args, **kwargs) 2025-09-07T07:10:24.7507951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7508035Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7508038Z 2025-09-07T07:10:24.7508147Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7508343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7508410Z return mod(**inputs) 2025-09-07T07:10:24.7508706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7508770Z outputs = self.bert( 2025-09-07T07:10:24.7509063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7509156Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7509443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7509522Z layer_outputs = layer_module( 2025-09-07T07:10:24.7509742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7509827Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7510109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7510198Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7510483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7510554Z self_outputs = self.self( 2025-09-07T07:10:24.7510801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7510871Z return func(*args, **kwargs) 2025-09-07T07:10:24.7511162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7511260Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7511264Z 2025-09-07T07:10:24.7511368Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7511579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7511645Z return mod(**inputs) 2025-09-07T07:10:24.7511954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7512020Z outputs = self.bert( 2025-09-07T07:10:24.7512330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7512412Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7512695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7512777Z layer_outputs = layer_module( 2025-09-07T07:10:24.7512996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7513081Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7513365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7513448Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7513759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7513832Z self_outputs = self.self( 2025-09-07T07:10:24.7514081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7514151Z return func(*args, **kwargs) 2025-09-07T07:10:24.7514436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7514523Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7514527Z 2025-09-07T07:10:24.7514608Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7514693Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7514794Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7514999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7515063Z return mod(**inputs) 2025-09-07T07:10:24.7515369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7515443Z outputs = self.bert( 2025-09-07T07:10:24.7515730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7515812Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7516098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7516169Z layer_outputs = layer_module( 2025-09-07T07:10:24.7516403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7516480Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7516772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7516854Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7517142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7517294Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7517590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7517676Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7517680Z 2025-09-07T07:10:24.7517777Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7517982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7518045Z return mod(**inputs) 2025-09-07T07:10:24.7518328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7518402Z outputs = self.bert( 2025-09-07T07:10:24.7518708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7518790Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7519075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7519146Z layer_outputs = layer_module( 2025-09-07T07:10:24.7519372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7519447Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7519968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7520070Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7520331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7520407Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7520711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7520823Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7521109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7521196Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7521200Z 2025-09-07T07:10:24.7521298Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7521502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7521594Z return mod(**inputs) 2025-09-07T07:10:24.7521877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7521950Z outputs = self.bert( 2025-09-07T07:10:24.7522229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7522309Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7522588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7522657Z layer_outputs = layer_module( 2025-09-07T07:10:24.7522879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7522954Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7523239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7523324Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7523575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7523681Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7523987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7524095Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7524373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7524491Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7524697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7524765Z return self.act(input) 2025-09-07T07:10:24.7524795Z 2025-09-07T07:10:24.7524903Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7525097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7525171Z return mod(**inputs) 2025-09-07T07:10:24.7525459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7525523Z outputs = self.bert( 2025-09-07T07:10:24.7525812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7525883Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7526185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7526257Z layer_outputs = layer_module( 2025-09-07T07:10:24.7526479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7526556Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7526835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7526929Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7527180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7527261Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7527571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7527698Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7528007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7528086Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7528089Z 2025-09-07T07:10:24.7528196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7528389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7528460Z return mod(**inputs) 2025-09-07T07:10:24.7528742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7528806Z outputs = self.bert( 2025-09-07T07:10:24.7529097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7529168Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7529461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7529530Z layer_outputs = layer_module( 2025-09-07T07:10:24.7529745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7529848Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7530125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7530220Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7530470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7530550Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7530858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7531000Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7531331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7531410Z return input_tensor + hidden_states 2025-09-07T07:10:24.7531414Z 2025-09-07T07:10:24.7531522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7531721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7531792Z return mod(**inputs) 2025-09-07T07:10:24.7532083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7532149Z outputs = self.bert( 2025-09-07T07:10:24.7532461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7532538Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7532831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7532903Z layer_outputs = layer_module( 2025-09-07T07:10:24.7533133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7533219Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7533498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7533584Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7533871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7533959Z self_outputs = self.self( 2025-09-07T07:10:24.7534212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7534283Z return func(*args, **kwargs) 2025-09-07T07:10:24.7534576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7534659Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7534662Z 2025-09-07T07:10:24.7534769Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7534968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7535031Z return mod(**inputs) 2025-09-07T07:10:24.7535330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7535396Z outputs = self.bert( 2025-09-07T07:10:24.7535688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7535761Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7536049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7536154Z layer_outputs = layer_module( 2025-09-07T07:10:24.7536371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7536455Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7536741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7536828Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7537114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7537207Z self_outputs = self.self( 2025-09-07T07:10:24.7537455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7537527Z return func(*args, **kwargs) 2025-09-07T07:10:24.7537817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7537895Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7537899Z 2025-09-07T07:10:24.7538000Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7538202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7538267Z return mod(**inputs) 2025-09-07T07:10:24.7538578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7538646Z outputs = self.bert( 2025-09-07T07:10:24.7538936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7539007Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7539293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7539371Z layer_outputs = layer_module( 2025-09-07T07:10:24.7539589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7539673Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7539957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7540037Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7540349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7540419Z self_outputs = self.self( 2025-09-07T07:10:24.7540668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7540739Z return func(*args, **kwargs) 2025-09-07T07:10:24.7541025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7541113Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7541116Z 2025-09-07T07:10:24.7541195Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7541283Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7541384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7541592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7541660Z return mod(**inputs) 2025-09-07T07:10:24.7541949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7542022Z outputs = self.bert( 2025-09-07T07:10:24.7542322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7542402Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7542686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7542755Z layer_outputs = layer_module( 2025-09-07T07:10:24.7542983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7543063Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7543374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7543457Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7543755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7543884Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7544178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7544268Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7544271Z 2025-09-07T07:10:24.7544373Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7544596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7544664Z return mod(**inputs) 2025-09-07T07:10:24.7544959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7545031Z outputs = self.bert( 2025-09-07T07:10:24.7545320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7545400Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7545747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7545835Z layer_outputs = layer_module( 2025-09-07T07:10:24.7546061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7546138Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7546443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7546548Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7546823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7546903Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7547237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7547350Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7547638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7547729Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7547732Z 2025-09-07T07:10:24.7547837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7548044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7548111Z return mod(**inputs) 2025-09-07T07:10:24.7548401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7548494Z outputs = self.bert( 2025-09-07T07:10:24.7548791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7548871Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7549174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7549242Z layer_outputs = layer_module( 2025-09-07T07:10:24.7549472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7549549Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7549861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7549944Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7550205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7550281Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7550590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7550698Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7550994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7551114Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7551321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7551389Z return self.act(input) 2025-09-07T07:10:24.7551392Z 2025-09-07T07:10:24.7551500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7551699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7551770Z return mod(**inputs) 2025-09-07T07:10:24.7552059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7552130Z outputs = self.bert( 2025-09-07T07:10:24.7552418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7552489Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7552779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7552869Z layer_outputs = layer_module( 2025-09-07T07:10:24.7553093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7553171Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7553477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7553572Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7553848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7553935Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7554275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7554425Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7554735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7554822Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7554844Z 2025-09-07T07:10:24.7554962Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7555178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7555257Z return mod(**inputs) 2025-09-07T07:10:24.7555555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7555620Z outputs = self.bert( 2025-09-07T07:10:24.7555916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7555990Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7556336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7556413Z layer_outputs = layer_module( 2025-09-07T07:10:24.7556669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7556749Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7557043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7557136Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7557447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7557527Z self_outputs = self.self( 2025-09-07T07:10:24.7557782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7557855Z return func(*args, **kwargs) 2025-09-07T07:10:24.7558161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:10:24.7558243Z query_layer = self.query(hidden_states) 2025-09-07T07:10:24.7558246Z 2025-09-07T07:10:24.7558352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7558545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7558616Z return mod(**inputs) 2025-09-07T07:10:24.7558898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7558963Z outputs = self.bert( 2025-09-07T07:10:24.7559248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7559342Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7559627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7559696Z layer_outputs = layer_module( 2025-09-07T07:10:24.7559909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7559990Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7560267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7560352Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7560631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7560701Z self_outputs = self.self( 2025-09-07T07:10:24.7560947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7561015Z return func(*args, **kwargs) 2025-09-07T07:10:24.7561318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:10:24.7561393Z key_layer = self.key(current_states) 2025-09-07T07:10:24.7561397Z 2025-09-07T07:10:24.7561502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7561696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7561759Z return mod(**inputs) 2025-09-07T07:10:24.7562049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7562115Z outputs = self.bert( 2025-09-07T07:10:24.7562426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7562500Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7562782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7562864Z layer_outputs = layer_module( 2025-09-07T07:10:24.7563083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7563166Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7563459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7563560Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7563839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:10:24.7563908Z self_outputs = self.self( 2025-09-07T07:10:24.7564150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:10:24.7564219Z return func(*args, **kwargs) 2025-09-07T07:10:24.7564504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:10:24.7564579Z value_layer = self.value(current_states) 2025-09-07T07:10:24.7564583Z 2025-09-07T07:10:24.7564661Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7564746Z cudagraph partition due to non gpu ops 2025-09-07T07:10:24.7564846Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7565047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7565128Z return mod(**inputs) 2025-09-07T07:10:24.7565420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7565493Z outputs = self.bert( 2025-09-07T07:10:24.7565774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7565856Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7566140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7566216Z layer_outputs = layer_module( 2025-09-07T07:10:24.7566440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7566517Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7566811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:10:24.7566893Z self_attention_outputs = self.attention( 2025-09-07T07:10:24.7567183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:10:24.7567328Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:10:24.7567611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:10:24.7567700Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7567703Z 2025-09-07T07:10:24.7567804Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7568005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7568070Z return mod(**inputs) 2025-09-07T07:10:24.7568364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7568444Z outputs = self.bert( 2025-09-07T07:10:24.7568731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7568812Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7569113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7569188Z layer_outputs = layer_module( 2025-09-07T07:10:24.7569406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7569481Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7569793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7569879Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7570148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7570224Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7570547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7570649Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7570934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:10:24.7571022Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7571026Z 2025-09-07T07:10:24.7571128Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7571332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7571416Z return mod(**inputs) 2025-09-07T07:10:24.7571710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7571785Z outputs = self.bert( 2025-09-07T07:10:24.7572076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7572157Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7572447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7572525Z layer_outputs = layer_module( 2025-09-07T07:10:24.7572747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7572826Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7573127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7573210Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7573476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7573572Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7573891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:10:24.7574001Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:10:24.7574285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:10:24.7574407Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:10:24.7574620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:10:24.7574715Z return self.act(input) 2025-09-07T07:10:24.7574718Z 2025-09-07T07:10:24.7574820Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7575020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7575095Z return mod(**inputs) 2025-09-07T07:10:24.7575385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7575458Z outputs = self.bert( 2025-09-07T07:10:24.7575744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7575815Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7576129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7576201Z layer_outputs = layer_module( 2025-09-07T07:10:24.7576429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7576508Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7576805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7576887Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7577144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7577225Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7577537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7577690Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7577977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:10:24.7578058Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7578069Z 2025-09-07T07:10:24.7578171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7578368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7578441Z return mod(**inputs) 2025-09-07T07:10:24.7578730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-09-07T07:10:24.7578806Z outputs = self.bert( 2025-09-07T07:10:24.7579092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:10:24.7579165Z encoder_outputs = self.encoder( 2025-09-07T07:10:24.7579462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:10:24.7579534Z layer_outputs = layer_module( 2025-09-07T07:10:24.7579761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:10:24.7579890Z return super().__call__(*args, **kwargs) 2025-09-07T07:10:24.7580172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:10:24.7580260Z layer_output = apply_chunking_to_forward( 2025-09-07T07:10:24.7580519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:10:24.7580603Z return forward_fn(*input_tensors) 2025-09-07T07:10:24.7580915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:10:24.7581075Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:10:24.7581362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:10:24.7581440Z return input_tensor + hidden_states 2025-09-07T07:10:24.7581443Z 2025-09-07T07:10:24.7581554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7581754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7581827Z return mod(**inputs) 2025-09-07T07:10:24.7582116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-09-07T07:10:24.7582236Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:10:24.7582532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-09-07T07:10:24.7582644Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:10:24.7582939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 640, in forward 2025-09-07T07:10:24.7583035Z hidden_states = self.transform(hidden_states) 2025-09-07T07:10:24.7583333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 615, in forward 2025-09-07T07:10:24.7583415Z hidden_states = self.dense(hidden_states) 2025-09-07T07:10:24.7583419Z 2025-09-07T07:10:24.7583523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7583732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7583799Z return mod(**inputs) 2025-09-07T07:10:24.7584126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-09-07T07:10:24.7584225Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:10:24.7584541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-09-07T07:10:24.7584661Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:10:24.7584973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 641, in forward 2025-09-07T07:10:24.7585077Z hidden_states = self.decoder(hidden_states) 2025-09-07T07:10:24.7585081Z 2025-09-07T07:10:24.7585190Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:10:24.7585409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:10:24.7585480Z return mod(**inputs) 2025-09-07T07:10:24.7585873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1086, in forward 2025-09-07T07:10:24.7585965Z lm_loss = self.loss_function( 2025-09-07T07:10:24.7586236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-09-07T07:10:24.7586465Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-09-07T07:10:24.7586755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-09-07T07:10:24.7586976Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-09-07T07:10:24.7586981Z 2025-09-07T07:10:38.7413882Z Compilation time (from dynamo_timed): 26.800904741 2025-09-07T07:10:38.7477212Z pass 2025-09-07T07:10:38.7477623Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:10:38.7478738Z TIMING: _recursive_pre_grad_passes:0.01134 _recursive_joint_graph_passes:0.79516 _recursive_post_grad_passes:0.1259 async_compile.wait:0.8467 code_gen:12.60975 inductor_compile:14.94282 backend_compile:21.32012 gc:0.0009 entire_frame_compile:26.8009 total_wall_time:26.8009 2025-09-07T07:10:38.7479728Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:28467 | FakeTensor.__torch_dispatch__:8250 | ProxyTorchDispatchMode.__torch_dispatch__:10946 2025-09-07T07:10:38.7480270Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-09-07T07:10:41.7127281Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:10:41.7129571Z import pynvml # type: ignore[import] 2025-09-07T07:10:44.5119206Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:10:44.5120381Z from pkg_resources import resource_filename 2025-09-07T07:10:45.1783058Z 2025-09-07T07:10:48.1011371Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:10:48.1014120Z loading model: 0it [00:02, ?it/s] 2025-09-07T07:10:48.1040696Z cpu eval MegatronBertForQuestionAnswering 2025-09-07T07:10:49.6305410Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:10:50.2142312Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:10:50.8812671Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:11:05.3333084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3336590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3341166Z return mod(**inputs) 2025-09-07T07:11:05.3341749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3342244Z outputs = self.bert( 2025-09-07T07:11:05.3342696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3345477Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3346203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3346731Z layer_outputs = layer_module( 2025-09-07T07:11:05.3347152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3347573Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3348051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3348859Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3349334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3349811Z self_outputs = self.self( 2025-09-07T07:11:05.3350230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3350649Z return func(*args, **kwargs) 2025-09-07T07:11:05.3351136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3351616Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3351770Z 2025-09-07T07:11:05.3351966Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3352382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3352739Z return mod(**inputs) 2025-09-07T07:11:05.3353191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3353660Z outputs = self.bert( 2025-09-07T07:11:05.3354102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3354572Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3355109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3355576Z layer_outputs = layer_module( 2025-09-07T07:11:05.3355967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3356377Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3356852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3357336Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3357828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3358324Z self_outputs = self.self( 2025-09-07T07:11:05.3358736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3359182Z return func(*args, **kwargs) 2025-09-07T07:11:05.3359641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3360170Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3360322Z 2025-09-07T07:11:05.3360452Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3360861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3361225Z return mod(**inputs) 2025-09-07T07:11:05.3361665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3362113Z outputs = self.bert( 2025-09-07T07:11:05.3362540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3362993Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3363477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3363936Z layer_outputs = layer_module( 2025-09-07T07:11:05.3364319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3364709Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3365185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3365653Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3366118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3366572Z self_outputs = self.self( 2025-09-07T07:11:05.3366970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3367371Z return func(*args, **kwargs) 2025-09-07T07:11:05.3367837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3368304Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3368449Z 2025-09-07T07:11:05.3368822Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3369056Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3369306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3369700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3370053Z return mod(**inputs) 2025-09-07T07:11:05.3370495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3370941Z outputs = self.bert( 2025-09-07T07:11:05.3371393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3371858Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3372319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3372772Z layer_outputs = layer_module( 2025-09-07T07:11:05.3373145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3373537Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3374007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3374487Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3374966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3375521Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3376059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3376555Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3376708Z 2025-09-07T07:11:05.3376834Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3377228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3377578Z return mod(**inputs) 2025-09-07T07:11:05.3378021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3378472Z outputs = self.bert( 2025-09-07T07:11:05.3378911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3379366Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3379832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3380293Z layer_outputs = layer_module( 2025-09-07T07:11:05.3380669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3381090Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3381552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3382043Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3382509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3382958Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3383469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3384022Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3384528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3385013Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3385165Z 2025-09-07T07:11:05.3385287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3385777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3386153Z return mod(**inputs) 2025-09-07T07:11:05.3386611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3387116Z outputs = self.bert( 2025-09-07T07:11:05.3387561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3388030Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3388500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3388972Z layer_outputs = layer_module( 2025-09-07T07:11:05.3389359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3389764Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3390232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3390714Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3391169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3391634Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3392142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3392679Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3393184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3393696Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3394125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3394513Z return self.act(input) 2025-09-07T07:11:05.3394640Z 2025-09-07T07:11:05.3394757Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3395162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3395526Z return mod(**inputs) 2025-09-07T07:11:05.3395974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3396435Z outputs = self.bert( 2025-09-07T07:11:05.3396895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3397373Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3397841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3398313Z layer_outputs = layer_module( 2025-09-07T07:11:05.3398700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3399107Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3399632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3400114Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3400565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3400991Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3401480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3402030Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3402549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3403030Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3403186Z 2025-09-07T07:11:05.3403301Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3403696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3404052Z return mod(**inputs) 2025-09-07T07:11:05.3404509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3404974Z outputs = self.bert( 2025-09-07T07:11:05.3405414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3405882Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3406351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3406808Z layer_outputs = layer_module( 2025-09-07T07:11:05.3407183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3407598Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3408070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3408554Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3409031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3409477Z self_outputs = self.self( 2025-09-07T07:11:05.3409876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3410297Z return func(*args, **kwargs) 2025-09-07T07:11:05.3410771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3411242Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3411400Z 2025-09-07T07:11:05.3411517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3411917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3412268Z return mod(**inputs) 2025-09-07T07:11:05.3412717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3413161Z outputs = self.bert( 2025-09-07T07:11:05.3413599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3414057Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3414523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3414992Z layer_outputs = layer_module( 2025-09-07T07:11:05.3415388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3415790Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3416276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3416757Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3417236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3417705Z self_outputs = self.self( 2025-09-07T07:11:05.3418107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3418530Z return func(*args, **kwargs) 2025-09-07T07:11:05.3419004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3419473Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3419945Z 2025-09-07T07:11:05.3420072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3420524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3420900Z return mod(**inputs) 2025-09-07T07:11:05.3421346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3421807Z outputs = self.bert( 2025-09-07T07:11:05.3422252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3422725Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3423193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3423719Z layer_outputs = layer_module( 2025-09-07T07:11:05.3424099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3424500Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3424976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3425458Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3425984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3426457Z self_outputs = self.self( 2025-09-07T07:11:05.3426860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3427281Z return func(*args, **kwargs) 2025-09-07T07:11:05.3427746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3428215Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3428373Z 2025-09-07T07:11:05.3428465Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3429606Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3429871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3430268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3430631Z return mod(**inputs) 2025-09-07T07:11:05.3431083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3431556Z outputs = self.bert( 2025-09-07T07:11:05.3432005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3432474Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3432974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3433441Z layer_outputs = layer_module( 2025-09-07T07:11:05.3433839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3434205Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3434658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3435130Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3435623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3436158Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3436667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3437105Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3437253Z 2025-09-07T07:11:05.3437360Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3437728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3438055Z return mod(**inputs) 2025-09-07T07:11:05.3438455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3438881Z outputs = self.bert( 2025-09-07T07:11:05.3439288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3439715Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3440161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3440584Z layer_outputs = layer_module( 2025-09-07T07:11:05.3440939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3441307Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3441737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3442183Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3442587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3442990Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3443455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3443956Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3444422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3444871Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3445017Z 2025-09-07T07:11:05.3445124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3445495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3445826Z return mod(**inputs) 2025-09-07T07:11:05.3446228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3446656Z outputs = self.bert( 2025-09-07T07:11:05.3447065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3447518Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3447948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3448372Z layer_outputs = layer_module( 2025-09-07T07:11:05.3448730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3449105Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3449540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3449986Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3450413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3450823Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3451284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3451776Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3452238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3452701Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3453092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3453439Z return self.act(input) 2025-09-07T07:11:05.3453552Z 2025-09-07T07:11:05.3453664Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3454028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3454378Z return mod(**inputs) 2025-09-07T07:11:05.3454791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3455220Z outputs = self.bert( 2025-09-07T07:11:05.3455658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3456110Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3456567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3457021Z layer_outputs = layer_module( 2025-09-07T07:11:05.3457399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3457794Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3458253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3458724Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3459159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3459641Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3460193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3460759Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3461303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3461795Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3461951Z 2025-09-07T07:11:05.3462073Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3462476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3462878Z return mod(**inputs) 2025-09-07T07:11:05.3463310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3463773Z outputs = self.bert( 2025-09-07T07:11:05.3464214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3464675Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3465145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3465615Z layer_outputs = layer_module( 2025-09-07T07:11:05.3466112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3466521Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3466997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3467466Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3468007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3468442Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3468923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3469474Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3469999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.3470487Z return input_tensor + hidden_states 2025-09-07T07:11:05.3470632Z 2025-09-07T07:11:05.3470757Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3471143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3471495Z return mod(**inputs) 2025-09-07T07:11:05.3471930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3472357Z outputs = self.bert( 2025-09-07T07:11:05.3472762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3473186Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3473611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3474041Z layer_outputs = layer_module( 2025-09-07T07:11:05.3474397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3474787Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3475239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3475730Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3476191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3476651Z self_outputs = self.self( 2025-09-07T07:11:05.3477040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3477449Z return func(*args, **kwargs) 2025-09-07T07:11:05.3477892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3478337Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3478499Z 2025-09-07T07:11:05.3478615Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3478978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3479315Z return mod(**inputs) 2025-09-07T07:11:05.3479719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3480147Z outputs = self.bert( 2025-09-07T07:11:05.3480555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3480986Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3481440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3481898Z layer_outputs = layer_module( 2025-09-07T07:11:05.3482272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3482668Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3483105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3483546Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3483985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3484409Z self_outputs = self.self( 2025-09-07T07:11:05.3484783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3485169Z return func(*args, **kwargs) 2025-09-07T07:11:05.3485651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3486128Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3486283Z 2025-09-07T07:11:05.3486392Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3486769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3487107Z return mod(**inputs) 2025-09-07T07:11:05.3487525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3487969Z outputs = self.bert( 2025-09-07T07:11:05.3488410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3488871Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3489340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3489800Z layer_outputs = layer_module( 2025-09-07T07:11:05.3490169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3490630Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3491137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3491600Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3492072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3492527Z self_outputs = self.self( 2025-09-07T07:11:05.3492923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3493340Z return func(*args, **kwargs) 2025-09-07T07:11:05.3493806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3494282Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3494428Z 2025-09-07T07:11:05.3494517Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3494748Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3495005Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3495394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3495737Z return mod(**inputs) 2025-09-07T07:11:05.3496172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3496639Z outputs = self.bert( 2025-09-07T07:11:05.3497077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3497547Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3498005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3498470Z layer_outputs = layer_module( 2025-09-07T07:11:05.3498859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3499264Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3499734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3500227Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3500713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3501257Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3501779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3502245Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3502405Z 2025-09-07T07:11:05.3502518Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3502909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3503276Z return mod(**inputs) 2025-09-07T07:11:05.3503710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3504161Z outputs = self.bert( 2025-09-07T07:11:05.3504594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3505057Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3505513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3506041Z layer_outputs = layer_module( 2025-09-07T07:11:05.3506441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3506833Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3507293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3507741Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3508152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3508559Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3509047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3509541Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3510004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3510441Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3510590Z 2025-09-07T07:11:05.3510698Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3511067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3511395Z return mod(**inputs) 2025-09-07T07:11:05.3511823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3512249Z outputs = self.bert( 2025-09-07T07:11:05.3512662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3513097Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3513525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3513955Z layer_outputs = layer_module( 2025-09-07T07:11:05.3514304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3514674Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3515114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3515586Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3516014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3516454Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3516916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3517415Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3517871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3518330Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3518726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3519078Z return self.act(input) 2025-09-07T07:11:05.3519191Z 2025-09-07T07:11:05.3519307Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3519865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3520198Z return mod(**inputs) 2025-09-07T07:11:05.3520609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3521096Z outputs = self.bert( 2025-09-07T07:11:05.3521519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3521959Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3522384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3522819Z layer_outputs = layer_module( 2025-09-07T07:11:05.3523180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3523555Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3524017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3524473Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3524896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3525313Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3525780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3526304Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3526804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3527286Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3527432Z 2025-09-07T07:11:05.3527547Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3527918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3528259Z return mod(**inputs) 2025-09-07T07:11:05.3528722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3529151Z outputs = self.bert( 2025-09-07T07:11:05.3529555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3529993Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3530414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3530842Z layer_outputs = layer_module( 2025-09-07T07:11:05.3531227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3531601Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3532033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3532508Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3533003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3533424Z self_outputs = self.self( 2025-09-07T07:11:05.3533788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3534161Z return func(*args, **kwargs) 2025-09-07T07:11:05.3534577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3535018Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3535154Z 2025-09-07T07:11:05.3535268Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3535627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3535967Z return mod(**inputs) 2025-09-07T07:11:05.3536370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3536790Z outputs = self.bert( 2025-09-07T07:11:05.3537180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3537596Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3538017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3538437Z layer_outputs = layer_module( 2025-09-07T07:11:05.3538801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3539171Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3539592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3540042Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3540489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3540938Z self_outputs = self.self( 2025-09-07T07:11:05.3541341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3541764Z return func(*args, **kwargs) 2025-09-07T07:11:05.3542207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3542676Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3542825Z 2025-09-07T07:11:05.3542939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3543307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3543683Z return mod(**inputs) 2025-09-07T07:11:05.3544118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3544568Z outputs = self.bert( 2025-09-07T07:11:05.3544997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3545458Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3545983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3546475Z layer_outputs = layer_module( 2025-09-07T07:11:05.3546855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3547254Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3547712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3548180Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3548643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3549100Z self_outputs = self.self( 2025-09-07T07:11:05.3549488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3549893Z return func(*args, **kwargs) 2025-09-07T07:11:05.3550343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3550813Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3550958Z 2025-09-07T07:11:05.3551057Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3551315Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3551571Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3551962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3552313Z return mod(**inputs) 2025-09-07T07:11:05.3552748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3553200Z outputs = self.bert( 2025-09-07T07:11:05.3553596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3554023Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3554456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3554871Z layer_outputs = layer_module( 2025-09-07T07:11:05.3555221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3555584Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3556005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3556432Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3556868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3557351Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3557828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3558272Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3558414Z 2025-09-07T07:11:05.3558525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3558883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3559208Z return mod(**inputs) 2025-09-07T07:11:05.3559609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3560027Z outputs = self.bert( 2025-09-07T07:11:05.3560415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3560853Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3561274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3561695Z layer_outputs = layer_module( 2025-09-07T07:11:05.3562041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3562392Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3562818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3563247Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3563649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3564045Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3564484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3564969Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3565414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3565867Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3566002Z 2025-09-07T07:11:05.3566112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3566464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3566789Z return mod(**inputs) 2025-09-07T07:11:05.3567185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3567604Z outputs = self.bert( 2025-09-07T07:11:05.3567991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3568440Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3568855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3569271Z layer_outputs = layer_module( 2025-09-07T07:11:05.3569614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3569971Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3570390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3570823Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3571248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3571644Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3572085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3572565Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3573012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3573470Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3573852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3574174Z return self.act(input) 2025-09-07T07:11:05.3574288Z 2025-09-07T07:11:05.3574388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3574743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3575086Z return mod(**inputs) 2025-09-07T07:11:05.3575474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3575891Z outputs = self.bert( 2025-09-07T07:11:05.3576290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3576718Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3577121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3577523Z layer_outputs = layer_module( 2025-09-07T07:11:05.3577861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3578217Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3578641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3579074Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3579469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3579883Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3580336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3580845Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3581319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3581754Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3581902Z 2025-09-07T07:11:05.3582011Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3582451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3582784Z return mod(**inputs) 2025-09-07T07:11:05.3583185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3583614Z outputs = self.bert( 2025-09-07T07:11:05.3584019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3584455Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3584905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3585350Z layer_outputs = layer_module( 2025-09-07T07:11:05.3585838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3586248Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3586714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3587190Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3587623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3588051Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3588512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3589018Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3589520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.3590013Z return input_tensor + hidden_states 2025-09-07T07:11:05.3590167Z 2025-09-07T07:11:05.3590282Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3590675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3591041Z return mod(**inputs) 2025-09-07T07:11:05.3591474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3591934Z outputs = self.bert( 2025-09-07T07:11:05.3592374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3592843Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3593306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3593766Z layer_outputs = layer_module( 2025-09-07T07:11:05.3594156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3594558Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3595022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3595528Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3595993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3596456Z self_outputs = self.self( 2025-09-07T07:11:05.3596843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3597224Z return func(*args, **kwargs) 2025-09-07T07:11:05.3597634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3598079Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3598226Z 2025-09-07T07:11:05.3598329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3598687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3599012Z return mod(**inputs) 2025-09-07T07:11:05.3599410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3599833Z outputs = self.bert( 2025-09-07T07:11:05.3600215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3600640Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3601073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3601489Z layer_outputs = layer_module( 2025-09-07T07:11:05.3601839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3602201Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3602627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3603057Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3603484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3603888Z self_outputs = self.self( 2025-09-07T07:11:05.3604250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3604633Z return func(*args, **kwargs) 2025-09-07T07:11:05.3605035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3605458Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3605595Z 2025-09-07T07:11:05.3605699Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3606061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3606381Z return mod(**inputs) 2025-09-07T07:11:05.3606770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3607182Z outputs = self.bert( 2025-09-07T07:11:05.3607585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3608003Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3608420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3608833Z layer_outputs = layer_module( 2025-09-07T07:11:05.3609176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3609560Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3609992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3610405Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3610822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3611235Z self_outputs = self.self( 2025-09-07T07:11:05.3611601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3611977Z return func(*args, **kwargs) 2025-09-07T07:11:05.3612399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3612836Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3612978Z 2025-09-07T07:11:05.3613060Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3613281Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3613520Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3613875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3614198Z return mod(**inputs) 2025-09-07T07:11:05.3614616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3615035Z outputs = self.bert( 2025-09-07T07:11:05.3615458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3615921Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3616401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3616883Z layer_outputs = layer_module( 2025-09-07T07:11:05.3617278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3617682Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3618152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3618596Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3619046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3619691Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3620211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3620696Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3620854Z 2025-09-07T07:11:05.3620965Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3621358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3621726Z return mod(**inputs) 2025-09-07T07:11:05.3622162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3622618Z outputs = self.bert( 2025-09-07T07:11:05.3623060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3623521Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3623977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3624477Z layer_outputs = layer_module( 2025-09-07T07:11:05.3624855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3625256Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3625773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3626271Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3626732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3627179Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3627704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3628193Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3628642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3629084Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3629235Z 2025-09-07T07:11:05.3629343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3629712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3630041Z return mod(**inputs) 2025-09-07T07:11:05.3630558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3630994Z outputs = self.bert( 2025-09-07T07:11:05.3631411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3631854Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3632292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3632760Z layer_outputs = layer_module( 2025-09-07T07:11:05.3633136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3633502Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3633930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3634360Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3634768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3635197Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3635686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3636210Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3636687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3637193Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3637605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3637975Z return self.act(input) 2025-09-07T07:11:05.3638095Z 2025-09-07T07:11:05.3638215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3638605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3638959Z return mod(**inputs) 2025-09-07T07:11:05.3639390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3639860Z outputs = self.bert( 2025-09-07T07:11:05.3640274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3640728Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3641179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3641628Z layer_outputs = layer_module( 2025-09-07T07:11:05.3642000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3642383Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3642861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3643319Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3643719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3644113Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3644554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3645065Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3645573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3646021Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3646162Z 2025-09-07T07:11:05.3646274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3646637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3646968Z return mod(**inputs) 2025-09-07T07:11:05.3647379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3647807Z outputs = self.bert( 2025-09-07T07:11:05.3648211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3648629Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3649043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3649457Z layer_outputs = layer_module( 2025-09-07T07:11:05.3649823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3650178Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3650601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3651034Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3651469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3651902Z self_outputs = self.self( 2025-09-07T07:11:05.3652271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3652653Z return func(*args, **kwargs) 2025-09-07T07:11:05.3653078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3653511Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3653648Z 2025-09-07T07:11:05.3653752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3654118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3654469Z return mod(**inputs) 2025-09-07T07:11:05.3654879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3655310Z outputs = self.bert( 2025-09-07T07:11:05.3655713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3656147Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3656584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3657014Z layer_outputs = layer_module( 2025-09-07T07:11:05.3657390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3657756Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3658195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3658637Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3659074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3659494Z self_outputs = self.self( 2025-09-07T07:11:05.3659867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3660265Z return func(*args, **kwargs) 2025-09-07T07:11:05.3660691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3661131Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3661272Z 2025-09-07T07:11:05.3661384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3661778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3662145Z return mod(**inputs) 2025-09-07T07:11:05.3662581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3663033Z outputs = self.bert( 2025-09-07T07:11:05.3663464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3663928Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3664383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3664864Z layer_outputs = layer_module( 2025-09-07T07:11:05.3665234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3665629Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3666167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3666651Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3667126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3667544Z self_outputs = self.self( 2025-09-07T07:11:05.3667923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3668310Z return func(*args, **kwargs) 2025-09-07T07:11:05.3668733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3669171Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3669310Z 2025-09-07T07:11:05.3669418Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3669643Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3669896Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3670290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3670633Z return mod(**inputs) 2025-09-07T07:11:05.3671047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3671475Z outputs = self.bert( 2025-09-07T07:11:05.3671885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3672342Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3672765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3673201Z layer_outputs = layer_module( 2025-09-07T07:11:05.3673559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3673924Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3674367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3674834Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3675324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3675847Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3676365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3676815Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3676971Z 2025-09-07T07:11:05.3677083Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3677464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3677809Z return mod(**inputs) 2025-09-07T07:11:05.3678231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3678664Z outputs = self.bert( 2025-09-07T07:11:05.3679086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3679551Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3679997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3680443Z layer_outputs = layer_module( 2025-09-07T07:11:05.3680792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3681163Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3681593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3682033Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3682449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3682854Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3683318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3683813Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3684273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3684739Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3684887Z 2025-09-07T07:11:05.3684993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3685361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3685692Z return mod(**inputs) 2025-09-07T07:11:05.3686099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3686516Z outputs = self.bert( 2025-09-07T07:11:05.3686939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3687393Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3687844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3688296Z layer_outputs = layer_module( 2025-09-07T07:11:05.3688664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3689051Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3689486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3689920Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3690340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3690750Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3691212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3691726Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3692213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3692704Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3693115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3693478Z return self.act(input) 2025-09-07T07:11:05.3693592Z 2025-09-07T07:11:05.3693707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3694092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3694461Z return mod(**inputs) 2025-09-07T07:11:05.3694901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3695360Z outputs = self.bert( 2025-09-07T07:11:05.3695799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3696260Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3696706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3697154Z layer_outputs = layer_module( 2025-09-07T07:11:05.3697541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3697943Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3698407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3698883Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3699326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3699805Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3700293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3700837Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3701370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3701845Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3701995Z 2025-09-07T07:11:05.3702115Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3702527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3702871Z return mod(**inputs) 2025-09-07T07:11:05.3703308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3703764Z outputs = self.bert( 2025-09-07T07:11:05.3704196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3704656Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3705099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3705573Z layer_outputs = layer_module( 2025-09-07T07:11:05.3706023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3706422Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3706876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3707352Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3707792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3708220Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3708709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3709250Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3709777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.3710269Z return input_tensor + hidden_states 2025-09-07T07:11:05.3710420Z 2025-09-07T07:11:05.3710536Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3710905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3711245Z return mod(**inputs) 2025-09-07T07:11:05.3711683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3712137Z outputs = self.bert( 2025-09-07T07:11:05.3712567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3713025Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3713451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3713882Z layer_outputs = layer_module( 2025-09-07T07:11:05.3714246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3714634Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3715106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3715550Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3715992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3716447Z self_outputs = self.self( 2025-09-07T07:11:05.3716845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3717227Z return func(*args, **kwargs) 2025-09-07T07:11:05.3717652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3718110Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3718253Z 2025-09-07T07:11:05.3718366Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3718734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3719056Z return mod(**inputs) 2025-09-07T07:11:05.3719466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3720018Z outputs = self.bert( 2025-09-07T07:11:05.3720425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3720906Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3721331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3721770Z layer_outputs = layer_module( 2025-09-07T07:11:05.3722122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3722484Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3722901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3723329Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3723758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3724177Z self_outputs = self.self( 2025-09-07T07:11:05.3724550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3724950Z return func(*args, **kwargs) 2025-09-07T07:11:05.3725366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3725792Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3725925Z 2025-09-07T07:11:05.3726038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3726400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3726723Z return mod(**inputs) 2025-09-07T07:11:05.3727132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3727564Z outputs = self.bert( 2025-09-07T07:11:05.3727974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3728402Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3728829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3729255Z layer_outputs = layer_module( 2025-09-07T07:11:05.3729609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3730014Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3730461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3730915Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3731359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3731795Z self_outputs = self.self( 2025-09-07T07:11:05.3732175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3732561Z return func(*args, **kwargs) 2025-09-07T07:11:05.3733020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3733468Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3733607Z 2025-09-07T07:11:05.3733699Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3733917Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3734160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3734528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3734864Z return mod(**inputs) 2025-09-07T07:11:05.3735289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3735714Z outputs = self.bert( 2025-09-07T07:11:05.3736121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3736552Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3736978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3737405Z layer_outputs = layer_module( 2025-09-07T07:11:05.3737752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3738129Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3738616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3739083Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3739528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3740050Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3740537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3740986Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3741130Z 2025-09-07T07:11:05.3741245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3741614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3741949Z return mod(**inputs) 2025-09-07T07:11:05.3742362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3742799Z outputs = self.bert( 2025-09-07T07:11:05.3743207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3743635Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3744065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3744515Z layer_outputs = layer_module( 2025-09-07T07:11:05.3744872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3745233Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3745739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3746231Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3746696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3747136Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3747660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3748207Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3748714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3749206Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3749363Z 2025-09-07T07:11:05.3749490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3749890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3750272Z return mod(**inputs) 2025-09-07T07:11:05.3750733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3751211Z outputs = self.bert( 2025-09-07T07:11:05.3751664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3752129Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3752620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3753090Z layer_outputs = layer_module( 2025-09-07T07:11:05.3753481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3753883Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3754340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3754775Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3755209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3755620Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3756075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3756575Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3757018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3757475Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3757854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3758188Z return self.act(input) 2025-09-07T07:11:05.3758308Z 2025-09-07T07:11:05.3758412Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3758776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3759107Z return mod(**inputs) 2025-09-07T07:11:05.3759516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3759943Z outputs = self.bert( 2025-09-07T07:11:05.3760334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3760749Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3761160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3761566Z layer_outputs = layer_module( 2025-09-07T07:11:05.3761916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3762275Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3762712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3763150Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3763542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3763937Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3764386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3764891Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3765390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3765820Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3765966Z 2025-09-07T07:11:05.3766072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3766433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3766758Z return mod(**inputs) 2025-09-07T07:11:05.3767158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3767567Z outputs = self.bert( 2025-09-07T07:11:05.3767975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3768407Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3768846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3769274Z layer_outputs = layer_module( 2025-09-07T07:11:05.3769620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3769979Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3770407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3770846Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3771281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3771706Z self_outputs = self.self( 2025-09-07T07:11:05.3772079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3772468Z return func(*args, **kwargs) 2025-09-07T07:11:05.3772890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3773332Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3773488Z 2025-09-07T07:11:05.3773599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3773983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3774358Z return mod(**inputs) 2025-09-07T07:11:05.3774784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3774861Z outputs = self.bert( 2025-09-07T07:11:05.3775182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3775267Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3775579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3775665Z layer_outputs = layer_module( 2025-09-07T07:11:05.3775929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3776017Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3776332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3776419Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3776747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3776825Z self_outputs = self.self( 2025-09-07T07:11:05.3777101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3777184Z return func(*args, **kwargs) 2025-09-07T07:11:05.3777490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3777580Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3777584Z 2025-09-07T07:11:05.3777694Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3777911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3777981Z return mod(**inputs) 2025-09-07T07:11:05.3778294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3778371Z outputs = self.bert( 2025-09-07T07:11:05.3778686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3778773Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3779109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3779187Z layer_outputs = layer_module( 2025-09-07T07:11:05.3779434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3779519Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3779842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3779930Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3780253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3780328Z self_outputs = self.self( 2025-09-07T07:11:05.3780590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3780676Z return func(*args, **kwargs) 2025-09-07T07:11:05.3780999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3781092Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3781113Z 2025-09-07T07:11:05.3781203Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3781289Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3781408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3781623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3781701Z return mod(**inputs) 2025-09-07T07:11:05.3782024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3782096Z outputs = self.bert( 2025-09-07T07:11:05.3782440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3782536Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3782886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3782964Z layer_outputs = layer_module( 2025-09-07T07:11:05.3783221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3783306Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3783628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3783722Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3784067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3784217Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3784539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3784630Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3784641Z 2025-09-07T07:11:05.3784753Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3784971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3785050Z return mod(**inputs) 2025-09-07T07:11:05.3785367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3785448Z outputs = self.bert( 2025-09-07T07:11:05.3785846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3785955Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3786293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3786372Z layer_outputs = layer_module( 2025-09-07T07:11:05.3786631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3786725Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3787056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3787156Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3787447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3787538Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3787893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3788017Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3788326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3788427Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3788435Z 2025-09-07T07:11:05.3788546Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3788746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3788821Z return mod(**inputs) 2025-09-07T07:11:05.3789118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3789187Z outputs = self.bert( 2025-09-07T07:11:05.3789503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3789578Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3789892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3789968Z layer_outputs = layer_module( 2025-09-07T07:11:05.3790213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3790294Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3790619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3790729Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3791012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3791104Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3791448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3791560Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3791884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3791998Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3792222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3792292Z return self.act(input) 2025-09-07T07:11:05.3792296Z 2025-09-07T07:11:05.3792412Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3792650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3792721Z return mod(**inputs) 2025-09-07T07:11:05.3793043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3793116Z outputs = self.bert( 2025-09-07T07:11:05.3793430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3793506Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3793811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3793893Z layer_outputs = layer_module( 2025-09-07T07:11:05.3794133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3794224Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3794536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3794630Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3794908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3795010Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3795363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3795507Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3795832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3795920Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3795926Z 2025-09-07T07:11:05.3796038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3796283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3796357Z return mod(**inputs) 2025-09-07T07:11:05.3796689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3796763Z outputs = self.bert( 2025-09-07T07:11:05.3797089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3797165Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3797474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3797581Z layer_outputs = layer_module( 2025-09-07T07:11:05.3797823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3797915Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3798222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3798312Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3798596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3798678Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3799026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3799168Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3799497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.3799601Z return input_tensor + hidden_states 2025-09-07T07:11:05.3799605Z 2025-09-07T07:11:05.3799717Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3799940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3800012Z return mod(**inputs) 2025-09-07T07:11:05.3800335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3800405Z outputs = self.bert( 2025-09-07T07:11:05.3800715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3800799Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3801114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3801198Z layer_outputs = layer_module( 2025-09-07T07:11:05.3801439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3801530Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3801870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3801958Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3802273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3802348Z self_outputs = self.self( 2025-09-07T07:11:05.3802627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3802700Z return func(*args, **kwargs) 2025-09-07T07:11:05.3803011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3803105Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3803108Z 2025-09-07T07:11:05.3803212Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3803418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3803486Z return mod(**inputs) 2025-09-07T07:11:05.3803787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3803853Z outputs = self.bert( 2025-09-07T07:11:05.3804143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3804240Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3804538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3804622Z layer_outputs = layer_module( 2025-09-07T07:11:05.3804862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3804948Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3805267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3805354Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3805671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3805746Z self_outputs = self.self( 2025-09-07T07:11:05.3806014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3806119Z return func(*args, **kwargs) 2025-09-07T07:11:05.3806442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3806540Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3806545Z 2025-09-07T07:11:05.3806650Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3806859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3806925Z return mod(**inputs) 2025-09-07T07:11:05.3807225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3807297Z outputs = self.bert( 2025-09-07T07:11:05.3807598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3807679Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3807977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3808049Z layer_outputs = layer_module( 2025-09-07T07:11:05.3808287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3808384Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3808684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3808766Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3809069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3809141Z self_outputs = self.self( 2025-09-07T07:11:05.3809390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3809491Z return func(*args, **kwargs) 2025-09-07T07:11:05.3809783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3809871Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3809875Z 2025-09-07T07:11:05.3809961Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3810041Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3810153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3810351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3810424Z return mod(**inputs) 2025-09-07T07:11:05.3810740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3810810Z outputs = self.bert( 2025-09-07T07:11:05.3811112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3811185Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3811483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3811558Z layer_outputs = layer_module( 2025-09-07T07:11:05.3811790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3811872Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3812176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3812267Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3812572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3812711Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3813001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3813087Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3813098Z 2025-09-07T07:11:05.3813204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3813404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3813480Z return mod(**inputs) 2025-09-07T07:11:05.3813778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3813861Z outputs = self.bert( 2025-09-07T07:11:05.3814172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3814251Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3814569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3814665Z layer_outputs = layer_module( 2025-09-07T07:11:05.3814915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3814997Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3815324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3815425Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3815712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3815803Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3816175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3816305Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3816603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3816687Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3816691Z 2025-09-07T07:11:05.3816803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3817005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3817079Z return mod(**inputs) 2025-09-07T07:11:05.3817388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3817459Z outputs = self.bert( 2025-09-07T07:11:05.3817759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3817833Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3818138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3818215Z layer_outputs = layer_module( 2025-09-07T07:11:05.3818461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3818545Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3818867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3818964Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3819266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3819355Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3819897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3820019Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3820350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3820473Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3820713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3820791Z return self.act(input) 2025-09-07T07:11:05.3820796Z 2025-09-07T07:11:05.3820915Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3821151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3821224Z return mod(**inputs) 2025-09-07T07:11:05.3821552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3821665Z outputs = self.bert( 2025-09-07T07:11:05.3821992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3822071Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3822392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3822474Z layer_outputs = layer_module( 2025-09-07T07:11:05.3822725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3822817Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3823164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3823263Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3823544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3823624Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3823990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3824131Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3824484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3824576Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3824580Z 2025-09-07T07:11:05.3824698Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3824915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3824988Z return mod(**inputs) 2025-09-07T07:11:05.3825311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3825381Z outputs = self.bert( 2025-09-07T07:11:05.3825753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3825839Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3826178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3826291Z layer_outputs = layer_module( 2025-09-07T07:11:05.3826539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3826632Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3827019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3827124Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3827459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3827536Z self_outputs = self.self( 2025-09-07T07:11:05.3827805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3827883Z return func(*args, **kwargs) 2025-09-07T07:11:05.3828203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3828294Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3828299Z 2025-09-07T07:11:05.3828409Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3828631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3828721Z return mod(**inputs) 2025-09-07T07:11:05.3829049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3829119Z outputs = self.bert( 2025-09-07T07:11:05.3829446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3829531Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3829849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3829950Z layer_outputs = layer_module( 2025-09-07T07:11:05.3830191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3830283Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3830598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3830686Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3831006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3831081Z self_outputs = self.self( 2025-09-07T07:11:05.3831371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3831452Z return func(*args, **kwargs) 2025-09-07T07:11:05.3831763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3831864Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3831868Z 2025-09-07T07:11:05.3831974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3832182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3832247Z return mod(**inputs) 2025-09-07T07:11:05.3832554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3832624Z outputs = self.bert( 2025-09-07T07:11:05.3832944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3833033Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3833363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3833446Z layer_outputs = layer_module( 2025-09-07T07:11:05.3833670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3833751Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3834051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3834135Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3834438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3834511Z self_outputs = self.self( 2025-09-07T07:11:05.3834774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3834859Z return func(*args, **kwargs) 2025-09-07T07:11:05.3835170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3835259Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3835289Z 2025-09-07T07:11:05.3835373Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3835464Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3835572Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3835778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3835855Z return mod(**inputs) 2025-09-07T07:11:05.3836157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3836245Z outputs = self.bert( 2025-09-07T07:11:05.3836550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3836622Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3836919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3836992Z layer_outputs = layer_module( 2025-09-07T07:11:05.3837221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3837299Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3837588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3837676Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3837984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3838123Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3838412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3838504Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3838507Z 2025-09-07T07:11:05.3838608Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3838806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3838879Z return mod(**inputs) 2025-09-07T07:11:05.3839169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3839240Z outputs = self.bert( 2025-09-07T07:11:05.3839538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3839638Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3839931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3840004Z layer_outputs = layer_module( 2025-09-07T07:11:05.3840235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3840315Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3840614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3840697Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3840966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3841054Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3841380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3841492Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3841813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3841900Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3841903Z 2025-09-07T07:11:05.3842004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3842199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3842271Z return mod(**inputs) 2025-09-07T07:11:05.3842562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3842635Z outputs = self.bert( 2025-09-07T07:11:05.3842933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3843006Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3843297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3843368Z layer_outputs = layer_module( 2025-09-07T07:11:05.3843593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3843671Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3843957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3844062Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3844326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3844410Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3844728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3844838Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3845122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3845234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3845457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3845527Z return self.act(input) 2025-09-07T07:11:05.3845530Z 2025-09-07T07:11:05.3845642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3845886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3845959Z return mod(**inputs) 2025-09-07T07:11:05.3846249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3846317Z outputs = self.bert( 2025-09-07T07:11:05.3846617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3846691Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3846994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3847064Z layer_outputs = layer_module( 2025-09-07T07:11:05.3847282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3847371Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3847654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3847744Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3848018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3848093Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3848412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3848542Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3848839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3848920Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3848924Z 2025-09-07T07:11:05.3849048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3849249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3849313Z return mod(**inputs) 2025-09-07T07:11:05.3849610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3849685Z outputs = self.bert( 2025-09-07T07:11:05.3849976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3850047Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3850354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3850426Z layer_outputs = layer_module( 2025-09-07T07:11:05.3850652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3850738Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3851025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3851114Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3851376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3851450Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3851775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3851905Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3852197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.3852292Z return input_tensor + hidden_states 2025-09-07T07:11:05.3852296Z 2025-09-07T07:11:05.3852403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3852607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3852674Z return mod(**inputs) 2025-09-07T07:11:05.3852971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3853037Z outputs = self.bert( 2025-09-07T07:11:05.3853333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3853404Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3853690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3853771Z layer_outputs = layer_module( 2025-09-07T07:11:05.3853991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3854076Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3854381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3854472Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3854788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3854863Z self_outputs = self.self( 2025-09-07T07:11:05.3855133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3855210Z return func(*args, **kwargs) 2025-09-07T07:11:05.3855540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3855629Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3855633Z 2025-09-07T07:11:05.3855744Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3855967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3856036Z return mod(**inputs) 2025-09-07T07:11:05.3856361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3856431Z outputs = self.bert( 2025-09-07T07:11:05.3856749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3856832Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3857137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3857219Z layer_outputs = layer_module( 2025-09-07T07:11:05.3857458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3857549Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3857858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3857943Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3858269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3858344Z self_outputs = self.self( 2025-09-07T07:11:05.3858613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3858713Z return func(*args, **kwargs) 2025-09-07T07:11:05.3859028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3859119Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3859124Z 2025-09-07T07:11:05.3859237Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3859460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3859530Z return mod(**inputs) 2025-09-07T07:11:05.3859861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3859932Z outputs = self.bert( 2025-09-07T07:11:05.3860257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3860345Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3860659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3860742Z layer_outputs = layer_module( 2025-09-07T07:11:05.3861009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3861094Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3861432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3861522Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3861859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3861937Z self_outputs = self.self( 2025-09-07T07:11:05.3862236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3862316Z return func(*args, **kwargs) 2025-09-07T07:11:05.3862651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3862746Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3862750Z 2025-09-07T07:11:05.3862840Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3862937Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3863053Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3863276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3863358Z return mod(**inputs) 2025-09-07T07:11:05.3863711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3863796Z outputs = self.bert( 2025-09-07T07:11:05.3864130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3864210Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3864551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3864630Z layer_outputs = layer_module( 2025-09-07T07:11:05.3864882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3864968Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3865306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3865396Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3865813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3865976Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3866307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3866412Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3866416Z 2025-09-07T07:11:05.3866530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3866751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3866833Z return mod(**inputs) 2025-09-07T07:11:05.3867181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3867263Z outputs = self.bert( 2025-09-07T07:11:05.3867580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3867669Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3867992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3868092Z layer_outputs = layer_module( 2025-09-07T07:11:05.3868340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3868424Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3868747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3868835Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3869125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3869219Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3869588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3869716Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3870048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3870146Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3870149Z 2025-09-07T07:11:05.3870261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3870497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3870578Z return mod(**inputs) 2025-09-07T07:11:05.3870930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3871014Z outputs = self.bert( 2025-09-07T07:11:05.3871350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3871433Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3871817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3871893Z layer_outputs = layer_module( 2025-09-07T07:11:05.3872144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3872228Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3872563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3872667Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3872953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3873045Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3873401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3873525Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3873845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3873971Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3874215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3874295Z return self.act(input) 2025-09-07T07:11:05.3874301Z 2025-09-07T07:11:05.3874424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3874626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3874699Z return mod(**inputs) 2025-09-07T07:11:05.3875002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3875090Z outputs = self.bert( 2025-09-07T07:11:05.3875396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3875469Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3875770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3875843Z layer_outputs = layer_module( 2025-09-07T07:11:05.3876068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3876174Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3876480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3876577Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3876854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3876941Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3877281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3877421Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3877753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3877844Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3877849Z 2025-09-07T07:11:05.3877967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3878179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3878263Z return mod(**inputs) 2025-09-07T07:11:05.3878568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3878634Z outputs = self.bert( 2025-09-07T07:11:05.3878936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3879008Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3879309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3880269Z layer_outputs = layer_module( 2025-09-07T07:11:05.3880494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3880580Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3880869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3880955Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3881246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3881318Z self_outputs = self.self( 2025-09-07T07:11:05.3881574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3881647Z return func(*args, **kwargs) 2025-09-07T07:11:05.3881955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3882039Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3882042Z 2025-09-07T07:11:05.3882153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3882375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3882441Z return mod(**inputs) 2025-09-07T07:11:05.3882757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3882823Z outputs = self.bert( 2025-09-07T07:11:05.3883125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3883200Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3883496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3883595Z layer_outputs = layer_module( 2025-09-07T07:11:05.3883822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3883908Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3884203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3884291Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3884609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3884684Z self_outputs = self.self( 2025-09-07T07:11:05.3884987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3885067Z return func(*args, **kwargs) 2025-09-07T07:11:05.3885383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3885466Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3885471Z 2025-09-07T07:11:05.3885581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3885799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3885869Z return mod(**inputs) 2025-09-07T07:11:05.3886189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3886258Z outputs = self.bert( 2025-09-07T07:11:05.3886567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3886671Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3886988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3887072Z layer_outputs = layer_module( 2025-09-07T07:11:05.3887318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3887412Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3887732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3887818Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3888142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3888219Z self_outputs = self.self( 2025-09-07T07:11:05.3888495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3888574Z return func(*args, **kwargs) 2025-09-07T07:11:05.3888890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3889002Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3889006Z 2025-09-07T07:11:05.3889095Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3889187Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3889297Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3889517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3889588Z return mod(**inputs) 2025-09-07T07:11:05.3889905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3889985Z outputs = self.bert( 2025-09-07T07:11:05.3890306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3890390Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3890705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3890782Z layer_outputs = layer_module( 2025-09-07T07:11:05.3891028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3891109Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3891426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3891527Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3891839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3891985Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3892296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3892392Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3892396Z 2025-09-07T07:11:05.3892506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3892728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3892799Z return mod(**inputs) 2025-09-07T07:11:05.3893113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3893194Z outputs = self.bert( 2025-09-07T07:11:05.3893507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3893606Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3893920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3893998Z layer_outputs = layer_module( 2025-09-07T07:11:05.3894251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3894336Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3894651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3894740Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3895029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3895114Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3895460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3895580Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3895903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3896000Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3896004Z 2025-09-07T07:11:05.3896114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3896338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3896409Z return mod(**inputs) 2025-09-07T07:11:05.3896722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3896802Z outputs = self.bert( 2025-09-07T07:11:05.3897128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3897215Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3897532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3897608Z layer_outputs = layer_module( 2025-09-07T07:11:05.3897857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3897943Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3898271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3898379Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3898664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3898757Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3899101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3899223Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3899535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3899663Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3899894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3899969Z return self.act(input) 2025-09-07T07:11:05.3899974Z 2025-09-07T07:11:05.3900094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3900364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3900444Z return mod(**inputs) 2025-09-07T07:11:05.3900759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3900831Z outputs = self.bert( 2025-09-07T07:11:05.3901153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3901232Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3901553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3901627Z layer_outputs = layer_module( 2025-09-07T07:11:05.3901874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3901958Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3902270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3902365Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3902664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3902753Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3903105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3903249Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3903572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3903663Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3903666Z 2025-09-07T07:11:05.3903802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3904020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3904101Z return mod(**inputs) 2025-09-07T07:11:05.3904433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3904506Z outputs = self.bert( 2025-09-07T07:11:05.3904852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3904933Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3905290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3905372Z layer_outputs = layer_module( 2025-09-07T07:11:05.3905620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3905916Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3906252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3906356Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3906643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3906734Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3907085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3907231Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3907602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.3907689Z return input_tensor + hidden_states 2025-09-07T07:11:05.3907693Z 2025-09-07T07:11:05.3908155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3908376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3908455Z return mod(**inputs) 2025-09-07T07:11:05.3908788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3908859Z outputs = self.bert( 2025-09-07T07:11:05.3909189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3909271Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3909612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3909690Z layer_outputs = layer_module( 2025-09-07T07:11:05.3909936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3910050Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3910379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3910477Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3910806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3910891Z self_outputs = self.self( 2025-09-07T07:11:05.3911166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3911246Z return func(*args, **kwargs) 2025-09-07T07:11:05.3911603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3911694Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3911700Z 2025-09-07T07:11:05.3911820Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3912048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3912121Z return mod(**inputs) 2025-09-07T07:11:05.3912466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3912539Z outputs = self.bert( 2025-09-07T07:11:05.3912887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3912971Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3913297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3913375Z layer_outputs = layer_module( 2025-09-07T07:11:05.3913620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3913717Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3914036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3914132Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3914449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3914528Z self_outputs = self.self( 2025-09-07T07:11:05.3914808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3914907Z return func(*args, **kwargs) 2025-09-07T07:11:05.3915234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3915322Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3915326Z 2025-09-07T07:11:05.3915448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3915673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3915747Z return mod(**inputs) 2025-09-07T07:11:05.3916078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3916150Z outputs = self.bert( 2025-09-07T07:11:05.3916480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3916562Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3916885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3916988Z layer_outputs = layer_module( 2025-09-07T07:11:05.3917234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3917328Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3917651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3917739Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3918074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3918154Z self_outputs = self.self( 2025-09-07T07:11:05.3918455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3918534Z return func(*args, **kwargs) 2025-09-07T07:11:05.3918868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3918957Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3918961Z 2025-09-07T07:11:05.3919048Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3919144Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3919256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3919491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3919741Z return mod(**inputs) 2025-09-07T07:11:05.3920131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3920218Z outputs = self.bert( 2025-09-07T07:11:05.3920534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3920623Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3920938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3921013Z layer_outputs = layer_module( 2025-09-07T07:11:05.3921262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3921346Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3921670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3921784Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3922100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3922239Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3922553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3922654Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3922658Z 2025-09-07T07:11:05.3922768Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3922991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3923063Z return mod(**inputs) 2025-09-07T07:11:05.3923383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3923463Z outputs = self.bert( 2025-09-07T07:11:05.3923778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3923864Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3924174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3924284Z layer_outputs = layer_module( 2025-09-07T07:11:05.3924532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3924613Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3924937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3925029Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3925324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3925437Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3925781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3925903Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3926215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3926313Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3926316Z 2025-09-07T07:11:05.3926426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3926648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3926737Z return mod(**inputs) 2025-09-07T07:11:05.3927054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3927136Z outputs = self.bert( 2025-09-07T07:11:05.3927444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3927533Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3927842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3927916Z layer_outputs = layer_module( 2025-09-07T07:11:05.3928162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3928245Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3928562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3932996Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3933296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3933390Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3933752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3933860Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3934167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3934283Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3934515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3934591Z return self.act(input) 2025-09-07T07:11:05.3934595Z 2025-09-07T07:11:05.3934761Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3934974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3935042Z return mod(**inputs) 2025-09-07T07:11:05.3935341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3935415Z outputs = self.bert( 2025-09-07T07:11:05.3935710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3935790Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3936086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3936158Z layer_outputs = layer_module( 2025-09-07T07:11:05.3936394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3936501Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3936797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3936881Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3937151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3937228Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3937550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3937693Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3938006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3938102Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3938106Z 2025-09-07T07:11:05.3938212Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3938415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3938490Z return mod(**inputs) 2025-09-07T07:11:05.3938798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3938874Z outputs = self.bert( 2025-09-07T07:11:05.3939178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3939260Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3939563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3939730Z layer_outputs = layer_module( 2025-09-07T07:11:05.3939974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3940058Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3940382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3940470Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3940792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3940876Z self_outputs = self.self( 2025-09-07T07:11:05.3941146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3941228Z return func(*args, **kwargs) 2025-09-07T07:11:05.3941543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3941633Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3941644Z 2025-09-07T07:11:05.3941755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3941970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3942049Z return mod(**inputs) 2025-09-07T07:11:05.3942374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3942451Z outputs = self.bert( 2025-09-07T07:11:05.3942777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3942858Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3943196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3943278Z layer_outputs = layer_module( 2025-09-07T07:11:05.3943524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3943608Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3943928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3944024Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3944390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3944472Z self_outputs = self.self( 2025-09-07T07:11:05.3944754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3944842Z return func(*args, **kwargs) 2025-09-07T07:11:05.3945155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3945240Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3945244Z 2025-09-07T07:11:05.3945363Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3945574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3945649Z return mod(**inputs) 2025-09-07T07:11:05.3946049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3946126Z outputs = self.bert( 2025-09-07T07:11:05.3946451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3946530Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3946918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3946990Z layer_outputs = layer_module( 2025-09-07T07:11:05.3947227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3947309Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3947621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3947717Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3948027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3948112Z self_outputs = self.self( 2025-09-07T07:11:05.3948375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3948455Z return func(*args, **kwargs) 2025-09-07T07:11:05.3948773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3948858Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3948861Z 2025-09-07T07:11:05.3948957Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3949044Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3949154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3949375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3949447Z return mod(**inputs) 2025-09-07T07:11:05.3949772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3949844Z outputs = self.bert( 2025-09-07T07:11:05.3950184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3950265Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3950586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3950669Z layer_outputs = layer_module( 2025-09-07T07:11:05.3950908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3950997Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3951308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3951411Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3951731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3951878Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3952198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3952287Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3952291Z 2025-09-07T07:11:05.3952409Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3952624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3952694Z return mod(**inputs) 2025-09-07T07:11:05.3953020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3953092Z outputs = self.bert( 2025-09-07T07:11:05.3953413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3953527Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3953839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3953921Z layer_outputs = layer_module( 2025-09-07T07:11:05.3954160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3954249Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3954562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3954658Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3954942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3955028Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3955381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3955493Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3955815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3955904Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3955908Z 2025-09-07T07:11:05.3956034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3956239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3956306Z return mod(**inputs) 2025-09-07T07:11:05.3956609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3956678Z outputs = self.bert( 2025-09-07T07:11:05.3957016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3957092Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3957383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3957464Z layer_outputs = layer_module( 2025-09-07T07:11:05.3957688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3957775Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3958079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3958164Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3958441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3958518Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3958848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3958951Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3959249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3959362Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3959577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3959657Z return self.act(input) 2025-09-07T07:11:05.3959661Z 2025-09-07T07:11:05.3959764Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3960010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3960076Z return mod(**inputs) 2025-09-07T07:11:05.3960372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3960445Z outputs = self.bert( 2025-09-07T07:11:05.3960737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3960817Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3961121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3961203Z layer_outputs = layer_module( 2025-09-07T07:11:05.3961442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3961529Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3961848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3961936Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3962224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3962315Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3962637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3962778Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3963069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3963163Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3963167Z 2025-09-07T07:11:05.3963290Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3963504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3963571Z return mod(**inputs) 2025-09-07T07:11:05.3963869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3963945Z outputs = self.bert( 2025-09-07T07:11:05.3964235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3964317Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3964636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3964710Z layer_outputs = layer_module( 2025-09-07T07:11:05.3964948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3965028Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3965328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3965411Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3965681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3965758Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3966078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3966219Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3966546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.3966633Z return input_tensor + hidden_states 2025-09-07T07:11:05.3966636Z 2025-09-07T07:11:05.3966740Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3966944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3967020Z return mod(**inputs) 2025-09-07T07:11:05.3967316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3967391Z outputs = self.bert( 2025-09-07T07:11:05.3967683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3967763Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3968066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3968140Z layer_outputs = layer_module( 2025-09-07T07:11:05.3968382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3968466Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3968780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3968867Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3969175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3969257Z self_outputs = self.self( 2025-09-07T07:11:05.3969532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3969614Z return func(*args, **kwargs) 2025-09-07T07:11:05.3969920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3970012Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3970016Z 2025-09-07T07:11:05.3970121Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3970322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3970397Z return mod(**inputs) 2025-09-07T07:11:05.3970692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3970766Z outputs = self.bert( 2025-09-07T07:11:05.3971094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3971179Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3971503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3971576Z layer_outputs = layer_module( 2025-09-07T07:11:05.3971809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3971888Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3972186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3972270Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3972563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3972643Z self_outputs = self.self( 2025-09-07T07:11:05.3972892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3973010Z return func(*args, **kwargs) 2025-09-07T07:11:05.3973304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3973382Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3973386Z 2025-09-07T07:11:05.3973498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3973700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3973775Z return mod(**inputs) 2025-09-07T07:11:05.3974081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3974153Z outputs = self.bert( 2025-09-07T07:11:05.3974469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3974551Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3974874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3974948Z layer_outputs = layer_module( 2025-09-07T07:11:05.3975195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3975277Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3975590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3975685Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3975990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3976069Z self_outputs = self.self( 2025-09-07T07:11:05.3976344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3976422Z return func(*args, **kwargs) 2025-09-07T07:11:05.3976739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.3976823Z value_layer = self.value(current_states) 2025-09-07T07:11:05.3976826Z 2025-09-07T07:11:05.3976921Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3977005Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.3977120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3977336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3977423Z return mod(**inputs) 2025-09-07T07:11:05.3977745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3977821Z outputs = self.bert( 2025-09-07T07:11:05.3978138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3978215Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3978527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3978611Z layer_outputs = layer_module( 2025-09-07T07:11:05.3978848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3978938Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3979259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3979345Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3979690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.3979829Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.3980151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.3980240Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3980244Z 2025-09-07T07:11:05.3980361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3980574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3980645Z return mod(**inputs) 2025-09-07T07:11:05.3980970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3981044Z outputs = self.bert( 2025-09-07T07:11:05.3981363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3981442Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3981758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3981834Z layer_outputs = layer_module( 2025-09-07T07:11:05.3982075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3982167Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3982488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3982588Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3982876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3982980Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3983330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3983441Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3983758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.3983848Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3983852Z 2025-09-07T07:11:05.3983968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3984200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3984272Z return mod(**inputs) 2025-09-07T07:11:05.3984597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3984669Z outputs = self.bert( 2025-09-07T07:11:05.3984988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3985065Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3985375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3985456Z layer_outputs = layer_module( 2025-09-07T07:11:05.3985766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3985871Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3986185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3986318Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3986618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3986703Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3987067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.3987177Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.3987497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.3987619Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.3987851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.3987939Z return self.act(input) 2025-09-07T07:11:05.3987945Z 2025-09-07T07:11:05.3988056Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3988285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3988356Z return mod(**inputs) 2025-09-07T07:11:05.3988681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3988758Z outputs = self.bert( 2025-09-07T07:11:05.3989090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3989175Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3989495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3989577Z layer_outputs = layer_module( 2025-09-07T07:11:05.3989845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3989939Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3990258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.3990344Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.3990631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.3990711Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.3991067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.3991227Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.3991551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.3991649Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.3991653Z 2025-09-07T07:11:05.3991768Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3991980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3992046Z return mod(**inputs) 2025-09-07T07:11:05.3992359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3992427Z outputs = self.bert( 2025-09-07T07:11:05.3992724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3992808Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3993103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3993222Z layer_outputs = layer_module( 2025-09-07T07:11:05.3993450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3993529Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3993832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3993914Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3994213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3994285Z self_outputs = self.self( 2025-09-07T07:11:05.3994542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3994614Z return func(*args, **kwargs) 2025-09-07T07:11:05.3994938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.3995036Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.3995039Z 2025-09-07T07:11:05.3995151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3995372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3995442Z return mod(**inputs) 2025-09-07T07:11:05.3995770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3995846Z outputs = self.bert( 2025-09-07T07:11:05.3996170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3996254Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3996600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.3996694Z layer_outputs = layer_module( 2025-09-07T07:11:05.3996918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.3996994Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.3997294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.3997375Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.3997675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.3997763Z self_outputs = self.self( 2025-09-07T07:11:05.3998013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.3998093Z return func(*args, **kwargs) 2025-09-07T07:11:05.3998387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.3998473Z key_layer = self.key(current_states) 2025-09-07T07:11:05.3998476Z 2025-09-07T07:11:05.3998581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.3998787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.3998854Z return mod(**inputs) 2025-09-07T07:11:05.3999149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.3999221Z outputs = self.bert( 2025-09-07T07:11:05.3999513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.3999633Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.3999938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4000010Z layer_outputs = layer_module( 2025-09-07T07:11:05.4000250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4000327Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4000632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4000713Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4001014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4001093Z self_outputs = self.self( 2025-09-07T07:11:05.4001348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4001430Z return func(*args, **kwargs) 2025-09-07T07:11:05.4001735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4001821Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4001824Z 2025-09-07T07:11:05.4001908Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4001990Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4002103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4002311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4002385Z return mod(**inputs) 2025-09-07T07:11:05.4002693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4002763Z outputs = self.bert( 2025-09-07T07:11:05.4003087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4003162Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4003463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4003536Z layer_outputs = layer_module( 2025-09-07T07:11:05.4003764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4003850Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4004167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4004257Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4004549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4004689Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4004982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4005068Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4005072Z 2025-09-07T07:11:05.4005189Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4005401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4005480Z return mod(**inputs) 2025-09-07T07:11:05.4005792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4005871Z outputs = self.bert( 2025-09-07T07:11:05.4006216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4006304Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4006604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4006675Z layer_outputs = layer_module( 2025-09-07T07:11:05.4006904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4006982Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4007273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4007364Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4007630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4007718Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4008038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4008150Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4008440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4008522Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4008526Z 2025-09-07T07:11:05.4008637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4008840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4008916Z return mod(**inputs) 2025-09-07T07:11:05.4009209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4009294Z outputs = self.bert( 2025-09-07T07:11:05.4009599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4009671Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4009974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4010045Z layer_outputs = layer_module( 2025-09-07T07:11:05.4010275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4010360Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4010672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4010764Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4011030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4011114Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4011437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4011541Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4011842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4011954Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4012177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4012248Z return self.act(input) 2025-09-07T07:11:05.4012251Z 2025-09-07T07:11:05.4012410Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4012613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4012680Z return mod(**inputs) 2025-09-07T07:11:05.4012984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4013051Z outputs = self.bert( 2025-09-07T07:11:05.4013369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4013447Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4013756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4013841Z layer_outputs = layer_module( 2025-09-07T07:11:05.4014080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4014174Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4014485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4014574Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4014861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4014943Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4015294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4015438Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4015759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4015849Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4015869Z 2025-09-07T07:11:05.4015982Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4016205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4016275Z return mod(**inputs) 2025-09-07T07:11:05.4016602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4016673Z outputs = self.bert( 2025-09-07T07:11:05.4016981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4017069Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4017400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4017490Z layer_outputs = layer_module( 2025-09-07T07:11:05.4017732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4017824Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4018134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4018223Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4018509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4018592Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4018942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4019083Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4019437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.4019521Z return input_tensor + hidden_states 2025-09-07T07:11:05.4019525Z 2025-09-07T07:11:05.4019782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4020011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4020082Z return mod(**inputs) 2025-09-07T07:11:05.4020410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4020483Z outputs = self.bert( 2025-09-07T07:11:05.4020811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4020901Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4021234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4021319Z layer_outputs = layer_module( 2025-09-07T07:11:05.4021559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4021643Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4021963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4022053Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4022377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4022456Z self_outputs = self.self( 2025-09-07T07:11:05.4022732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4022811Z return func(*args, **kwargs) 2025-09-07T07:11:05.4023182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4023281Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4023285Z 2025-09-07T07:11:05.4023396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4023615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4023685Z return mod(**inputs) 2025-09-07T07:11:05.4023995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4024072Z outputs = self.bert( 2025-09-07T07:11:05.4024420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4024510Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4024821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4024905Z layer_outputs = layer_module( 2025-09-07T07:11:05.4025146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4025230Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4025555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4025641Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4026029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4026115Z self_outputs = self.self( 2025-09-07T07:11:05.4026453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4026537Z return func(*args, **kwargs) 2025-09-07T07:11:05.4026856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4026949Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4026953Z 2025-09-07T07:11:05.4027067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4027293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4027376Z return mod(**inputs) 2025-09-07T07:11:05.4027689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4027770Z outputs = self.bert( 2025-09-07T07:11:05.4028078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4028168Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4028476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4028553Z layer_outputs = layer_module( 2025-09-07T07:11:05.4028801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4028882Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4029198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4029285Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4029599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4029676Z self_outputs = self.self( 2025-09-07T07:11:05.4029956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4030039Z return func(*args, **kwargs) 2025-09-07T07:11:05.4030364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4030455Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4030458Z 2025-09-07T07:11:05.4030544Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4030630Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4030749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4030961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4031053Z return mod(**inputs) 2025-09-07T07:11:05.4031366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4031441Z outputs = self.bert( 2025-09-07T07:11:05.4031765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4031841Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4032162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4032237Z layer_outputs = layer_module( 2025-09-07T07:11:05.4032480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4032563Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4032882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4033009Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4033337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4033482Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4033806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4033894Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4033905Z 2025-09-07T07:11:05.4034015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4034232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4034311Z return mod(**inputs) 2025-09-07T07:11:05.4034639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4034720Z outputs = self.bert( 2025-09-07T07:11:05.4035043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4035121Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4035440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4035516Z layer_outputs = layer_module( 2025-09-07T07:11:05.4035763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4035846Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4036171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4036270Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4036579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4036666Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4036998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4037109Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4037404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4037488Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4037492Z 2025-09-07T07:11:05.4037604Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4037823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4037899Z return mod(**inputs) 2025-09-07T07:11:05.4038202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4038269Z outputs = self.bert( 2025-09-07T07:11:05.4038570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4038643Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4038941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4039014Z layer_outputs = layer_module( 2025-09-07T07:11:05.4039245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4039323Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4039617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4039752Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4040017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4040101Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4040425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4040528Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4040829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4040944Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4041167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4041243Z return self.act(input) 2025-09-07T07:11:05.4041247Z 2025-09-07T07:11:05.4041359Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4041567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4041639Z return mod(**inputs) 2025-09-07T07:11:05.4041963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4042032Z outputs = self.bert( 2025-09-07T07:11:05.4042353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4042431Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4042751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4042836Z layer_outputs = layer_module( 2025-09-07T07:11:05.4043092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4043183Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4043482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4043573Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4043845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4043923Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4044269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4044419Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4044729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4044817Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4044821Z 2025-09-07T07:11:05.4044925Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4045140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4045209Z return mod(**inputs) 2025-09-07T07:11:05.4045539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4045607Z outputs = self.bert( 2025-09-07T07:11:05.4045915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4045991Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4046293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4046453Z layer_outputs = layer_module( 2025-09-07T07:11:05.4046685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4046772Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4047068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4047152Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4047458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4047530Z self_outputs = self.self( 2025-09-07T07:11:05.4047794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4047872Z return func(*args, **kwargs) 2025-09-07T07:11:05.4048181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4048265Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4048269Z 2025-09-07T07:11:05.4048375Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4048588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4048657Z return mod(**inputs) 2025-09-07T07:11:05.4048968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4049038Z outputs = self.bert( 2025-09-07T07:11:05.4049363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4049449Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4049773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4049858Z layer_outputs = layer_module( 2025-09-07T07:11:05.4050092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4050174Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4050490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4050573Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4050879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4050969Z self_outputs = self.self( 2025-09-07T07:11:05.4051224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4051302Z return func(*args, **kwargs) 2025-09-07T07:11:05.4051599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4051689Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4051693Z 2025-09-07T07:11:05.4051800Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4052008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4052075Z return mod(**inputs) 2025-09-07T07:11:05.4052374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4052451Z outputs = self.bert( 2025-09-07T07:11:05.4052748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4052862Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4053165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4053246Z layer_outputs = layer_module( 2025-09-07T07:11:05.4053477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4053554Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4053864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4053947Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4054263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4054335Z self_outputs = self.self( 2025-09-07T07:11:05.4054595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4054674Z return func(*args, **kwargs) 2025-09-07T07:11:05.4054976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4055063Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4055066Z 2025-09-07T07:11:05.4055157Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4055245Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4055350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4055555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4055628Z return mod(**inputs) 2025-09-07T07:11:05.4055933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4056008Z outputs = self.bert( 2025-09-07T07:11:05.4056329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4056405Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4056705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4056776Z layer_outputs = layer_module( 2025-09-07T07:11:05.4057009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4057087Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4057394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4057485Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4057783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4057926Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4058221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4058313Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4058316Z 2025-09-07T07:11:05.4058420Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4058624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4058699Z return mod(**inputs) 2025-09-07T07:11:05.4059001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4059074Z outputs = self.bert( 2025-09-07T07:11:05.4059420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4059494Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4059796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4059869Z layer_outputs = layer_module( 2025-09-07T07:11:05.4060103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4060181Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4060489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4060575Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4060840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4060929Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4061255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4061368Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4061663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4061746Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4061757Z 2025-09-07T07:11:05.4061859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4062062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4062135Z return mod(**inputs) 2025-09-07T07:11:05.4062435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4062524Z outputs = self.bert( 2025-09-07T07:11:05.4062817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4062890Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4063189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4063260Z layer_outputs = layer_module( 2025-09-07T07:11:05.4063492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4063570Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4063883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4063976Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4064245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4064330Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4064657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4064768Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4065062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4065175Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4065401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4065473Z return self.act(input) 2025-09-07T07:11:05.4065507Z 2025-09-07T07:11:05.4065628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4065925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4066004Z return mod(**inputs) 2025-09-07T07:11:05.4066337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4066411Z outputs = self.bert( 2025-09-07T07:11:05.4066751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4066829Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4067148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4067227Z layer_outputs = layer_module( 2025-09-07T07:11:05.4067473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4067568Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4067866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4067959Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4068224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4068304Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4068644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4068779Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4069076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4069179Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4069183Z 2025-09-07T07:11:05.4069295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4069496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4069561Z return mod(**inputs) 2025-09-07T07:11:05.4069860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4069925Z outputs = self.bert( 2025-09-07T07:11:05.4070233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4070305Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4070613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4070696Z layer_outputs = layer_module( 2025-09-07T07:11:05.4070918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4071002Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4071286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4071374Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4071632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4071706Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4072029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4072159Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4072483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.4072561Z return input_tensor + hidden_states 2025-09-07T07:11:05.4072565Z 2025-09-07T07:11:05.4072674Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4072872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4072937Z return mod(**inputs) 2025-09-07T07:11:05.4073229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4073295Z outputs = self.bert( 2025-09-07T07:11:05.4073587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4073658Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4073951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4074028Z layer_outputs = layer_module( 2025-09-07T07:11:05.4074246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4074330Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4074618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4074702Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4075004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4075076Z self_outputs = self.self( 2025-09-07T07:11:05.4075330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4075422Z return func(*args, **kwargs) 2025-09-07T07:11:05.4075727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4075810Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4075813Z 2025-09-07T07:11:05.4075919Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4076128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4076196Z return mod(**inputs) 2025-09-07T07:11:05.4076506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4076571Z outputs = self.bert( 2025-09-07T07:11:05.4076873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4076955Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4077239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4077315Z layer_outputs = layer_module( 2025-09-07T07:11:05.4077532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4077614Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4077900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4077980Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4078273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4078342Z self_outputs = self.self( 2025-09-07T07:11:05.4078622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4078691Z return func(*args, **kwargs) 2025-09-07T07:11:05.4078979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4079064Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4079068Z 2025-09-07T07:11:05.4079167Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4079370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4079435Z return mod(**inputs) 2025-09-07T07:11:05.4079733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4079796Z outputs = self.bert( 2025-09-07T07:11:05.4080085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4080165Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4080450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4080527Z layer_outputs = layer_module( 2025-09-07T07:11:05.4080752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4080831Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4081131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4081212Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4081517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4081590Z self_outputs = self.self( 2025-09-07T07:11:05.4081854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4081933Z return func(*args, **kwargs) 2025-09-07T07:11:05.4082231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4082321Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4082324Z 2025-09-07T07:11:05.4082409Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4082497Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4082602Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4082824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4082906Z return mod(**inputs) 2025-09-07T07:11:05.4083215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4083296Z outputs = self.bert( 2025-09-07T07:11:05.4083623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4083702Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4084028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4084104Z layer_outputs = layer_module( 2025-09-07T07:11:05.4084354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4084438Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4084758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4084885Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4085199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4085337Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4085633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4085731Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4085734Z 2025-09-07T07:11:05.4085843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4086055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4086132Z return mod(**inputs) 2025-09-07T07:11:05.4086446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4086526Z outputs = self.bert( 2025-09-07T07:11:05.4086836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4086921Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4087230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4087306Z layer_outputs = layer_module( 2025-09-07T07:11:05.4087550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4087632Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4087962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4088051Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4088362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4088454Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4088806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4088925Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4089234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4089328Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4089331Z 2025-09-07T07:11:05.4089441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4089674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4089756Z return mod(**inputs) 2025-09-07T07:11:05.4090075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4090152Z outputs = self.bert( 2025-09-07T07:11:05.4090463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4090541Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4090859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4090935Z layer_outputs = layer_module( 2025-09-07T07:11:05.4091180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4091267Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4091588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4091708Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4091989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4092080Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4092422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4092534Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4092826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4092942Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4093171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4093245Z return self.act(input) 2025-09-07T07:11:05.4093249Z 2025-09-07T07:11:05.4093360Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4093563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4093637Z return mod(**inputs) 2025-09-07T07:11:05.4093935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4094000Z outputs = self.bert( 2025-09-07T07:11:05.4094302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4094374Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4094680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4094754Z layer_outputs = layer_module( 2025-09-07T07:11:05.4094994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4095083Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4095384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4095472Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4095739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4095814Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4096160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4096303Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4096599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4096680Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4096683Z 2025-09-07T07:11:05.4096791Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4096985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4097049Z return mod(**inputs) 2025-09-07T07:11:05.4097347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4097413Z outputs = self.bert( 2025-09-07T07:11:05.4097715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4097787Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4098091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4098196Z layer_outputs = layer_module( 2025-09-07T07:11:05.4098421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4098509Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4098804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4098894Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4099187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4099257Z self_outputs = self.self( 2025-09-07T07:11:05.4099518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4099593Z return func(*args, **kwargs) 2025-09-07T07:11:05.4099897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4099980Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4099983Z 2025-09-07T07:11:05.4100095Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4100297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4100366Z return mod(**inputs) 2025-09-07T07:11:05.4100699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4100770Z outputs = self.bert( 2025-09-07T07:11:05.4101100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4101182Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4101519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4101606Z layer_outputs = layer_module( 2025-09-07T07:11:05.4101844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4101936Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4102293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4102375Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4102693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4102767Z self_outputs = self.self( 2025-09-07T07:11:05.4103024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4103100Z return func(*args, **kwargs) 2025-09-07T07:11:05.4103398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4103477Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4103481Z 2025-09-07T07:11:05.4103585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4103806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4103877Z return mod(**inputs) 2025-09-07T07:11:05.4104198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4104270Z outputs = self.bert( 2025-09-07T07:11:05.4104585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4104705Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4105017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4105100Z layer_outputs = layer_module( 2025-09-07T07:11:05.4105347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4105438Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4105821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4105919Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4106251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4106332Z self_outputs = self.self( 2025-09-07T07:11:05.4106614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4106692Z return func(*args, **kwargs) 2025-09-07T07:11:05.4107008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4107114Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4107118Z 2025-09-07T07:11:05.4107206Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4107300Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4107410Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4107625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4107716Z return mod(**inputs) 2025-09-07T07:11:05.4108012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4108110Z outputs = self.bert( 2025-09-07T07:11:05.4108410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4108491Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4108785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4108856Z layer_outputs = layer_module( 2025-09-07T07:11:05.4109102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4109186Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4109529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4109616Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4109928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4110074Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4110382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4110478Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4110482Z 2025-09-07T07:11:05.4110591Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4110817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4110883Z return mod(**inputs) 2025-09-07T07:11:05.4111176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4111291Z outputs = self.bert( 2025-09-07T07:11:05.4111586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4111667Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4111964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4112040Z layer_outputs = layer_module( 2025-09-07T07:11:05.4112288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4112371Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4112692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4112783Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4113075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4113159Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4113504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4113623Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4113957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4114053Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4114056Z 2025-09-07T07:11:05.4114167Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4114385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4114463Z return mod(**inputs) 2025-09-07T07:11:05.4114791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4114873Z outputs = self.bert( 2025-09-07T07:11:05.4115202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4115287Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4115611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4115688Z layer_outputs = layer_module( 2025-09-07T07:11:05.4115936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4116018Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4116366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4116460Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4116742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4116833Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4117177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4117295Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4117661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4117790Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4118021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4118099Z return self.act(input) 2025-09-07T07:11:05.4118135Z 2025-09-07T07:11:05.4118253Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4118467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4118544Z return mod(**inputs) 2025-09-07T07:11:05.4118866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4118936Z outputs = self.bert( 2025-09-07T07:11:05.4119266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4119344Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4119789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4119874Z layer_outputs = layer_module( 2025-09-07T07:11:05.4120128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4120217Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4120547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4120657Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4120942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4121033Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4121393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4121538Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4121875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4122048Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4122052Z 2025-09-07T07:11:05.4122165Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4122375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4122448Z return mod(**inputs) 2025-09-07T07:11:05.4122748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4122817Z outputs = self.bert( 2025-09-07T07:11:05.4123116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4123211Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4123523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4123600Z layer_outputs = layer_module( 2025-09-07T07:11:05.4123829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4123924Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4124249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4124346Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4124628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4124718Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4125065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4125253Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4125576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.4125662Z return input_tensor + hidden_states 2025-09-07T07:11:05.4125666Z 2025-09-07T07:11:05.4125791Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4125993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4126059Z return mod(**inputs) 2025-09-07T07:11:05.4126369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4126440Z outputs = self.bert( 2025-09-07T07:11:05.4126762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4126841Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4127158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4127234Z layer_outputs = layer_module( 2025-09-07T07:11:05.4127471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4127561Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4127878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4127972Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4128290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4128365Z self_outputs = self.self( 2025-09-07T07:11:05.4128651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4128731Z return func(*args, **kwargs) 2025-09-07T07:11:05.4129045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4129131Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4129134Z 2025-09-07T07:11:05.4129251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4129464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4129533Z return mod(**inputs) 2025-09-07T07:11:05.4129852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4129940Z outputs = self.bert( 2025-09-07T07:11:05.4130262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4130343Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4130658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4130742Z layer_outputs = layer_module( 2025-09-07T07:11:05.4130983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4131074Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4131401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4131494Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4131820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4131925Z self_outputs = self.self( 2025-09-07T07:11:05.4132193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4132270Z return func(*args, **kwargs) 2025-09-07T07:11:05.4132587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4132673Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4132677Z 2025-09-07T07:11:05.4132787Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4133007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4133077Z return mod(**inputs) 2025-09-07T07:11:05.4133402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4133473Z outputs = self.bert( 2025-09-07T07:11:05.4133791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4133878Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4134188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4134271Z layer_outputs = layer_module( 2025-09-07T07:11:05.4134508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4134597Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4134918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4135008Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4135328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4135423Z self_outputs = self.self( 2025-09-07T07:11:05.4135701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4135776Z return func(*args, **kwargs) 2025-09-07T07:11:05.4136089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4136175Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4136178Z 2025-09-07T07:11:05.4136261Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4136349Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4136454Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4136683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4136751Z return mod(**inputs) 2025-09-07T07:11:05.4137051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4137123Z outputs = self.bert( 2025-09-07T07:11:05.4137414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4137493Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4137787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4137858Z layer_outputs = layer_module( 2025-09-07T07:11:05.4138089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4138170Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4138470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4138594Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4138889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4139026Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4139317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4139407Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4139410Z 2025-09-07T07:11:05.4139513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4139717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4139787Z return mod(**inputs) 2025-09-07T07:11:05.4140086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4140166Z outputs = self.bert( 2025-09-07T07:11:05.4140458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4140538Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4140832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4140904Z layer_outputs = layer_module( 2025-09-07T07:11:05.4141135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4141217Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4141531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4141620Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4142266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4142347Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4142677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4142799Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4143109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4143207Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4143211Z 2025-09-07T07:11:05.4143320Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4143558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4143634Z return mod(**inputs) 2025-09-07T07:11:05.4143956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4144039Z outputs = self.bert( 2025-09-07T07:11:05.4144362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4144450Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4144775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4144853Z layer_outputs = layer_module( 2025-09-07T07:11:05.4145107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4145197Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4145528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4145657Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4146015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4146112Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4146464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4146582Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4146905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4147044Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4147264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4147342Z return self.act(input) 2025-09-07T07:11:05.4147345Z 2025-09-07T07:11:05.4147457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4147654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4147726Z return mod(**inputs) 2025-09-07T07:11:05.4148015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4148082Z outputs = self.bert( 2025-09-07T07:11:05.4148376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4148451Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4148753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4148828Z layer_outputs = layer_module( 2025-09-07T07:11:05.4149091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4149173Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4149463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4149555Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4149822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4149907Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4150249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4150382Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4150684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4150768Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4150771Z 2025-09-07T07:11:05.4150883Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4151084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4151158Z return mod(**inputs) 2025-09-07T07:11:05.4151450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4151518Z outputs = self.bert( 2025-09-07T07:11:05.4151816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4151891Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4152230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4152303Z layer_outputs = layer_module( 2025-09-07T07:11:05.4152527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4152613Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4152908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4152998Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4153291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4153372Z self_outputs = self.self( 2025-09-07T07:11:05.4153619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4153695Z return func(*args, **kwargs) 2025-09-07T07:11:05.4153993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4154074Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4154077Z 2025-09-07T07:11:05.4154189Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4154388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4154455Z return mod(**inputs) 2025-09-07T07:11:05.4154758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4154824Z outputs = self.bert( 2025-09-07T07:11:05.4155125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4155202Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4155515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4155587Z layer_outputs = layer_module( 2025-09-07T07:11:05.4155809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4155895Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4156186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4156276Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4156583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4156656Z self_outputs = self.self( 2025-09-07T07:11:05.4156915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4156986Z return func(*args, **kwargs) 2025-09-07T07:11:05.4157282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4157362Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4157365Z 2025-09-07T07:11:05.4157476Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4157676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4157743Z return mod(**inputs) 2025-09-07T07:11:05.4158046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4158115Z outputs = self.bert( 2025-09-07T07:11:05.4158413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4158522Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4158816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4158895Z layer_outputs = layer_module( 2025-09-07T07:11:05.4159119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4159206Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4159497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4159585Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4159880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4159953Z self_outputs = self.self( 2025-09-07T07:11:05.4160209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4160279Z return func(*args, **kwargs) 2025-09-07T07:11:05.4160583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4160663Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4160666Z 2025-09-07T07:11:05.4160748Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4160835Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4160938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4161157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4161228Z return mod(**inputs) 2025-09-07T07:11:05.4161555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4161635Z outputs = self.bert( 2025-09-07T07:11:05.4161958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4162041Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4162349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4162430Z layer_outputs = layer_module( 2025-09-07T07:11:05.4162666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4162749Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4163089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4163175Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4163478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4163610Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4163909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4164002Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4164005Z 2025-09-07T07:11:05.4164109Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4164319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4164388Z return mod(**inputs) 2025-09-07T07:11:05.4164697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4164821Z outputs = self.bert( 2025-09-07T07:11:05.4165123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4165205Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4165503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4165584Z layer_outputs = layer_module( 2025-09-07T07:11:05.4165812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4165891Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4166200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4166284Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4166566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4166645Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4166983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4167090Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4167389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4167481Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4167484Z 2025-09-07T07:11:05.4167590Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4167805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4167872Z return mod(**inputs) 2025-09-07T07:11:05.4168193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4168270Z outputs = self.bert( 2025-09-07T07:11:05.4168572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4168653Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4168946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4169025Z layer_outputs = layer_module( 2025-09-07T07:11:05.4169251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4169345Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4169644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4169734Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4170015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4170090Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4170414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4170527Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4170823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4170946Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4171165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4171275Z return self.act(input) 2025-09-07T07:11:05.4171279Z 2025-09-07T07:11:05.4171384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4171586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4171659Z return mod(**inputs) 2025-09-07T07:11:05.4171957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4172032Z outputs = self.bert( 2025-09-07T07:11:05.4172327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4172401Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4172704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4172779Z layer_outputs = layer_module( 2025-09-07T07:11:05.4173013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4173092Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4173384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4173477Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4173741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4173825Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4174147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4174291Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4174606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4174694Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4174697Z 2025-09-07T07:11:05.4174810Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4175014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4175089Z return mod(**inputs) 2025-09-07T07:11:05.4175389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4175464Z outputs = self.bert( 2025-09-07T07:11:05.4175761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4175848Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4176152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4176228Z layer_outputs = layer_module( 2025-09-07T07:11:05.4176461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4176541Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4176832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4176925Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4177190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4177271Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4177598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4177771Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4178079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.4178161Z return input_tensor + hidden_states 2025-09-07T07:11:05.4178164Z 2025-09-07T07:11:05.4178280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4178501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4178573Z return mod(**inputs) 2025-09-07T07:11:05.4178866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4178934Z outputs = self.bert( 2025-09-07T07:11:05.4179235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4179313Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4179613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4179686Z layer_outputs = layer_module( 2025-09-07T07:11:05.4179915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4179994Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4180283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4180371Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4180664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4180743Z self_outputs = self.self( 2025-09-07T07:11:05.4181017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4181096Z return func(*args, **kwargs) 2025-09-07T07:11:05.4181421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4181507Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4181511Z 2025-09-07T07:11:05.4181627Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4181838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4181910Z return mod(**inputs) 2025-09-07T07:11:05.4182245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4182316Z outputs = self.bert( 2025-09-07T07:11:05.4182634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4182713Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4183034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4183109Z layer_outputs = layer_module( 2025-09-07T07:11:05.4183349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4183439Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4183748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4183842Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4184154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4184261Z self_outputs = self.self( 2025-09-07T07:11:05.4184547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4184624Z return func(*args, **kwargs) 2025-09-07T07:11:05.4184951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4185036Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4185039Z 2025-09-07T07:11:05.4185159Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4185376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4185447Z return mod(**inputs) 2025-09-07T07:11:05.4185842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4185923Z outputs = self.bert( 2025-09-07T07:11:05.4186241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4186319Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4186630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4186727Z layer_outputs = layer_module( 2025-09-07T07:11:05.4186965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4187062Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4187384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4187485Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4187824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4187904Z self_outputs = self.self( 2025-09-07T07:11:05.4188177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4188254Z return func(*args, **kwargs) 2025-09-07T07:11:05.4188576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4188663Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4188667Z 2025-09-07T07:11:05.4188757Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4188851Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4188960Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4189201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4189276Z return mod(**inputs) 2025-09-07T07:11:05.4189598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4189671Z outputs = self.bert( 2025-09-07T07:11:05.4189981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4190066Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4190376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4190460Z layer_outputs = layer_module( 2025-09-07T07:11:05.4190702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4190786Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4191105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4191228Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4191546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4191685Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4192001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4192090Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4192093Z 2025-09-07T07:11:05.4192202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4192424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4192493Z return mod(**inputs) 2025-09-07T07:11:05.4192812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4192885Z outputs = self.bert( 2025-09-07T07:11:05.4193194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4193280Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4193598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4193682Z layer_outputs = layer_module( 2025-09-07T07:11:05.4193919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4194003Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4194330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4194420Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4194722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4194805Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4195157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4195268Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4195573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4195668Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4195671Z 2025-09-07T07:11:05.4195799Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4196026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4196100Z return mod(**inputs) 2025-09-07T07:11:05.4196424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4196495Z outputs = self.bert( 2025-09-07T07:11:05.4196816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4196902Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4197226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4197309Z layer_outputs = layer_module( 2025-09-07T07:11:05.4197560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4197643Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4198008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4198097Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4198381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4198462Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4198815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4198934Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4199255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4199386Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4199617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4199707Z return self.act(input) 2025-09-07T07:11:05.4199710Z 2025-09-07T07:11:05.4199821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4200051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4200129Z return mod(**inputs) 2025-09-07T07:11:05.4200450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4200527Z outputs = self.bert( 2025-09-07T07:11:05.4200849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4200927Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4201256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4201355Z layer_outputs = layer_module( 2025-09-07T07:11:05.4201602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4201685Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4202009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4202098Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4202378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4202467Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4202834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4202984Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4203309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4203404Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4203408Z 2025-09-07T07:11:05.4203519Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4203751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4203831Z return mod(**inputs) 2025-09-07T07:11:05.4204154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4204231Z outputs = self.bert( 2025-09-07T07:11:05.4204549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4204628Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4204989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4205066Z layer_outputs = layer_module( 2025-09-07T07:11:05.4205312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4205395Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4205738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4205831Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4206149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4206233Z self_outputs = self.self( 2025-09-07T07:11:05.4206493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4206582Z return func(*args, **kwargs) 2025-09-07T07:11:05.4206893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-09-07T07:11:05.4206980Z query_layer = self.query(hidden_states) 2025-09-07T07:11:05.4206983Z 2025-09-07T07:11:05.4207099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4207310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4207387Z return mod(**inputs) 2025-09-07T07:11:05.4207696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4207768Z outputs = self.bert( 2025-09-07T07:11:05.4208082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4208180Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4208499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4208574Z layer_outputs = layer_module( 2025-09-07T07:11:05.4208817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4208899Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4209221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4209316Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4209654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4209737Z self_outputs = self.self( 2025-09-07T07:11:05.4210005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4210079Z return func(*args, **kwargs) 2025-09-07T07:11:05.4210396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-09-07T07:11:05.4210480Z key_layer = self.key(current_states) 2025-09-07T07:11:05.4210483Z 2025-09-07T07:11:05.4210602Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4210818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4210893Z return mod(**inputs) 2025-09-07T07:11:05.4211220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4211290Z outputs = self.bert( 2025-09-07T07:11:05.4211610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4211728Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4212045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4212118Z layer_outputs = layer_module( 2025-09-07T07:11:05.4212357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4212446Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4212764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4212857Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4213172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-09-07T07:11:05.4213257Z self_outputs = self.self( 2025-09-07T07:11:05.4213522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:11:05.4213599Z return func(*args, **kwargs) 2025-09-07T07:11:05.4213924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-09-07T07:11:05.4214011Z value_layer = self.value(current_states) 2025-09-07T07:11:05.4214014Z 2025-09-07T07:11:05.4214108Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4214194Z cudagraph partition due to non gpu ops 2025-09-07T07:11:05.4214305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4214530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4214601Z return mod(**inputs) 2025-09-07T07:11:05.4214949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4215024Z outputs = self.bert( 2025-09-07T07:11:05.4215336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4215421Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4215741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4215823Z layer_outputs = layer_module( 2025-09-07T07:11:05.4216062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4216152Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4216490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-09-07T07:11:05.4216584Z self_attention_outputs = self.attention( 2025-09-07T07:11:05.4216902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-09-07T07:11:05.4217040Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:11:05.4217358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-09-07T07:11:05.4217447Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4217450Z 2025-09-07T07:11:05.4217569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4217777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4217845Z return mod(**inputs) 2025-09-07T07:11:05.4218150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4218254Z outputs = self.bert( 2025-09-07T07:11:05.4218575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4218653Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4218963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4219047Z layer_outputs = layer_module( 2025-09-07T07:11:05.4219287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4219377Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4219809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4219906Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4220201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4220287Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4220641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4220756Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4221083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-09-07T07:11:05.4221175Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4221179Z 2025-09-07T07:11:05.4221291Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4221529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4221605Z return mod(**inputs) 2025-09-07T07:11:05.4221968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4222043Z outputs = self.bert( 2025-09-07T07:11:05.4222360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4222443Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4222752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4222835Z layer_outputs = layer_module( 2025-09-07T07:11:05.4223074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4223187Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4223501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4223594Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4223885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4223969Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4224319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-09-07T07:11:05.4224430Z intermediate_output = self.intermediate(ln_output) 2025-09-07T07:11:05.4224741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-09-07T07:11:05.4224874Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:05.4225105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:11:05.4225243Z return self.act(input) 2025-09-07T07:11:05.4225248Z 2025-09-07T07:11:05.4225361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4225582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4225653Z return mod(**inputs) 2025-09-07T07:11:05.4226022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4226106Z outputs = self.bert( 2025-09-07T07:11:05.4226423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4226510Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4226834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4226916Z layer_outputs = layer_module( 2025-09-07T07:11:05.4227165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4227244Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4227543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4227624Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4227896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4227973Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4228304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4228447Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4228770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-09-07T07:11:05.4228861Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:05.4228865Z 2025-09-07T07:11:05.4228970Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4229172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4229246Z return mod(**inputs) 2025-09-07T07:11:05.4229542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-09-07T07:11:05.4229616Z outputs = self.bert( 2025-09-07T07:11:05.4229926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-09-07T07:11:05.4230010Z encoder_outputs = self.encoder( 2025-09-07T07:11:05.4230316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-09-07T07:11:05.4230388Z layer_outputs = layer_module( 2025-09-07T07:11:05.4230622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:11:05.4230700Z return super().__call__(*args, **kwargs) 2025-09-07T07:11:05.4231002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-09-07T07:11:05.4231084Z layer_output = apply_chunking_to_forward( 2025-09-07T07:11:05.4231357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:11:05.4231440Z return forward_fn(*input_tensors) 2025-09-07T07:11:05.4231757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-09-07T07:11:05.4231923Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:11:05.4232215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-09-07T07:11:05.4232304Z return input_tensor + hidden_states 2025-09-07T07:11:05.4232307Z 2025-09-07T07:11:05.4232410Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4232610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4232687Z return mod(**inputs) 2025-09-07T07:11:05.4232980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1611, in forward 2025-09-07T07:11:05.4233079Z logits = self.qa_outputs(sequence_output) 2025-09-07T07:11:05.4233082Z 2025-09-07T07:11:05.4233186Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4233386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4233461Z return mod(**inputs) 2025-09-07T07:11:05.4233757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1629, in forward 2025-09-07T07:11:05.4233872Z start_loss = loss_fct(start_logits, start_positions) 2025-09-07T07:11:05.4233876Z 2025-09-07T07:11:05.4233979Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:05.4234183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:05.4234250Z return mod(**inputs) 2025-09-07T07:11:05.4234545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1630, in forward 2025-09-07T07:11:05.4234648Z end_loss = loss_fct(end_logits, end_positions) 2025-09-07T07:11:05.4234654Z 2025-09-07T07:11:18.0142329Z Compilation time (from dynamo_timed): 25.417988743 2025-09-07T07:11:18.0143097Z pass 2025-09-07T07:11:18.0143579Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:11:18.0145295Z TIMING: _recursive_pre_grad_passes:0.01123 _recursive_joint_graph_passes:0.78119 _recursive_post_grad_passes:0.13615 async_compile.wait:0.00338 code_gen:11.24849 inductor_compile:13.58102 backend_compile:19.88342 gc:0.00082 entire_frame_compile:25.41799 total_wall_time:25.41799 2025-09-07T07:11:18.0146586Z STATS: call_* op count: 724 | FakeTensorMode.__torch_dispatch__:28470 | FakeTensor.__torch_dispatch__:8283 | ProxyTorchDispatchMode.__torch_dispatch__:10973 2025-09-07T07:11:18.0147189Z Dynamo produced 1 graphs covering 724 ops with 0 graph breaks (0 unique) 2025-09-07T07:11:21.1056250Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:11:21.1057140Z import pynvml # type: ignore[import] 2025-09-07T07:11:23.8622398Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:11:23.8623319Z from pkg_resources import resource_filename 2025-09-07T07:11:24.5133363Z 2025-09-07T07:11:25.0587233Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:11:25.0591750Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:11:25.0657785Z cpu eval MobileBertForMaskedLM 2025-09-07T07:11:25.3431144Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:11:25.5104364Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:11:25.6685919Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:11:54.1734355Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.1734665Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.1734955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1735391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1735776Z return mod(**inputs) 2025-09-07T07:11:54.1736254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1736740Z outputs = self.mobilebert( 2025-09-07T07:11:54.1737252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-09-07T07:11:54.1737801Z embedding_output = self.embeddings( 2025-09-07T07:11:54.1738311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-09-07T07:11:54.1738815Z inputs_embeds = torch.cat( 2025-09-07T07:11:54.1738954Z 2025-09-07T07:11:54.1739081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1739507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1739881Z return mod(**inputs) 2025-09-07T07:11:54.1740324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-09-07T07:11:54.1740812Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:11:54.1741395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-09-07T07:11:54.1741892Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:11:54.1742741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-09-07T07:11:54.1743383Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-09-07T07:11:54.1743681Z 2025-09-07T07:11:54.1743809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1744212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1744589Z return mod(**inputs) 2025-09-07T07:11:54.1745043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1745534Z outputs = self.mobilebert( 2025-09-07T07:11:54.1746249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-09-07T07:11:54.1746740Z embedding_output = self.embeddings( 2025-09-07T07:11:54.1747224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-09-07T07:11:54.1747816Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-09-07T07:11:54.1748008Z 2025-09-07T07:11:54.1748131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1748512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1748875Z return mod(**inputs) 2025-09-07T07:11:54.1749296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1749759Z outputs = self.mobilebert( 2025-09-07T07:11:54.1750195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-09-07T07:11:54.1750742Z embedding_output = self.embeddings( 2025-09-07T07:11:54.1751202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-09-07T07:11:54.1751675Z embeddings = self.LayerNorm(embeddings) 2025-09-07T07:11:54.1752137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1752620Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1752785Z 2025-09-07T07:11:54.1752898Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1753291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1753649Z return mod(**inputs) 2025-09-07T07:11:54.1754078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1754527Z outputs = self.mobilebert( 2025-09-07T07:11:54.1754970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1755428Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1755884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1756358Z layer_outputs = layer_module( 2025-09-07T07:11:54.1756803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.1757377Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.1757959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.1758521Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.1759041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.1759495Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.1759650Z 2025-09-07T07:11:54.1759763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1760152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1760502Z return mod(**inputs) 2025-09-07T07:11:54.1760919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1761411Z outputs = self.mobilebert( 2025-09-07T07:11:54.1761902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1762353Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1762800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1763244Z layer_outputs = layer_module( 2025-09-07T07:11:54.1763676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1764145Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1764615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.1765057Z self_outputs = self.self( 2025-09-07T07:11:54.1765493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.1765954Z self.value(value_tensor) 2025-09-07T07:11:54.1766089Z 2025-09-07T07:11:54.1766202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1766643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1767002Z return mod(**inputs) 2025-09-07T07:11:54.1767418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1767870Z outputs = self.mobilebert( 2025-09-07T07:11:54.1768308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1768763Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1769214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1769656Z layer_outputs = layer_module( 2025-09-07T07:11:54.1770104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.1770659Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.1771210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.1771700Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.1772186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.1772627Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.1772784Z 2025-09-07T07:11:54.1772897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1773274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1773622Z return mod(**inputs) 2025-09-07T07:11:54.1774053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1774521Z outputs = self.mobilebert( 2025-09-07T07:11:54.1774959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1775390Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1775808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1776259Z layer_outputs = layer_module( 2025-09-07T07:11:54.1776701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.1777241Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.1777806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.1778314Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.1778802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.1779243Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.1779689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1780158Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1780318Z 2025-09-07T07:11:54.1780432Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1780828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1781179Z return mod(**inputs) 2025-09-07T07:11:54.1781601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1782114Z outputs = self.mobilebert( 2025-09-07T07:11:54.1782551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1782996Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1783451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1783919Z layer_outputs = layer_module( 2025-09-07T07:11:54.1784359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1784840Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1785298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.1785829Z self_outputs = self.self( 2025-09-07T07:11:54.1786271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.1786713Z self.query(query_tensor) 2025-09-07T07:11:54.1786846Z 2025-09-07T07:11:54.1786965Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1787334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1787671Z return mod(**inputs) 2025-09-07T07:11:54.1788066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1788513Z outputs = self.mobilebert( 2025-09-07T07:11:54.1788959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1789391Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1789817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1790294Z layer_outputs = layer_module( 2025-09-07T07:11:54.1790715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1791160Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1791593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.1792018Z self_outputs = self.self( 2025-09-07T07:11:54.1792445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.1792887Z self.key(key_tensor) 2025-09-07T07:11:54.1793010Z 2025-09-07T07:11:54.1793101Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.1793366Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.1793629Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1794038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1794403Z return mod(**inputs) 2025-09-07T07:11:54.1794843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1795295Z outputs = self.mobilebert( 2025-09-07T07:11:54.1795723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1796179Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1796624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1797073Z layer_outputs = layer_module( 2025-09-07T07:11:54.1797520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1798012Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1798471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.1798978Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.1799474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.1799935Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.1800086Z 2025-09-07T07:11:54.1800200Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1800596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1800953Z return mod(**inputs) 2025-09-07T07:11:54.1801370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1801807Z outputs = self.mobilebert( 2025-09-07T07:11:54.1802243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1802691Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1803133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1803574Z layer_outputs = layer_module( 2025-09-07T07:11:54.1804007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1804471Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1804927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.1805430Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.1805952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.1806454Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.1806957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1807425Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1807586Z 2025-09-07T07:11:54.1807708Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1808096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1808439Z return mod(**inputs) 2025-09-07T07:11:54.1808884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1809332Z outputs = self.mobilebert( 2025-09-07T07:11:54.1809794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1810262Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1810719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1811173Z layer_outputs = layer_module( 2025-09-07T07:11:54.1811620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1812103Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1812575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1813080Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1813584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.1814113Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.1814270Z 2025-09-07T07:11:54.1814393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1814788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1815161Z return mod(**inputs) 2025-09-07T07:11:54.1815597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1816078Z outputs = self.mobilebert( 2025-09-07T07:11:54.1816533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1817005Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1817462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1817938Z layer_outputs = layer_module( 2025-09-07T07:11:54.1818386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1818877Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1819355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1820086Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1820595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.1821113Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.1821306Z 2025-09-07T07:11:54.1821424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1821898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1822266Z return mod(**inputs) 2025-09-07T07:11:54.1822709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1823161Z outputs = self.mobilebert( 2025-09-07T07:11:54.1823596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1824078Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1824533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1825003Z layer_outputs = layer_module( 2025-09-07T07:11:54.1825484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1826044Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1826534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.1827055Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.1827575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.1828036Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.1828190Z 2025-09-07T07:11:54.1828303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1828692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1829048Z return mod(**inputs) 2025-09-07T07:11:54.1829471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1829983Z outputs = self.mobilebert( 2025-09-07T07:11:54.1830421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1830870Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1831311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1831844Z layer_outputs = layer_module( 2025-09-07T07:11:54.1832282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1832753Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1833227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.1833736Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.1834247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.1834741Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.1835259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1835731Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1835893Z 2025-09-07T07:11:54.1836012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1836401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1836750Z return mod(**inputs) 2025-09-07T07:11:54.1837172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1837619Z outputs = self.mobilebert( 2025-09-07T07:11:54.1838080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1838519Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1838960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1839403Z layer_outputs = layer_module( 2025-09-07T07:11:54.1839842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1840310Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1840767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1841276Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1841770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.1842235Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.1842388Z 2025-09-07T07:11:54.1842509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1842891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1843239Z return mod(**inputs) 2025-09-07T07:11:54.1843657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1844098Z outputs = self.mobilebert( 2025-09-07T07:11:54.1844529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1844969Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1845411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1845908Z layer_outputs = layer_module( 2025-09-07T07:11:54.1846342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1846817Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1847325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1847807Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1848273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.1848732Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.1848914Z 2025-09-07T07:11:54.1849023Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1849396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1849733Z return mod(**inputs) 2025-09-07T07:11:54.1850149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1850620Z outputs = self.mobilebert( 2025-09-07T07:11:54.1851032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1851459Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1851877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1852296Z layer_outputs = layer_module( 2025-09-07T07:11:54.1852702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1853149Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1853612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.1854087Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.1854569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.1855030Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.1855186Z 2025-09-07T07:11:54.1855299Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1855694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1856034Z return mod(**inputs) 2025-09-07T07:11:54.1856436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1856864Z outputs = self.mobilebert( 2025-09-07T07:11:54.1857271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1857693Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1858109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1858521Z layer_outputs = layer_module( 2025-09-07T07:11:54.1858938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1859393Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1859875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.1860394Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.1860943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.1861475Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.1861981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1862451Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1862614Z 2025-09-07T07:11:54.1862731Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1863118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1863469Z return mod(**inputs) 2025-09-07T07:11:54.1863889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1864339Z outputs = self.mobilebert( 2025-09-07T07:11:54.1864782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1865253Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1865782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1866269Z layer_outputs = layer_module( 2025-09-07T07:11:54.1866707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1867180Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1867652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1868145Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1868636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.1870029Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.1870214Z 2025-09-07T07:11:54.1870327Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1870723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1871110Z return mod(**inputs) 2025-09-07T07:11:54.1871535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1871981Z outputs = self.mobilebert( 2025-09-07T07:11:54.1872406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1872881Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1873328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1873784Z layer_outputs = layer_module( 2025-09-07T07:11:54.1874220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1874697Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1875171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1875671Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1876164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.1876656Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.1876846Z 2025-09-07T07:11:54.1876965Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1877385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1877804Z return mod(**inputs) 2025-09-07T07:11:54.1878223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1878687Z outputs = self.mobilebert( 2025-09-07T07:11:54.1879093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1879545Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1879992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1880449Z layer_outputs = layer_module( 2025-09-07T07:11:54.1880879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1881347Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1881788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.1882263Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.1882732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.1883169Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.1883321Z 2025-09-07T07:11:54.1883427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1883797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1884130Z return mod(**inputs) 2025-09-07T07:11:54.1884524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1884947Z outputs = self.mobilebert( 2025-09-07T07:11:54.1885380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1885834Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1886281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1886728Z layer_outputs = layer_module( 2025-09-07T07:11:54.1887164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1887650Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1888134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.1888658Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.1889130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.1889603Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.1890070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1890573Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1890733Z 2025-09-07T07:11:54.1890852Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1891234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1891581Z return mod(**inputs) 2025-09-07T07:11:54.1892011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1892460Z outputs = self.mobilebert( 2025-09-07T07:11:54.1892901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1893349Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1893759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1894179Z layer_outputs = layer_module( 2025-09-07T07:11:54.1894618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.1895100Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.1895567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.1896004Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.1896148Z 2025-09-07T07:11:54.1896273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1896663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1897006Z return mod(**inputs) 2025-09-07T07:11:54.1897422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1897867Z outputs = self.mobilebert( 2025-09-07T07:11:54.1898298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1898732Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1899162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1899602Z layer_outputs = layer_module( 2025-09-07T07:11:54.1900035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.1900553Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.1901042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.1901535Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.1901729Z 2025-09-07T07:11:54.1901853Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1902237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1902589Z return mod(**inputs) 2025-09-07T07:11:54.1903002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1903466Z outputs = self.mobilebert( 2025-09-07T07:11:54.1903928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1904398Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1904856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1905312Z layer_outputs = layer_module( 2025-09-07T07:11:54.1905868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.1906446Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.1907013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.1907513Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.1907680Z 2025-09-07T07:11:54.1907797Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1908190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1908603Z return mod(**inputs) 2025-09-07T07:11:54.1909031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1909473Z outputs = self.mobilebert( 2025-09-07T07:11:54.1909897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1910348Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1910787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1911230Z layer_outputs = layer_module( 2025-09-07T07:11:54.1911657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.1912201Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.1912744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.1913243Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.1913742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1914206Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1914374Z 2025-09-07T07:11:54.1914487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1914875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1915226Z return mod(**inputs) 2025-09-07T07:11:54.1915652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1916098Z outputs = self.mobilebert( 2025-09-07T07:11:54.1916555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1917007Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1917449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1917901Z layer_outputs = layer_module( 2025-09-07T07:11:54.1918338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.1918862Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.1919420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.1920060Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.1920572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.1921032Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.1921197Z 2025-09-07T07:11:54.1921311Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1921706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1922058Z return mod(**inputs) 2025-09-07T07:11:54.1922449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1922873Z outputs = self.mobilebert( 2025-09-07T07:11:54.1923287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1923708Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1924199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1924628Z layer_outputs = layer_module( 2025-09-07T07:11:54.1925071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.1925575Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.1926083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.1926555Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.1927017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.1927492Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.1927972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1928411Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1928561Z 2025-09-07T07:11:54.1928674Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1929036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1929363Z return mod(**inputs) 2025-09-07T07:11:54.1929757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1930175Z outputs = self.mobilebert( 2025-09-07T07:11:54.1930579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1931029Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1931497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1931924Z layer_outputs = layer_module( 2025-09-07T07:11:54.1932331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.1932833Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.1933345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.1933820Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.1934287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.1934753Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.1934896Z 2025-09-07T07:11:54.1935006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1935373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1935711Z return mod(**inputs) 2025-09-07T07:11:54.1936117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1936539Z outputs = self.mobilebert( 2025-09-07T07:11:54.1936941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1937365Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1937804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1938253Z layer_outputs = layer_module( 2025-09-07T07:11:54.1938695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1939222Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1939705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.1940173Z self_outputs = self.self( 2025-09-07T07:11:54.1940615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.1941080Z self.value(value_tensor) 2025-09-07T07:11:54.1941211Z 2025-09-07T07:11:54.1941322Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1941721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1942094Z return mod(**inputs) 2025-09-07T07:11:54.1942519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1943001Z outputs = self.mobilebert( 2025-09-07T07:11:54.1943484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1943965Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1944455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1944931Z layer_outputs = layer_module( 2025-09-07T07:11:54.1945369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.1946011Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.1946612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.1947128Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.1947635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.1948120Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.1948278Z 2025-09-07T07:11:54.1949542Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1949934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1950288Z return mod(**inputs) 2025-09-07T07:11:54.1950710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1951149Z outputs = self.mobilebert( 2025-09-07T07:11:54.1951598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1952060Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1952515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1952958Z layer_outputs = layer_module( 2025-09-07T07:11:54.1953406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.1953969Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.1954528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.1955029Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.1955519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.1955996Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.1956496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1956959Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1957110Z 2025-09-07T07:11:54.1957223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1957593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1957926Z return mod(**inputs) 2025-09-07T07:11:54.1958319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1958740Z outputs = self.mobilebert( 2025-09-07T07:11:54.1959145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1959562Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1960003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1960445Z layer_outputs = layer_module( 2025-09-07T07:11:54.1960881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1961332Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1961788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.1962227Z self_outputs = self.self( 2025-09-07T07:11:54.1962664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.1963079Z self.query(query_tensor) 2025-09-07T07:11:54.1963197Z 2025-09-07T07:11:54.1963304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1963672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1964025Z return mod(**inputs) 2025-09-07T07:11:54.1964448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1964888Z outputs = self.mobilebert( 2025-09-07T07:11:54.1965287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1965707Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1966122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1966547Z layer_outputs = layer_module( 2025-09-07T07:11:54.1966988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1967432Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1967865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.1968283Z self_outputs = self.self( 2025-09-07T07:11:54.1968687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.1969092Z self.key(key_tensor) 2025-09-07T07:11:54.1969205Z 2025-09-07T07:11:54.1969288Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.1969516Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.1969774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1970162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1970507Z return mod(**inputs) 2025-09-07T07:11:54.1970930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1971420Z outputs = self.mobilebert( 2025-09-07T07:11:54.1971860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1972299Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1972743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1973184Z layer_outputs = layer_module( 2025-09-07T07:11:54.1973618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1974075Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1974525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.1975028Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.1975529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.1975992Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.1976143Z 2025-09-07T07:11:54.1976262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1976641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1976996Z return mod(**inputs) 2025-09-07T07:11:54.1977416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1977858Z outputs = self.mobilebert( 2025-09-07T07:11:54.1978285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1978732Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1979194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1979642Z layer_outputs = layer_module( 2025-09-07T07:11:54.1980084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.1980548Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.1981019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.1981531Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.1982078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.1982591Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.1983093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.1983578Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.1983751Z 2025-09-07T07:11:54.1983868Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1984271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1984636Z return mod(**inputs) 2025-09-07T07:11:54.1985072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1985544Z outputs = self.mobilebert( 2025-09-07T07:11:54.1986084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1986540Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1986981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1987474Z layer_outputs = layer_module( 2025-09-07T07:11:54.1987919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1988397Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1988869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1989356Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1989844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.1990307Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.1990459Z 2025-09-07T07:11:54.1990582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1990979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1991329Z return mod(**inputs) 2025-09-07T07:11:54.1991750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1992195Z outputs = self.mobilebert( 2025-09-07T07:11:54.1992630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1993076Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.1993512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.1993959Z layer_outputs = layer_module( 2025-09-07T07:11:54.1994399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.1994875Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.1995371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.1995861Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.1996351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.1996846Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.1997027Z 2025-09-07T07:11:54.1997148Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.1997536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.1997892Z return mod(**inputs) 2025-09-07T07:11:54.1998337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.1998781Z outputs = self.mobilebert( 2025-09-07T07:11:54.1999196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.1999613Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2000043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2000467Z layer_outputs = layer_module( 2025-09-07T07:11:54.2000901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2001365Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2001827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2002327Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2002874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2003311Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2003455Z 2025-09-07T07:11:54.2003568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2003929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2004260Z return mod(**inputs) 2025-09-07T07:11:54.2004656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2005075Z outputs = self.mobilebert( 2025-09-07T07:11:54.2005475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2005904Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2006331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2006758Z layer_outputs = layer_module( 2025-09-07T07:11:54.2007169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2007607Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2008053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2008531Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2009007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2009514Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2010058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2010544Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2010715Z 2025-09-07T07:11:54.2010827Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2011218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2011576Z return mod(**inputs) 2025-09-07T07:11:54.2011998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2012446Z outputs = self.mobilebert( 2025-09-07T07:11:54.2012890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2013331Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2013752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2014190Z layer_outputs = layer_module( 2025-09-07T07:11:54.2014615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2015094Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2015571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2016065Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2016561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2017035Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2017195Z 2025-09-07T07:11:54.2017323Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2017747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2018115Z return mod(**inputs) 2025-09-07T07:11:54.2018547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2018995Z outputs = self.mobilebert( 2025-09-07T07:11:54.2019424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2020022Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2020482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2020944Z layer_outputs = layer_module( 2025-09-07T07:11:54.2021407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2021893Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2022359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2022852Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2023340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2023841Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2024027Z 2025-09-07T07:11:54.2024149Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2024538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2024903Z return mod(**inputs) 2025-09-07T07:11:54.2025347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2025873Z outputs = self.mobilebert( 2025-09-07T07:11:54.2026399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2026862Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2027306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2027752Z layer_outputs = layer_module( 2025-09-07T07:11:54.2028193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2028665Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2029152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2029660Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2030170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2030640Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2030793Z 2025-09-07T07:11:54.2030911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2031294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2031648Z return mod(**inputs) 2025-09-07T07:11:54.2032066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2032512Z outputs = self.mobilebert( 2025-09-07T07:11:54.2032941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2033389Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2033943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2034386Z layer_outputs = layer_module( 2025-09-07T07:11:54.2034821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2035289Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2035738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2036216Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2036691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2037166Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2037633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2038084Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2038244Z 2025-09-07T07:11:54.2038349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2038714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2039041Z return mod(**inputs) 2025-09-07T07:11:54.2039440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2039880Z outputs = self.mobilebert( 2025-09-07T07:11:54.2040314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2040760Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2041191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2041653Z layer_outputs = layer_module( 2025-09-07T07:11:54.2042071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2042528Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2042974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2043438Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2043906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2044352Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2044514Z 2025-09-07T07:11:54.2044629Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2045001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2045330Z return mod(**inputs) 2025-09-07T07:11:54.2045730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2046151Z outputs = self.mobilebert( 2025-09-07T07:11:54.2046559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2046977Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2047391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2047808Z layer_outputs = layer_module( 2025-09-07T07:11:54.2048223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2048698Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2049136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2049598Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2050054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2050514Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2050682Z 2025-09-07T07:11:54.2050795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2051152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2051508Z return mod(**inputs) 2025-09-07T07:11:54.2051934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2052356Z outputs = self.mobilebert( 2025-09-07T07:11:54.2052795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2053233Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2053681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2054136Z layer_outputs = layer_module( 2025-09-07T07:11:54.2054572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2055045Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2055524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2056026Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2056543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2057011Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2057163Z 2025-09-07T07:11:54.2057276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2057664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2058032Z return mod(**inputs) 2025-09-07T07:11:54.2058461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2058926Z outputs = self.mobilebert( 2025-09-07T07:11:54.2059332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2059751Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2060172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2060593Z layer_outputs = layer_module( 2025-09-07T07:11:54.2061011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2061486Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2061920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2062385Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2062856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2063325Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2063801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2064275Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2064433Z 2025-09-07T07:11:54.2064540Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2064906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2065230Z return mod(**inputs) 2025-09-07T07:11:54.2065627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2066112Z outputs = self.mobilebert( 2025-09-07T07:11:54.2066546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2067005Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2067439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2067876Z layer_outputs = layer_module( 2025-09-07T07:11:54.2068278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2068737Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2069187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2069608Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2069757Z 2025-09-07T07:11:54.2069862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2070240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2070592Z return mod(**inputs) 2025-09-07T07:11:54.2071000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2071463Z outputs = self.mobilebert( 2025-09-07T07:11:54.2071892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2072303Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2072705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2073107Z layer_outputs = layer_module( 2025-09-07T07:11:54.2073508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2073965Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2074439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2074891Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2075061Z 2025-09-07T07:11:54.2075167Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2075531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2075860Z return mod(**inputs) 2025-09-07T07:11:54.2076282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2076774Z outputs = self.mobilebert( 2025-09-07T07:11:54.2077214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2077668Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2078118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2078558Z layer_outputs = layer_module( 2025-09-07T07:11:54.2078992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2079530Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2080072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2080551Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2080714Z 2025-09-07T07:11:54.2080832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2081209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2081580Z return mod(**inputs) 2025-09-07T07:11:54.2082011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2082459Z outputs = self.mobilebert( 2025-09-07T07:11:54.2082904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2083353Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2083799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2084244Z layer_outputs = layer_module( 2025-09-07T07:11:54.2084676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2085213Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2085739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2086237Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2086756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2087225Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2087383Z 2025-09-07T07:11:54.2087500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2087880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2088228Z return mod(**inputs) 2025-09-07T07:11:54.2088649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2089083Z outputs = self.mobilebert( 2025-09-07T07:11:54.2089498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2089920Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2090341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2090763Z layer_outputs = layer_module( 2025-09-07T07:11:54.2091180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2091679Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2092189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2092666Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2093141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2093579Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2093759Z 2025-09-07T07:11:54.2093865Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2094233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2094567Z return mod(**inputs) 2025-09-07T07:11:54.2094969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2095391Z outputs = self.mobilebert( 2025-09-07T07:11:54.2095795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2096220Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2096638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2097061Z layer_outputs = layer_module( 2025-09-07T07:11:54.2097467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2097985Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2098497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2098969Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2099445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2099917Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2100417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2100889Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2101050Z 2025-09-07T07:11:54.2101173Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2101591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2101919Z return mod(**inputs) 2025-09-07T07:11:54.2102325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2102768Z outputs = self.mobilebert( 2025-09-07T07:11:54.2103198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2103644Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2104075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2104534Z layer_outputs = layer_module( 2025-09-07T07:11:54.2104974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2105535Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2106164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2106683Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2107170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2107673Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2107826Z 2025-09-07T07:11:54.2107945Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2108346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2108730Z return mod(**inputs) 2025-09-07T07:11:54.2109209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2109717Z outputs = self.mobilebert( 2025-09-07T07:11:54.2110155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2110630Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2111071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2111544Z layer_outputs = layer_module( 2025-09-07T07:11:54.2111994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2112494Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2112999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2113477Z self_outputs = self.self( 2025-09-07T07:11:54.2113911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2114382Z self.value(value_tensor) 2025-09-07T07:11:54.2114507Z 2025-09-07T07:11:54.2114619Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2115029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2115398Z return mod(**inputs) 2025-09-07T07:11:54.2115816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2116257Z outputs = self.mobilebert( 2025-09-07T07:11:54.2116682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2117136Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2117570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2117987Z layer_outputs = layer_module( 2025-09-07T07:11:54.2118398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2118902Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2119415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2120075Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2120579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2121012Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2121160Z 2025-09-07T07:11:54.2121269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2121641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2121975Z return mod(**inputs) 2025-09-07T07:11:54.2122374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2122787Z outputs = self.mobilebert( 2025-09-07T07:11:54.2123204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2123631Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2124050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2124473Z layer_outputs = layer_module( 2025-09-07T07:11:54.2124883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2125464Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2125997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2126469Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2126939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2127380Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2127825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2128289Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2128442Z 2025-09-07T07:11:54.2128558Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2128941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2129277Z return mod(**inputs) 2025-09-07T07:11:54.2129687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2130126Z outputs = self.mobilebert( 2025-09-07T07:11:54.2130552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2130972Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2131391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2131816Z layer_outputs = layer_module( 2025-09-07T07:11:54.2132235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2132709Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2133128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2133539Z self_outputs = self.self( 2025-09-07T07:11:54.2133935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2134348Z self.query(query_tensor) 2025-09-07T07:11:54.2134464Z 2025-09-07T07:11:54.2134577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2134930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2135256Z return mod(**inputs) 2025-09-07T07:11:54.2135658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2136076Z outputs = self.mobilebert( 2025-09-07T07:11:54.2136474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2136906Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2137322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2137743Z layer_outputs = layer_module( 2025-09-07T07:11:54.2138151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2138571Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2138992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2139400Z self_outputs = self.self( 2025-09-07T07:11:54.2139806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2140266Z self.key(key_tensor) 2025-09-07T07:11:54.2140380Z 2025-09-07T07:11:54.2140468Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2140704Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2140959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2141347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2141692Z return mod(**inputs) 2025-09-07T07:11:54.2142091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2142511Z outputs = self.mobilebert( 2025-09-07T07:11:54.2142920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2143346Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2143753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2144172Z layer_outputs = layer_module( 2025-09-07T07:11:54.2144584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2145018Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2145458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2146014Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2146522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2146979Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2147134Z 2025-09-07T07:11:54.2147272Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2147646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2147968Z return mod(**inputs) 2025-09-07T07:11:54.2148357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2148791Z outputs = self.mobilebert( 2025-09-07T07:11:54.2149222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2149661Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2150108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2150574Z layer_outputs = layer_module( 2025-09-07T07:11:54.2151016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2151481Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2151930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2152432Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2152931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2153433Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2153933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2154387Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2154540Z 2025-09-07T07:11:54.2154644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2155041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2155364Z return mod(**inputs) 2025-09-07T07:11:54.2155745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2156163Z outputs = self.mobilebert( 2025-09-07T07:11:54.2156571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2157018Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2157460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2157899Z layer_outputs = layer_module( 2025-09-07T07:11:54.2158338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2158798Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2159235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2159687Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2160126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2160559Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2160708Z 2025-09-07T07:11:54.2160815Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2161182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2161516Z return mod(**inputs) 2025-09-07T07:11:54.2161911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2162358Z outputs = self.mobilebert( 2025-09-07T07:11:54.2162757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2163166Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2163565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2163973Z layer_outputs = layer_module( 2025-09-07T07:11:54.2164371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2164805Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2165254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2165693Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2166144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2166592Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2166758Z 2025-09-07T07:11:54.2166868Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2167226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2167545Z return mod(**inputs) 2025-09-07T07:11:54.2167941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2168356Z outputs = self.mobilebert( 2025-09-07T07:11:54.2168753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2169173Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2169607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2170025Z layer_outputs = layer_module( 2025-09-07T07:11:54.2170457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2170891Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2171325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2171809Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2172275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2172712Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2172860Z 2025-09-07T07:11:54.2172971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2173330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2173664Z return mod(**inputs) 2025-09-07T07:11:54.2174062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2174485Z outputs = self.mobilebert( 2025-09-07T07:11:54.2174892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2175316Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2175732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2176153Z layer_outputs = layer_module( 2025-09-07T07:11:54.2176565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2177036Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2177473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2177952Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2178426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2178908Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2179372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2179831Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2179992Z 2025-09-07T07:11:54.2180099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2180315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2180382Z return mod(**inputs) 2025-09-07T07:11:54.2180666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2180746Z outputs = self.mobilebert( 2025-09-07T07:11:54.2181031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2181112Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2181396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2181468Z layer_outputs = layer_module( 2025-09-07T07:11:54.2181761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2181900Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2182206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2182327Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2182626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2182725Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2182729Z 2025-09-07T07:11:54.2182840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2183061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2183131Z return mod(**inputs) 2025-09-07T07:11:54.2183439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2183518Z outputs = self.mobilebert( 2025-09-07T07:11:54.2183822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2183906Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2184203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2184287Z layer_outputs = layer_module( 2025-09-07T07:11:54.2184585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2184687Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2184995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2185116Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2185448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2185571Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2185575Z 2025-09-07T07:11:54.2185692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2185978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2186052Z return mod(**inputs) 2025-09-07T07:11:54.2186360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2186437Z outputs = self.mobilebert( 2025-09-07T07:11:54.2186777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2186857Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2187160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2187245Z layer_outputs = layer_module( 2025-09-07T07:11:54.2187545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2187652Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2187936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2188068Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2188352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2188441Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2188444Z 2025-09-07T07:11:54.2188592Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2188797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2188872Z return mod(**inputs) 2025-09-07T07:11:54.2189164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2189238Z outputs = self.mobilebert( 2025-09-07T07:11:54.2189551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2189626Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2189936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2190012Z layer_outputs = layer_module( 2025-09-07T07:11:54.2190319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2190425Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2190727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2190867Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2191170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2191307Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2191609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2191710Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2191720Z 2025-09-07T07:11:54.2191831Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2192070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2192150Z return mod(**inputs) 2025-09-07T07:11:54.2192451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2192533Z outputs = self.mobilebert( 2025-09-07T07:11:54.2192843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2192919Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2193231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2193307Z layer_outputs = layer_module( 2025-09-07T07:11:54.2193636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2193740Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2194046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2194175Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2194479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2194576Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2194580Z 2025-09-07T07:11:54.2194692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2194914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2194985Z return mod(**inputs) 2025-09-07T07:11:54.2195286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2195411Z outputs = self.mobilebert( 2025-09-07T07:11:54.2195712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2195798Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2196097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2196173Z layer_outputs = layer_module( 2025-09-07T07:11:54.2196480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2196580Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2196891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2197012Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2197327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2197446Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2197450Z 2025-09-07T07:11:54.2197561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2197779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2197849Z return mod(**inputs) 2025-09-07T07:11:54.2198156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2198231Z outputs = self.mobilebert( 2025-09-07T07:11:54.2198531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2198617Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2198941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2199026Z layer_outputs = layer_module( 2025-09-07T07:11:54.2199325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2199433Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2199747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2199882Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2200189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2200320Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2200324Z 2025-09-07T07:11:54.2200448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2200662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2200732Z return mod(**inputs) 2025-09-07T07:11:54.2201041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2201115Z outputs = self.mobilebert( 2025-09-07T07:11:54.2201425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2201500Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2201788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2201862Z layer_outputs = layer_module( 2025-09-07T07:11:54.2202147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2202290Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2202577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2202709Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2202994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2203117Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2203410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2203506Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2203509Z 2025-09-07T07:11:54.2203622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2203829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2203903Z return mod(**inputs) 2025-09-07T07:11:54.2204192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2204265Z outputs = self.mobilebert( 2025-09-07T07:11:54.2204556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2204629Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2204920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2204992Z layer_outputs = layer_module( 2025-09-07T07:11:54.2205281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2205414Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2205717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2205814Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2205818Z 2025-09-07T07:11:54.2205924Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2206131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2206200Z return mod(**inputs) 2025-09-07T07:11:54.2206486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2206568Z outputs = self.mobilebert( 2025-09-07T07:11:54.2206875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2206962Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2207256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2207329Z layer_outputs = layer_module( 2025-09-07T07:11:54.2207625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2207746Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2208040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2208154Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2208158Z 2025-09-07T07:11:54.2208271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2208471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2208621Z return mod(**inputs) 2025-09-07T07:11:54.2208915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2208988Z outputs = self.mobilebert( 2025-09-07T07:11:54.2209296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2209375Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2209678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2209761Z layer_outputs = layer_module( 2025-09-07T07:11:54.2210065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2210248Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2210557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2210667Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2210671Z 2025-09-07T07:11:54.2210781Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2210992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2211070Z return mod(**inputs) 2025-09-07T07:11:54.2211375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2211459Z outputs = self.mobilebert( 2025-09-07T07:11:54.2211762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2211841Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2212185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2212274Z layer_outputs = layer_module( 2025-09-07T07:11:54.2212569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2212730Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2213021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2213147Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2213450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2213551Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2213555Z 2025-09-07T07:11:54.2213664Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2213874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2213940Z return mod(**inputs) 2025-09-07T07:11:54.2214232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2214303Z outputs = self.mobilebert( 2025-09-07T07:11:54.2214589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2214668Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2214951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2215031Z layer_outputs = layer_module( 2025-09-07T07:11:54.2215320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2215511Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2215798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2215923Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2216213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2216297Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2216301Z 2025-09-07T07:11:54.2216409Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2216609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2216677Z return mod(**inputs) 2025-09-07T07:11:54.2216967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2217041Z outputs = self.mobilebert( 2025-09-07T07:11:54.2217332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2217404Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2217690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2217768Z layer_outputs = layer_module( 2025-09-07T07:11:54.2218050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2218217Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2218504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2218654Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2218939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2219064Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2219355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2219450Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2219453Z 2025-09-07T07:11:54.2219693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2219954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2220026Z return mod(**inputs) 2025-09-07T07:11:54.2220335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2220416Z outputs = self.mobilebert( 2025-09-07T07:11:54.2220724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2220802Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2221109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2221187Z layer_outputs = layer_module( 2025-09-07T07:11:54.2221485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2221669Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2221971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2222167Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2222469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2222560Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2222572Z 2025-09-07T07:11:54.2222682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2222899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2222976Z return mod(**inputs) 2025-09-07T07:11:54.2223276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2223358Z outputs = self.mobilebert( 2025-09-07T07:11:54.2223658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2223738Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2224059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2224135Z layer_outputs = layer_module( 2025-09-07T07:11:54.2224441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2224534Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2224843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2224925Z self_outputs = self.self( 2025-09-07T07:11:54.2225225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2225308Z self.value(value_tensor) 2025-09-07T07:11:54.2225315Z 2025-09-07T07:11:54.2225460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2225676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2225790Z return mod(**inputs) 2025-09-07T07:11:54.2226103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2226188Z outputs = self.mobilebert( 2025-09-07T07:11:54.2226499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2226582Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2226914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2226992Z layer_outputs = layer_module( 2025-09-07T07:11:54.2227300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2227476Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2227794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2227913Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2228235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2228323Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2228327Z 2025-09-07T07:11:54.2228437Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2228661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2228733Z return mod(**inputs) 2025-09-07T07:11:54.2229082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2229158Z outputs = self.mobilebert( 2025-09-07T07:11:54.2229457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2229541Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2229840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2229923Z layer_outputs = layer_module( 2025-09-07T07:11:54.2230221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2230404Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2230705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2230827Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2231132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2231223Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2231541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2231635Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2231638Z 2025-09-07T07:11:54.2231747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2231946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2232013Z return mod(**inputs) 2025-09-07T07:11:54.2232306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2232395Z outputs = self.mobilebert( 2025-09-07T07:11:54.2232685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2232758Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2233037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2233115Z layer_outputs = layer_module( 2025-09-07T07:11:54.2233394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2233492Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2233806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2233888Z self_outputs = self.self( 2025-09-07T07:11:54.2234201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2234277Z self.query(query_tensor) 2025-09-07T07:11:54.2234281Z 2025-09-07T07:11:54.2234397Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2234606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2234979Z return mod(**inputs) 2025-09-07T07:11:54.2235408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2235839Z outputs = self.mobilebert( 2025-09-07T07:11:54.2236260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2236691Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2237187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2237623Z layer_outputs = layer_module( 2025-09-07T07:11:54.2238029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2238468Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2238907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2239337Z self_outputs = self.self( 2025-09-07T07:11:54.2239751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2240195Z self.key(key_tensor) 2025-09-07T07:11:54.2240322Z 2025-09-07T07:11:54.2240415Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2240664Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2240914Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2247027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2247481Z return mod(**inputs) 2025-09-07T07:11:54.2247922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2248371Z outputs = self.mobilebert( 2025-09-07T07:11:54.2248796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2249235Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2249686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2250107Z layer_outputs = layer_module( 2025-09-07T07:11:54.2250630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2251078Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2251520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2252049Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2252492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2252912Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2253060Z 2025-09-07T07:11:54.2253170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2253559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2253895Z return mod(**inputs) 2025-09-07T07:11:54.2254285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2254700Z outputs = self.mobilebert( 2025-09-07T07:11:54.2255102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2255520Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2255910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2256320Z layer_outputs = layer_module( 2025-09-07T07:11:54.2256727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2257154Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2257576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2258069Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2258537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2258987Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2259451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2259885Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2260035Z 2025-09-07T07:11:54.2260142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2260511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2260848Z return mod(**inputs) 2025-09-07T07:11:54.2261249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2261666Z outputs = self.mobilebert( 2025-09-07T07:11:54.2262059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2262473Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2262883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2263290Z layer_outputs = layer_module( 2025-09-07T07:11:54.2263685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2264118Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2264556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2265025Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2265510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2266208Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2266367Z 2025-09-07T07:11:54.2266474Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2266845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2267191Z return mod(**inputs) 2025-09-07T07:11:54.2267618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2268055Z outputs = self.mobilebert( 2025-09-07T07:11:54.2268517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2268979Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2269403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2269817Z layer_outputs = layer_module( 2025-09-07T07:11:54.2270233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2270675Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2271136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2271626Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2272119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2272586Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2272804Z 2025-09-07T07:11:54.2272916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2273286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2273624Z return mod(**inputs) 2025-09-07T07:11:54.2274017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2274441Z outputs = self.mobilebert( 2025-09-07T07:11:54.2274851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2275274Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2275700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2276121Z layer_outputs = layer_module( 2025-09-07T07:11:54.2276540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2276991Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2277436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2277913Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2278384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2278822Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2278971Z 2025-09-07T07:11:54.2279080Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2279463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2279800Z return mod(**inputs) 2025-09-07T07:11:54.2280218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2280655Z outputs = self.mobilebert( 2025-09-07T07:11:54.2281089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2281515Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2281934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2282352Z layer_outputs = layer_module( 2025-09-07T07:11:54.2282765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2283211Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2283661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2284147Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2284626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2285101Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2285579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2286008Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2286167Z 2025-09-07T07:11:54.2286275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2286654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2286982Z return mod(**inputs) 2025-09-07T07:11:54.2287362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2287801Z outputs = self.mobilebert( 2025-09-07T07:11:54.2288196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2288609Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2289022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2289432Z layer_outputs = layer_module( 2025-09-07T07:11:54.2289844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2290290Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2290767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2291260Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2291752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2292218Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2292369Z 2025-09-07T07:11:54.2292474Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2292842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2293174Z return mod(**inputs) 2025-09-07T07:11:54.2293561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2294057Z outputs = self.mobilebert( 2025-09-07T07:11:54.2294500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2294921Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2295351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2295775Z layer_outputs = layer_module( 2025-09-07T07:11:54.2296195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2296642Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2297089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2297550Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2298052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2298521Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2298696Z 2025-09-07T07:11:54.2298811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2299184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2299509Z return mod(**inputs) 2025-09-07T07:11:54.2299912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2300359Z outputs = self.mobilebert( 2025-09-07T07:11:54.2300789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2301243Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2301687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2302141Z layer_outputs = layer_module( 2025-09-07T07:11:54.2302589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2303091Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2303552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2304055Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2304563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2305022Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2305174Z 2025-09-07T07:11:54.2305293Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2305673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2306220Z return mod(**inputs) 2025-09-07T07:11:54.2306649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2307105Z outputs = self.mobilebert( 2025-09-07T07:11:54.2307547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2307992Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2308441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2308900Z layer_outputs = layer_module( 2025-09-07T07:11:54.2309342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2309807Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2310284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2310819Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2311321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2311821Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2312309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2312775Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2312943Z 2025-09-07T07:11:54.2313056Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2313446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2313815Z return mod(**inputs) 2025-09-07T07:11:54.2314232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2314684Z outputs = self.mobilebert( 2025-09-07T07:11:54.2315115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2315560Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2316007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2316443Z layer_outputs = layer_module( 2025-09-07T07:11:54.2316881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2317351Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2317817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2318291Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2318820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2319276Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2319433Z 2025-09-07T07:11:54.2319735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2320153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2320512Z return mod(**inputs) 2025-09-07T07:11:54.2320951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2321411Z outputs = self.mobilebert( 2025-09-07T07:11:54.2321834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2322264Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2322682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2323109Z layer_outputs = layer_module( 2025-09-07T07:11:54.2323548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2324020Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2324468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2324923Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2325385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2325850Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2326027Z 2025-09-07T07:11:54.2326141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2326597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2326928Z return mod(**inputs) 2025-09-07T07:11:54.2327329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2327752Z outputs = self.mobilebert( 2025-09-07T07:11:54.2328162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2328583Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2329004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2329460Z layer_outputs = layer_module( 2025-09-07T07:11:54.2329872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2330320Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2330757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2331237Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2331712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2332147Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2332289Z 2025-09-07T07:11:54.2332403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2332774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2333099Z return mod(**inputs) 2025-09-07T07:11:54.2333495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2333996Z outputs = self.mobilebert( 2025-09-07T07:11:54.2334420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2334865Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2335315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2335734Z layer_outputs = layer_module( 2025-09-07T07:11:54.2336148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2336574Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2337010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2337487Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2337951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2338409Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2338860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2339296Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2339450Z 2025-09-07T07:11:54.2339562Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2339954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2340305Z return mod(**inputs) 2025-09-07T07:11:54.2340716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2341194Z outputs = self.mobilebert( 2025-09-07T07:11:54.2341627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2342088Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2342523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2342962Z layer_outputs = layer_module( 2025-09-07T07:11:54.2343402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2343912Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2344428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2344896Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2345060Z 2025-09-07T07:11:54.2345175Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2345566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2345996Z return mod(**inputs) 2025-09-07T07:11:54.2346436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2346878Z outputs = self.mobilebert( 2025-09-07T07:11:54.2347289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2347713Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2348149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2348565Z layer_outputs = layer_module( 2025-09-07T07:11:54.2349013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2349478Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2349939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2350394Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2350561Z 2025-09-07T07:11:54.2350671Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2351025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2351352Z return mod(**inputs) 2025-09-07T07:11:54.2351753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2352170Z outputs = self.mobilebert( 2025-09-07T07:11:54.2352573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2352988Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2353400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2353819Z layer_outputs = layer_module( 2025-09-07T07:11:54.2354228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2354727Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2355241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2355680Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2355835Z 2025-09-07T07:11:54.2355957Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2356357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2356680Z return mod(**inputs) 2025-09-07T07:11:54.2357067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2357492Z outputs = self.mobilebert( 2025-09-07T07:11:54.2357891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2358298Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2358712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2359154Z layer_outputs = layer_module( 2025-09-07T07:11:54.2359565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2360076Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2360566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2361018Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2361470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2361901Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2362049Z 2025-09-07T07:11:54.2362161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2362516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2362845Z return mod(**inputs) 2025-09-07T07:11:54.2363238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2363684Z outputs = self.mobilebert( 2025-09-07T07:11:54.2364076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2364478Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2364892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2365286Z layer_outputs = layer_module( 2025-09-07T07:11:54.2365675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2366157Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2366632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2367080Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2367540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2367963Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2368101Z 2025-09-07T07:11:54.2368212Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2368561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2368882Z return mod(**inputs) 2025-09-07T07:11:54.2369274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2369670Z outputs = self.mobilebert( 2025-09-07T07:11:54.2370051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2370492Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2370901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2371368Z layer_outputs = layer_module( 2025-09-07T07:11:54.2371784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2372288Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2372799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2373279Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2373769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2374247Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2374711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2375160Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2375320Z 2025-09-07T07:11:54.2375427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2375794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2376128Z return mod(**inputs) 2025-09-07T07:11:54.2376527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2376947Z outputs = self.mobilebert( 2025-09-07T07:11:54.2377358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2377811Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2378219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2378640Z layer_outputs = layer_module( 2025-09-07T07:11:54.2379052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2379562Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2380077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2380534Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2380995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2381431Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2381574Z 2025-09-07T07:11:54.2381692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2382061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2382388Z return mod(**inputs) 2025-09-07T07:11:54.2382786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2383210Z outputs = self.mobilebert( 2025-09-07T07:11:54.2383624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2384057Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2384470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2384915Z layer_outputs = layer_module( 2025-09-07T07:11:54.2385396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2385942Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2386406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2386871Z self_outputs = self.self( 2025-09-07T07:11:54.2387332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2387790Z self.value(value_tensor) 2025-09-07T07:11:54.2387918Z 2025-09-07T07:11:54.2388039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2388453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2388810Z return mod(**inputs) 2025-09-07T07:11:54.2389255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2389687Z outputs = self.mobilebert( 2025-09-07T07:11:54.2390124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2390567Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2391016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2391464Z layer_outputs = layer_module( 2025-09-07T07:11:54.2391905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2392454Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2393006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2394421Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2394914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2395379Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2395529Z 2025-09-07T07:11:54.2395645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2396047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2396404Z return mod(**inputs) 2025-09-07T07:11:54.2396832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2397282Z outputs = self.mobilebert( 2025-09-07T07:11:54.2397704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2398156Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2398598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2399045Z layer_outputs = layer_module( 2025-09-07T07:11:54.2399542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2400044Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2400556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2401022Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2401480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2401945Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2402373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2402823Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2402982Z 2025-09-07T07:11:54.2403090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2403455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2403776Z return mod(**inputs) 2025-09-07T07:11:54.2404176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2404600Z outputs = self.mobilebert( 2025-09-07T07:11:54.2405029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2405458Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2405865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2406282Z layer_outputs = layer_module( 2025-09-07T07:11:54.2406694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2407129Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2407562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2407979Z self_outputs = self.self( 2025-09-07T07:11:54.2408462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2408887Z self.query(query_tensor) 2025-09-07T07:11:54.2409055Z 2025-09-07T07:11:54.2409167Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2409530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2409866Z return mod(**inputs) 2025-09-07T07:11:54.2410263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2410685Z outputs = self.mobilebert( 2025-09-07T07:11:54.2411095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2411517Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2411923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2412339Z layer_outputs = layer_module( 2025-09-07T07:11:54.2412742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2413174Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2413590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2413997Z self_outputs = self.self( 2025-09-07T07:11:54.2414394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2414803Z self.key(key_tensor) 2025-09-07T07:11:54.2414909Z 2025-09-07T07:11:54.2414999Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2415212Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2415453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2415820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2416144Z return mod(**inputs) 2025-09-07T07:11:54.2416551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2416968Z outputs = self.mobilebert( 2025-09-07T07:11:54.2417369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2417786Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2418187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2418599Z layer_outputs = layer_module( 2025-09-07T07:11:54.2419004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2419491Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2420107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2420593Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2421072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2421515Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2421660Z 2025-09-07T07:11:54.2421775Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2422148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2422475Z return mod(**inputs) 2025-09-07T07:11:54.2422874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2423298Z outputs = self.mobilebert( 2025-09-07T07:11:54.2423708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2424206Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2424616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2425040Z layer_outputs = layer_module( 2025-09-07T07:11:54.2425454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2425947Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2426387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2426867Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2427388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2427873Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2428357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2428798Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2428959Z 2025-09-07T07:11:54.2429069Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2429439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2429781Z return mod(**inputs) 2025-09-07T07:11:54.2430185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2430604Z outputs = self.mobilebert( 2025-09-07T07:11:54.2431021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2431452Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2431915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2432346Z layer_outputs = layer_module( 2025-09-07T07:11:54.2432755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2433205Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2433655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2434121Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2434629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2435061Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2435216Z 2025-09-07T07:11:54.2435324Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2435694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2436035Z return mod(**inputs) 2025-09-07T07:11:54.2436405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2436805Z outputs = self.mobilebert( 2025-09-07T07:11:54.2437193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2437593Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2437989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2438379Z layer_outputs = layer_module( 2025-09-07T07:11:54.2438772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2439233Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2439672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2440121Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2440562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2441025Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2441191Z 2025-09-07T07:11:54.2441293Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2441644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2441961Z return mod(**inputs) 2025-09-07T07:11:54.2442337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2442742Z outputs = self.mobilebert( 2025-09-07T07:11:54.2443129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2443530Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2443922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2444325Z layer_outputs = layer_module( 2025-09-07T07:11:54.2444718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2445151Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2445570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2446040Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2446500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2446919Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2447061Z 2025-09-07T07:11:54.2447174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2447536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2447856Z return mod(**inputs) 2025-09-07T07:11:54.2448249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2448681Z outputs = self.mobilebert( 2025-09-07T07:11:54.2449089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2449498Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2449912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2450335Z layer_outputs = layer_module( 2025-09-07T07:11:54.2450744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2451190Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2451472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2451605Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2451890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2452075Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2452371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2452467Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2452470Z 2025-09-07T07:11:54.2452580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2452789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2452853Z return mod(**inputs) 2025-09-07T07:11:54.2453130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2453200Z outputs = self.mobilebert( 2025-09-07T07:11:54.2453494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2453571Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2453861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2453934Z layer_outputs = layer_module( 2025-09-07T07:11:54.2454218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2454323Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2454611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2454734Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2455021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2455108Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2455122Z 2025-09-07T07:11:54.2455242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2455446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2455520Z return mod(**inputs) 2025-09-07T07:11:54.2455807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2455888Z outputs = self.mobilebert( 2025-09-07T07:11:54.2456175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2456250Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2456575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2456655Z layer_outputs = layer_module( 2025-09-07T07:11:54.2456969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2457075Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2457379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2457509Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2457811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2457938Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2457942Z 2025-09-07T07:11:54.2458056Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2458292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2458361Z return mod(**inputs) 2025-09-07T07:11:54.2458690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2458773Z outputs = self.mobilebert( 2025-09-07T07:11:54.2459057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2459137Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2459420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2459495Z layer_outputs = layer_module( 2025-09-07T07:11:54.2459799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2459902Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2460209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2460349Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2460657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2460749Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2460753Z 2025-09-07T07:11:54.2460865Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2461083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2461153Z return mod(**inputs) 2025-09-07T07:11:54.2461459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2461536Z outputs = self.mobilebert( 2025-09-07T07:11:54.2461835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2461946Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2462249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2462332Z layer_outputs = layer_module( 2025-09-07T07:11:54.2462632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2462740Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2463049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2463184Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2463507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2463642Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2463955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2464056Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2464060Z 2025-09-07T07:11:54.2464172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2464394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2464465Z return mod(**inputs) 2025-09-07T07:11:54.2464774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2464850Z outputs = self.mobilebert( 2025-09-07T07:11:54.2465157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2465267Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2465571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2465656Z layer_outputs = layer_module( 2025-09-07T07:11:54.2466102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2466233Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2466532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2466651Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2466968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2467058Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2467067Z 2025-09-07T07:11:54.2467191Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2467395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2467470Z return mod(**inputs) 2025-09-07T07:11:54.2467752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2467823Z outputs = self.mobilebert( 2025-09-07T07:11:54.2468120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2468193Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2468485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2468557Z layer_outputs = layer_module( 2025-09-07T07:11:54.2468870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2468978Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2469265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2469385Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2469672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2469791Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2469795Z 2025-09-07T07:11:54.2469897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2470118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2470198Z return mod(**inputs) 2025-09-07T07:11:54.2470484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2470570Z outputs = self.mobilebert( 2025-09-07T07:11:54.2470853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2470927Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2471219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2471288Z layer_outputs = layer_module( 2025-09-07T07:11:54.2471565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2471655Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2471939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2472094Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2472370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2472464Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2472468Z 2025-09-07T07:11:54.2472572Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2472782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2472850Z return mod(**inputs) 2025-09-07T07:11:54.2473132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2473212Z outputs = self.mobilebert( 2025-09-07T07:11:54.2473493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2473588Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2473863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2473941Z layer_outputs = layer_module( 2025-09-07T07:11:54.2474216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2474307Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2474589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2474711Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2474994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2475115Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2475408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2475509Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2475513Z 2025-09-07T07:11:54.2475617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2475824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2475891Z return mod(**inputs) 2025-09-07T07:11:54.2476193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2476262Z outputs = self.mobilebert( 2025-09-07T07:11:54.2476558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2476643Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2476917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2476993Z layer_outputs = layer_module( 2025-09-07T07:11:54.2477272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2477401Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2477680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2477760Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2477763Z 2025-09-07T07:11:54.2477869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2478060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2478166Z return mod(**inputs) 2025-09-07T07:11:54.2478443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2478510Z outputs = self.mobilebert( 2025-09-07T07:11:54.2478787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2478857Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2479137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2479205Z layer_outputs = layer_module( 2025-09-07T07:11:54.2479480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2479607Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2479887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2480007Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2480012Z 2025-09-07T07:11:54.2480112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2480312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2480378Z return mod(**inputs) 2025-09-07T07:11:54.2480655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2480734Z outputs = self.mobilebert( 2025-09-07T07:11:54.2481011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2481089Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2481371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2481462Z layer_outputs = layer_module( 2025-09-07T07:11:54.2481748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2481907Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2482194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2482289Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2482293Z 2025-09-07T07:11:54.2482402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2482621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2482689Z return mod(**inputs) 2025-09-07T07:11:54.2482980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2483053Z outputs = self.mobilebert( 2025-09-07T07:11:54.2483345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2483416Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2483704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2483782Z layer_outputs = layer_module( 2025-09-07T07:11:54.2484062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2484232Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2484513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2484672Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2484950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2485040Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2485043Z 2025-09-07T07:11:54.2485151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2485348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2485419Z return mod(**inputs) 2025-09-07T07:11:54.2485696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2485767Z outputs = self.mobilebert( 2025-09-07T07:11:54.2486053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2486129Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2486421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2486494Z layer_outputs = layer_module( 2025-09-07T07:11:54.2486786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2486948Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2487230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2487362Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2487649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2487761Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2487765Z 2025-09-07T07:11:54.2487871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2488082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2488150Z return mod(**inputs) 2025-09-07T07:11:54.2488435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2488515Z outputs = self.mobilebert( 2025-09-07T07:11:54.2488799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2488880Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2489188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2489263Z layer_outputs = layer_module( 2025-09-07T07:11:54.2489555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2489714Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2490003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2490124Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2490419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2490542Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2490830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2490963Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2490967Z 2025-09-07T07:11:54.2491070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2491277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2491343Z return mod(**inputs) 2025-09-07T07:11:54.2491630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2491710Z outputs = self.mobilebert( 2025-09-07T07:11:54.2491996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2492077Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2492362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2492440Z layer_outputs = layer_module( 2025-09-07T07:11:54.2492730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2492891Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2493185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2493299Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2493590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2493673Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2493677Z 2025-09-07T07:11:54.2493784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2493990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2494060Z return mod(**inputs) 2025-09-07T07:11:54.2494384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2494455Z outputs = self.mobilebert( 2025-09-07T07:11:54.2494748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2494820Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2495107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2495187Z layer_outputs = layer_module( 2025-09-07T07:11:54.2495491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2495589Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2495878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2495951Z self_outputs = self.self( 2025-09-07T07:11:54.2496241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2496312Z self.value(value_tensor) 2025-09-07T07:11:54.2496316Z 2025-09-07T07:11:54.2496427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2496623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2496696Z return mod(**inputs) 2025-09-07T07:11:54.2496977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2497050Z outputs = self.mobilebert( 2025-09-07T07:11:54.2497340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2497447Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2497743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2497816Z layer_outputs = layer_module( 2025-09-07T07:11:54.2498104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2498277Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2498570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2498694Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2498982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2499075Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2499079Z 2025-09-07T07:11:54.2499184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2499388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2499462Z return mod(**inputs) 2025-09-07T07:11:54.2499753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2499832Z outputs = self.mobilebert( 2025-09-07T07:11:54.2500122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2500195Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2500494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2500568Z layer_outputs = layer_module( 2025-09-07T07:11:54.2500884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2501050Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2501346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2501464Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2501764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2501865Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2502187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2502298Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2502303Z 2025-09-07T07:11:54.2502408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2502611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2502687Z return mod(**inputs) 2025-09-07T07:11:54.2502973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2503051Z outputs = self.mobilebert( 2025-09-07T07:11:54.2503356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2503442Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2503749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2503824Z layer_outputs = layer_module( 2025-09-07T07:11:54.2504171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2504263Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2504579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2504655Z self_outputs = self.self( 2025-09-07T07:11:54.2504969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2505049Z self.query(query_tensor) 2025-09-07T07:11:54.2505052Z 2025-09-07T07:11:54.2505157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2505370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2505436Z return mod(**inputs) 2025-09-07T07:11:54.2505793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2505879Z outputs = self.mobilebert( 2025-09-07T07:11:54.2506166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2506250Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2506532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2506613Z layer_outputs = layer_module( 2025-09-07T07:11:54.2506893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2506988Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2507277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2507352Z self_outputs = self.self( 2025-09-07T07:11:54.2507662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2507729Z self.key(key_tensor) 2025-09-07T07:11:54.2507734Z 2025-09-07T07:11:54.2507815Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2507901Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2508005Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2508211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2508277Z return mod(**inputs) 2025-09-07T07:11:54.2508573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2508649Z outputs = self.mobilebert( 2025-09-07T07:11:54.2508926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2509007Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2509279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2509358Z layer_outputs = layer_module( 2025-09-07T07:11:54.2509632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2509712Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2509994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2510116Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2510399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2510515Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2510518Z 2025-09-07T07:11:54.2510626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2510821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2510887Z return mod(**inputs) 2025-09-07T07:11:54.2511175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2511247Z outputs = self.mobilebert( 2025-09-07T07:11:54.2511536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2511609Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2511903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2511984Z layer_outputs = layer_module( 2025-09-07T07:11:54.2512260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2512350Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2512624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2512742Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2513024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2513148Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2513430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2513525Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2513529Z 2025-09-07T07:11:54.2513652Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2513849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2513917Z return mod(**inputs) 2025-09-07T07:11:54.2514218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2514286Z outputs = self.mobilebert( 2025-09-07T07:11:54.2514572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2514643Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2514936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2515015Z layer_outputs = layer_module( 2025-09-07T07:11:54.2515299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2515403Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2515679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2515796Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2516073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2516156Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2516160Z 2025-09-07T07:11:54.2516269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2516464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2516536Z return mod(**inputs) 2025-09-07T07:11:54.2516844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2516913Z outputs = self.mobilebert( 2025-09-07T07:11:54.2517195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2517267Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2517551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2517622Z layer_outputs = layer_module( 2025-09-07T07:11:54.2517902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2517995Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2518269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2518400Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2518670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2518784Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2518787Z 2025-09-07T07:11:54.2518884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2519077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2519140Z return mod(**inputs) 2025-09-07T07:11:54.2519405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2519482Z outputs = self.mobilebert( 2025-09-07T07:11:54.2519941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2520080Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2520362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2520434Z layer_outputs = layer_module( 2025-09-07T07:11:54.2520721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2520815Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2521104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2521230Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2521530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2521624Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2521629Z 2025-09-07T07:11:54.2521732Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2521948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2522011Z return mod(**inputs) 2025-09-07T07:11:54.2522291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2522358Z outputs = self.mobilebert( 2025-09-07T07:11:54.2522630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2522708Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2522984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2523103Z layer_outputs = layer_module( 2025-09-07T07:11:54.2523374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2523466Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2523743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2523865Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2524142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2524259Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2524538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2524625Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2524635Z 2025-09-07T07:11:54.2524733Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2524930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2524995Z return mod(**inputs) 2025-09-07T07:11:54.2525274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2525342Z outputs = self.mobilebert( 2025-09-07T07:11:54.2525611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2525688Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2525959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2526033Z layer_outputs = layer_module( 2025-09-07T07:11:54.2526325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2526427Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2526705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2526816Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2527116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2527196Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2527199Z 2025-09-07T07:11:54.2527303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2527510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2527574Z return mod(**inputs) 2025-09-07T07:11:54.2527855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2527924Z outputs = self.mobilebert( 2025-09-07T07:11:54.2528199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2528267Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2528544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2528612Z layer_outputs = layer_module( 2025-09-07T07:11:54.2528882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2528981Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2529255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2529417Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2529690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2529800Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2529811Z 2025-09-07T07:11:54.2529912Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2530111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2530184Z return mod(**inputs) 2025-09-07T07:11:54.2530465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2530546Z outputs = self.mobilebert( 2025-09-07T07:11:54.2530829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2530908Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2531200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2531275Z layer_outputs = layer_module( 2025-09-07T07:11:54.2531563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2531659Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2531943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2532077Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2532360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2532454Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2532479Z 2025-09-07T07:11:54.2532581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2532785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2532850Z return mod(**inputs) 2025-09-07T07:11:54.2533126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2533203Z outputs = self.mobilebert( 2025-09-07T07:11:54.2533474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2533552Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2533848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2533921Z layer_outputs = layer_module( 2025-09-07T07:11:54.2534210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2534303Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2534590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2534713Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2535002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2535125Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2535409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2535507Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2535537Z 2025-09-07T07:11:54.2535639Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2535842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2535907Z return mod(**inputs) 2025-09-07T07:11:54.2536194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2536265Z outputs = self.mobilebert( 2025-09-07T07:11:54.2536544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2536623Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2536901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2536977Z layer_outputs = layer_module( 2025-09-07T07:11:54.2537260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2537352Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2537634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2537743Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2538024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2538107Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2538110Z 2025-09-07T07:11:54.2538217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2538414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2538479Z return mod(**inputs) 2025-09-07T07:11:54.2538821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2538892Z outputs = self.mobilebert( 2025-09-07T07:11:54.2539177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2539247Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2539524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2539603Z layer_outputs = layer_module( 2025-09-07T07:11:54.2539888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2540008Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2540292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2540408Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2540702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2540813Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2540817Z 2025-09-07T07:11:54.2540926Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2541129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2541202Z return mod(**inputs) 2025-09-07T07:11:54.2541492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2541568Z outputs = self.mobilebert( 2025-09-07T07:11:54.2541857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2541968Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2542258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2542328Z layer_outputs = layer_module( 2025-09-07T07:11:54.2542609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2542709Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2542991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2543122Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2543408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2543506Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2543510Z 2025-09-07T07:11:54.2543615Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2543816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2543891Z return mod(**inputs) 2025-09-07T07:11:54.2544171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2544251Z outputs = self.mobilebert( 2025-09-07T07:11:54.2544549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2544628Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2544934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2545009Z layer_outputs = layer_module( 2025-09-07T07:11:54.2545334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2545436Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2545806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2545948Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2546254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2546392Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2546717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2546826Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2546833Z 2025-09-07T07:11:54.2546944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2547168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2547239Z return mod(**inputs) 2025-09-07T07:11:54.2547538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2547623Z outputs = self.mobilebert( 2025-09-07T07:11:54.2547913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2547992Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2548274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2548347Z layer_outputs = layer_module( 2025-09-07T07:11:54.2548663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2548786Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2549072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2549155Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2549158Z 2025-09-07T07:11:54.2549264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2549466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2549532Z return mod(**inputs) 2025-09-07T07:11:54.2549825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2549895Z outputs = self.mobilebert( 2025-09-07T07:11:54.2550190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2550267Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2550554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2550636Z layer_outputs = layer_module( 2025-09-07T07:11:54.2550923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2551051Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2551344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2551465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2551469Z 2025-09-07T07:11:54.2551568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2551786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2551860Z return mod(**inputs) 2025-09-07T07:11:54.2552136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2552215Z outputs = self.mobilebert( 2025-09-07T07:11:54.2552495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2552567Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2552858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2552930Z layer_outputs = layer_module( 2025-09-07T07:11:54.2553235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2553407Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2553698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2553795Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2553798Z 2025-09-07T07:11:54.2553902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2554112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2554178Z return mod(**inputs) 2025-09-07T07:11:54.2554472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2554545Z outputs = self.mobilebert( 2025-09-07T07:11:54.2554829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2554961Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2555247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2555327Z layer_outputs = layer_module( 2025-09-07T07:11:54.2555608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2555775Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2556060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2556185Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2556479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2556577Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2556581Z 2025-09-07T07:11:54.2556691Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2556893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2556959Z return mod(**inputs) 2025-09-07T07:11:54.2557248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2557319Z outputs = self.mobilebert( 2025-09-07T07:11:54.2557608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2557679Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2557969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2558044Z layer_outputs = layer_module( 2025-09-07T07:11:54.2558343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2558513Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2558800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2558932Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2559218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2559305Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2559317Z 2025-09-07T07:11:54.2559437Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2559640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2559715Z return mod(**inputs) 2025-09-07T07:11:54.2559999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2560077Z outputs = self.mobilebert( 2025-09-07T07:11:54.2560361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2560435Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2560727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2560798Z layer_outputs = layer_module( 2025-09-07T07:11:54.2561093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2561253Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2561573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2561705Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2561998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2562124Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2562402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2562503Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2562507Z 2025-09-07T07:11:54.2562615Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2562814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2562893Z return mod(**inputs) 2025-09-07T07:11:54.2563178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2563258Z outputs = self.mobilebert( 2025-09-07T07:11:54.2563537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2563612Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2563897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2563971Z layer_outputs = layer_module( 2025-09-07T07:11:54.2564258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2564424Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2564731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2564843Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2565113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2565201Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2565205Z 2025-09-07T07:11:54.2565306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2565511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2565576Z return mod(**inputs) 2025-09-07T07:11:54.2565868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2565950Z outputs = self.mobilebert( 2025-09-07T07:11:54.2566231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2566311Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2566590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2566674Z layer_outputs = layer_module( 2025-09-07T07:11:54.2566950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2567037Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2567320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2567392Z self_outputs = self.self( 2025-09-07T07:11:54.2567676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2567777Z self.value(value_tensor) 2025-09-07T07:11:54.2567781Z 2025-09-07T07:11:54.2567880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2568083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2568148Z return mod(**inputs) 2025-09-07T07:11:54.2568429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2568497Z outputs = self.mobilebert( 2025-09-07T07:11:54.2568782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2568856Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2569134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2569215Z layer_outputs = layer_module( 2025-09-07T07:11:54.2569492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2569656Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2569939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2570048Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2570336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2570420Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2570425Z 2025-09-07T07:11:54.2570539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2570739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2570842Z return mod(**inputs) 2025-09-07T07:11:54.2571122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2571191Z outputs = self.mobilebert( 2025-09-07T07:11:54.2571476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2571546Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2571826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2571894Z layer_outputs = layer_module( 2025-09-07T07:11:54.2572183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2572353Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2572633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2572748Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2573023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2573116Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2573393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2573485Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2573488Z 2025-09-07T07:11:54.2573599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2573795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2573901Z return mod(**inputs) 2025-09-07T07:11:54.2574176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2574245Z outputs = self.mobilebert( 2025-09-07T07:11:54.2574528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2574599Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2574881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2574953Z layer_outputs = layer_module( 2025-09-07T07:11:54.2575242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2575326Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2575604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2575686Z self_outputs = self.self( 2025-09-07T07:11:54.2575960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2576037Z self.query(query_tensor) 2025-09-07T07:11:54.2576040Z 2025-09-07T07:11:54.2576141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2576332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2576404Z return mod(**inputs) 2025-09-07T07:11:54.2576677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2576753Z outputs = self.mobilebert( 2025-09-07T07:11:54.2577028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2577127Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2577402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2577485Z layer_outputs = layer_module( 2025-09-07T07:11:54.2577762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2577845Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2578118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2578186Z self_outputs = self.self( 2025-09-07T07:11:54.2578475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2578551Z self.key(key_tensor) 2025-09-07T07:11:54.2578557Z 2025-09-07T07:11:54.2578638Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2578724Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2578823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2579015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2579086Z return mod(**inputs) 2025-09-07T07:11:54.2579371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2579448Z outputs = self.mobilebert( 2025-09-07T07:11:54.2579732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2579814Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2580103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2580209Z layer_outputs = layer_module( 2025-09-07T07:11:54.2580500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2580586Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2580884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2581011Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2581304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2581399Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2581402Z 2025-09-07T07:11:54.2581508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2581711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2581783Z return mod(**inputs) 2025-09-07T07:11:54.2582071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2582142Z outputs = self.mobilebert( 2025-09-07T07:11:54.2582421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2582501Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2582780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2582858Z layer_outputs = layer_module( 2025-09-07T07:11:54.2583139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2583222Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2583531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2583655Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2583945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2584075Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2584381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2584479Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2584482Z 2025-09-07T07:11:54.2584591Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2584827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2584903Z return mod(**inputs) 2025-09-07T07:11:54.2585214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2585291Z outputs = self.mobilebert( 2025-09-07T07:11:54.2585589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2585677Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2586071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2586160Z layer_outputs = layer_module( 2025-09-07T07:11:54.2586472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2586586Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2586896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2587058Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2587367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2587465Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2587469Z 2025-09-07T07:11:54.2587579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2587782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2587850Z return mod(**inputs) 2025-09-07T07:11:54.2588150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2588222Z outputs = self.mobilebert( 2025-09-07T07:11:54.2588506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2588584Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2588863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2588933Z layer_outputs = layer_module( 2025-09-07T07:11:54.2589206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2589307Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2589582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2589696Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2589974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2590107Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2590117Z 2025-09-07T07:11:54.2590217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2590408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2590480Z return mod(**inputs) 2025-09-07T07:11:54.2590757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2590834Z outputs = self.mobilebert( 2025-09-07T07:11:54.2591112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2591184Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2591494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2591570Z layer_outputs = layer_module( 2025-09-07T07:11:54.2591863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2591957Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2592232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2592363Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2592636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2592729Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2592732Z 2025-09-07T07:11:54.2592835Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2593040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2593138Z return mod(**inputs) 2025-09-07T07:11:54.2593420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2593498Z outputs = self.mobilebert( 2025-09-07T07:11:54.2593781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2593858Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2594137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2594208Z layer_outputs = layer_module( 2025-09-07T07:11:54.2594496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2594589Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2594882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2595009Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2595299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2595424Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2595708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2595809Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2595812Z 2025-09-07T07:11:54.2595913Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2596125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2596193Z return mod(**inputs) 2025-09-07T07:11:54.2596496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2596601Z outputs = self.mobilebert( 2025-09-07T07:11:54.2596876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2596955Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2597228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2597307Z layer_outputs = layer_module( 2025-09-07T07:11:54.2597587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2597696Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2597984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2598098Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2598386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2598470Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2598474Z 2025-09-07T07:11:54.2598575Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2598780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2598846Z return mod(**inputs) 2025-09-07T07:11:54.2599135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2599208Z outputs = self.mobilebert( 2025-09-07T07:11:54.2599494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2599607Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2599884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2599961Z layer_outputs = layer_module( 2025-09-07T07:11:54.2600238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2600336Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2600616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2600730Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2601025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2601141Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2601145Z 2025-09-07T07:11:54.2601256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2601455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2601527Z return mod(**inputs) 2025-09-07T07:11:54.2601820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2601889Z outputs = self.mobilebert( 2025-09-07T07:11:54.2602174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2602247Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2602535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2602607Z layer_outputs = layer_module( 2025-09-07T07:11:54.2602911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2603011Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2603297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2603426Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2603705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2603796Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2603801Z 2025-09-07T07:11:54.2603920Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2604116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2604192Z return mod(**inputs) 2025-09-07T07:11:54.2604468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2604545Z outputs = self.mobilebert( 2025-09-07T07:11:54.2604822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2604894Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2605181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2605250Z layer_outputs = layer_module( 2025-09-07T07:11:54.2605543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2605635Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2605951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2606075Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2606353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2606483Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2606760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2606857Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2606860Z 2025-09-07T07:11:54.2606959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2607153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2607227Z return mod(**inputs) 2025-09-07T07:11:54.2607510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2607589Z outputs = self.mobilebert( 2025-09-07T07:11:54.2607866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2607945Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2608227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2608296Z layer_outputs = layer_module( 2025-09-07T07:11:54.2608578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2608674Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2608956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2609087Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2609363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2609453Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2609456Z 2025-09-07T07:11:54.2609556Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2609761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2609826Z return mod(**inputs) 2025-09-07T07:11:54.2610109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2610193Z outputs = self.mobilebert( 2025-09-07T07:11:54.2610475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2610558Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2610840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2610917Z layer_outputs = layer_module( 2025-09-07T07:11:54.2611194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2611291Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2611616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2611734Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2612044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2612188Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2612191Z 2025-09-07T07:11:54.2612301Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2612499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2612565Z return mod(**inputs) 2025-09-07T07:11:54.2612851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2612921Z outputs = self.mobilebert( 2025-09-07T07:11:54.2613215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2613286Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2613556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2613636Z layer_outputs = layer_module( 2025-09-07T07:11:54.2613904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2614000Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2614270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2614396Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2614662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2614744Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2614747Z 2025-09-07T07:11:54.2614854Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2615049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2615123Z return mod(**inputs) 2025-09-07T07:11:54.2615413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2615484Z outputs = self.mobilebert( 2025-09-07T07:11:54.2615769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2615841Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2616129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2616201Z layer_outputs = layer_module( 2025-09-07T07:11:54.2616512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2616608Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2616893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2617026Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2617308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2617437Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2617721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2617815Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2617826Z 2025-09-07T07:11:54.2617942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2618137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2618211Z return mod(**inputs) 2025-09-07T07:11:54.2618522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2618603Z outputs = self.mobilebert( 2025-09-07T07:11:54.2618878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2618949Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2619232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2619302Z layer_outputs = layer_module( 2025-09-07T07:11:54.2619713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2619852Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2620155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2620260Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2620264Z 2025-09-07T07:11:54.2620377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2620600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2620670Z return mod(**inputs) 2025-09-07T07:11:54.2620989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2621066Z outputs = self.mobilebert( 2025-09-07T07:11:54.2621374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2621463Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2621776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2621913Z layer_outputs = layer_module( 2025-09-07T07:11:54.2622197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2622318Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2622614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2622727Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2622730Z 2025-09-07T07:11:54.2622841Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2623039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2623134Z return mod(**inputs) 2025-09-07T07:11:54.2623420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2623495Z outputs = self.mobilebert( 2025-09-07T07:11:54.2623783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2623856Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2624144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2624218Z layer_outputs = layer_module( 2025-09-07T07:11:54.2624500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2624673Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2624957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2625119Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2625124Z 2025-09-07T07:11:54.2625227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2625435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2625501Z return mod(**inputs) 2025-09-07T07:11:54.2625831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2625916Z outputs = self.mobilebert( 2025-09-07T07:11:54.2626202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2626285Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2626569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2626643Z layer_outputs = layer_module( 2025-09-07T07:11:54.2626936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2627093Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2627384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2627507Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2627800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2627896Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2627899Z 2025-09-07T07:11:54.2628003Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2628210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2628280Z return mod(**inputs) 2025-09-07T07:11:54.2628593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2628666Z outputs = self.mobilebert( 2025-09-07T07:11:54.2628952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2629034Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2629319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2629401Z layer_outputs = layer_module( 2025-09-07T07:11:54.2629702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2629867Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2630231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2630359Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2630653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2630738Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2630741Z 2025-09-07T07:11:54.2630852Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2631053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2631132Z return mod(**inputs) 2025-09-07T07:11:54.2631423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2631493Z outputs = self.mobilebert( 2025-09-07T07:11:54.2631823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2631896Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2632188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2632260Z layer_outputs = layer_module( 2025-09-07T07:11:54.2632542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2632706Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2632989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2633119Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2633408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2633536Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2633824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2633919Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2633923Z 2025-09-07T07:11:54.2634034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2634233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2634306Z return mod(**inputs) 2025-09-07T07:11:54.2634590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2634662Z outputs = self.mobilebert( 2025-09-07T07:11:54.2634972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2635047Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2635335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2635407Z layer_outputs = layer_module( 2025-09-07T07:11:54.2635696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2635862Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2636149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2636286Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2636579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2636675Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2636679Z 2025-09-07T07:11:54.2636784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2636981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2637057Z return mod(**inputs) 2025-09-07T07:11:54.2637354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2637436Z outputs = self.mobilebert( 2025-09-07T07:11:54.2637721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2637811Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2638097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2638235Z layer_outputs = layer_module( 2025-09-07T07:11:54.2638515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2638600Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2638884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2638954Z self_outputs = self.self( 2025-09-07T07:11:54.2639228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2639305Z self.value(value_tensor) 2025-09-07T07:11:54.2639308Z 2025-09-07T07:11:54.2639410Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2669793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2670039Z return mod(**inputs) 2025-09-07T07:11:54.2670398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2670482Z outputs = self.mobilebert( 2025-09-07T07:11:54.2670786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2670869Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2671156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2671238Z layer_outputs = layer_module( 2025-09-07T07:11:54.2671526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2671705Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2672115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2672244Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2672529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2672615Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2672625Z 2025-09-07T07:11:54.2672751Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2672963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2673041Z return mod(**inputs) 2025-09-07T07:11:54.2673401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2673484Z outputs = self.mobilebert( 2025-09-07T07:11:54.2673774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2673850Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2674135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2674207Z layer_outputs = layer_module( 2025-09-07T07:11:54.2674489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2674651Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2674931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2675053Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2675388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2675478Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2675764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2675865Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2675869Z 2025-09-07T07:11:54.2675983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2676186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2676255Z return mod(**inputs) 2025-09-07T07:11:54.2676542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2676617Z outputs = self.mobilebert( 2025-09-07T07:11:54.2676907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2676982Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2677258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2677338Z layer_outputs = layer_module( 2025-09-07T07:11:54.2677616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2677711Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2677988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2678061Z self_outputs = self.self( 2025-09-07T07:11:54.2678348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2678443Z self.query(query_tensor) 2025-09-07T07:11:54.2678447Z 2025-09-07T07:11:54.2678564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2678765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2678840Z return mod(**inputs) 2025-09-07T07:11:54.2679119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2679194Z outputs = self.mobilebert( 2025-09-07T07:11:54.2679484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2679557Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2679916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2679993Z layer_outputs = layer_module( 2025-09-07T07:11:54.2680294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2680392Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2680687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2680766Z self_outputs = self.self( 2025-09-07T07:11:54.2681043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2681117Z self.key(key_tensor) 2025-09-07T07:11:54.2681120Z 2025-09-07T07:11:54.2681212Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2681292Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2681401Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2681593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2681691Z return mod(**inputs) 2025-09-07T07:11:54.2681965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2682035Z outputs = self.mobilebert( 2025-09-07T07:11:54.2682301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2682380Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2682645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2682719Z layer_outputs = layer_module( 2025-09-07T07:11:54.2682986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2683079Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2683347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2683466Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2683743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2683824Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2683828Z 2025-09-07T07:11:54.2683934Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2684126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2684196Z return mod(**inputs) 2025-09-07T07:11:54.2684462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2684534Z outputs = self.mobilebert( 2025-09-07T07:11:54.2684835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2684908Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2685189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2685259Z layer_outputs = layer_module( 2025-09-07T07:11:54.2685546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2685640Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2685919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2686064Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2686353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2686490Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2686785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2686879Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2686882Z 2025-09-07T07:11:54.2686993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2687195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2687269Z return mod(**inputs) 2025-09-07T07:11:54.2687555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2687627Z outputs = self.mobilebert( 2025-09-07T07:11:54.2687926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2688064Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2688348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2688418Z layer_outputs = layer_module( 2025-09-07T07:11:54.2688695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2688797Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2689071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2689193Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2689470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2689567Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2689570Z 2025-09-07T07:11:54.2689670Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2689864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2689936Z return mod(**inputs) 2025-09-07T07:11:54.2690214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2690293Z outputs = self.mobilebert( 2025-09-07T07:11:54.2690571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2690642Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2690929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2691005Z layer_outputs = layer_module( 2025-09-07T07:11:54.2691307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2691404Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2691687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2691796Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2692083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2692202Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2692205Z 2025-09-07T07:11:54.2692323Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2692522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2692590Z return mod(**inputs) 2025-09-07T07:11:54.2692854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2692928Z outputs = self.mobilebert( 2025-09-07T07:11:54.2693198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2693271Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2693542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2693615Z layer_outputs = layer_module( 2025-09-07T07:11:54.2693899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2693990Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2694306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2694433Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2694713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2694794Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2694798Z 2025-09-07T07:11:54.2694903Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2695108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2695170Z return mod(**inputs) 2025-09-07T07:11:54.2695443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2695512Z outputs = self.mobilebert( 2025-09-07T07:11:54.2695786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2695855Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2696120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2696194Z layer_outputs = layer_module( 2025-09-07T07:11:54.2696469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2696568Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2696842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2696968Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2697270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2697395Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2697678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2697769Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2697772Z 2025-09-07T07:11:54.2697880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2698074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2698135Z return mod(**inputs) 2025-09-07T07:11:54.2698430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2698500Z outputs = self.mobilebert( 2025-09-07T07:11:54.2698797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2698870Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2699147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2699224Z layer_outputs = layer_module( 2025-09-07T07:11:54.2699499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2699599Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2699878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2700003Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2700282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2700396Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2700400Z 2025-09-07T07:11:54.2700504Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2700702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2700771Z return mod(**inputs) 2025-09-07T07:11:54.2701052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2701130Z outputs = self.mobilebert( 2025-09-07T07:11:54.2701413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2701484Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2701774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2701852Z layer_outputs = layer_module( 2025-09-07T07:11:54.2702160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2702260Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2702558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2702685Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2702997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2703124Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2703127Z 2025-09-07T07:11:54.2703242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2703470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2703562Z return mod(**inputs) 2025-09-07T07:11:54.2703866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2703952Z outputs = self.mobilebert( 2025-09-07T07:11:54.2704267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2704354Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2704666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2704739Z layer_outputs = layer_module( 2025-09-07T07:11:54.2705061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2705163Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2705477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2705613Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2706029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2706129Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2706133Z 2025-09-07T07:11:54.2706243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2706469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2706538Z return mod(**inputs) 2025-09-07T07:11:54.2706865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2706944Z outputs = self.mobilebert( 2025-09-07T07:11:54.2707582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2707669Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2707984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2708062Z layer_outputs = layer_module( 2025-09-07T07:11:54.2708336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2708429Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2708717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2708840Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2709124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2709245Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2709529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2709622Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2709625Z 2025-09-07T07:11:54.2709728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2709942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2710010Z return mod(**inputs) 2025-09-07T07:11:54.2710302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2710375Z outputs = self.mobilebert( 2025-09-07T07:11:54.2710687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2710764Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2711048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2711125Z layer_outputs = layer_module( 2025-09-07T07:11:54.2711405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2711498Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2711781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2711911Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2712198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2712287Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2712290Z 2025-09-07T07:11:54.2712397Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2712598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2712670Z return mod(**inputs) 2025-09-07T07:11:54.2712955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2713027Z outputs = self.mobilebert( 2025-09-07T07:11:54.2713319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2713391Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2713679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2713781Z layer_outputs = layer_module( 2025-09-07T07:11:54.2714065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2714161Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2714449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2714566Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2714853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2714976Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2714979Z 2025-09-07T07:11:54.2715087Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2715289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2715367Z return mod(**inputs) 2025-09-07T07:11:54.2715654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2715733Z outputs = self.mobilebert( 2025-09-07T07:11:54.2716019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2716093Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2716391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2716462Z layer_outputs = layer_module( 2025-09-07T07:11:54.2716757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2716851Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2717165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2717291Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2717572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2717658Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2717662Z 2025-09-07T07:11:54.2717765Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2717971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2718039Z return mod(**inputs) 2025-09-07T07:11:54.2718346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2718426Z outputs = self.mobilebert( 2025-09-07T07:11:54.2718713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2718793Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2719076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2719148Z layer_outputs = layer_module( 2025-09-07T07:11:54.2719434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2719529Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2719970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2720101Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2720392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2720608Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2720894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2720996Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2721002Z 2025-09-07T07:11:54.2721106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2721314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2721380Z return mod(**inputs) 2025-09-07T07:11:54.2721670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2721742Z outputs = self.mobilebert( 2025-09-07T07:11:54.2722029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2722117Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2722399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2722481Z layer_outputs = layer_module( 2025-09-07T07:11:54.2722767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2722892Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2723182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2723266Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2723272Z 2025-09-07T07:11:54.2723384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2723612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2723688Z return mod(**inputs) 2025-09-07T07:11:54.2723971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2724042Z outputs = self.mobilebert( 2025-09-07T07:11:54.2724331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2724404Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2724699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2724780Z layer_outputs = layer_module( 2025-09-07T07:11:54.2725075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2725202Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2725474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2725591Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2725595Z 2025-09-07T07:11:54.2725692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2725888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2725951Z return mod(**inputs) 2025-09-07T07:11:54.2726223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2726301Z outputs = self.mobilebert( 2025-09-07T07:11:54.2726578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2726686Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2726951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2727016Z layer_outputs = layer_module( 2025-09-07T07:11:54.2727289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2727442Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2727714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2727805Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2727809Z 2025-09-07T07:11:54.2727908Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2728099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2728165Z return mod(**inputs) 2025-09-07T07:11:54.2728443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2728512Z outputs = self.mobilebert( 2025-09-07T07:11:54.2728787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2728857Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2729126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2729202Z layer_outputs = layer_module( 2025-09-07T07:11:54.2729473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2729634Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2729922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2730052Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2730321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2730408Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2730412Z 2025-09-07T07:11:54.2730517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2730706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2730775Z return mod(**inputs) 2025-09-07T07:11:54.2731058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2731127Z outputs = self.mobilebert( 2025-09-07T07:11:54.2731407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2731476Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2731754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2731823Z layer_outputs = layer_module( 2025-09-07T07:11:54.2732097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2732249Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2732521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2732643Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2732940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2733027Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2733030Z 2025-09-07T07:11:54.2733128Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2733318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2733381Z return mod(**inputs) 2025-09-07T07:11:54.2733649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2733726Z outputs = self.mobilebert( 2025-09-07T07:11:54.2733996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2734070Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2734337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2734406Z layer_outputs = layer_module( 2025-09-07T07:11:54.2734681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2734830Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2735106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2735221Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2735487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2735612Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2735894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2735993Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2735997Z 2025-09-07T07:11:54.2736095Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2736295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2736359Z return mod(**inputs) 2025-09-07T07:11:54.2736628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2736704Z outputs = self.mobilebert( 2025-09-07T07:11:54.2736976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2737067Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2737336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2737409Z layer_outputs = layer_module( 2025-09-07T07:11:54.2737681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2737835Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2738107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2738210Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2738484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2738567Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2738571Z 2025-09-07T07:11:54.2738668Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2738901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2738964Z return mod(**inputs) 2025-09-07T07:11:54.2739244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2739313Z outputs = self.mobilebert( 2025-09-07T07:11:54.2739582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2739660Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2739937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2740015Z layer_outputs = layer_module( 2025-09-07T07:11:54.2740294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2740390Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2740676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2740747Z self_outputs = self.self( 2025-09-07T07:11:54.2741034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2741104Z self.value(value_tensor) 2025-09-07T07:11:54.2741108Z 2025-09-07T07:11:54.2741216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2741412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2741475Z return mod(**inputs) 2025-09-07T07:11:54.2741764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2741838Z outputs = self.mobilebert( 2025-09-07T07:11:54.2742161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2742235Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2742522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2742592Z layer_outputs = layer_module( 2025-09-07T07:11:54.2742871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2743037Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2743336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2743454Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2743739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2743823Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2743830Z 2025-09-07T07:11:54.2743933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2744130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2744202Z return mod(**inputs) 2025-09-07T07:11:54.2744487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2744564Z outputs = self.mobilebert( 2025-09-07T07:11:54.2744853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2744926Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2745247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2745319Z layer_outputs = layer_module( 2025-09-07T07:11:54.2745605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2745843Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2746157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2746276Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2746587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2746694Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2746994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2747104Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2747109Z 2025-09-07T07:11:54.2747229Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2747423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2747495Z return mod(**inputs) 2025-09-07T07:11:54.2747769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2747847Z outputs = self.mobilebert( 2025-09-07T07:11:54.2748123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2748207Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2748486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2748579Z layer_outputs = layer_module( 2025-09-07T07:11:54.2748865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2748949Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2749233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2749303Z self_outputs = self.self( 2025-09-07T07:11:54.2749579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2749655Z self.query(query_tensor) 2025-09-07T07:11:54.2749658Z 2025-09-07T07:11:54.2749776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2749977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2750043Z return mod(**inputs) 2025-09-07T07:11:54.2750326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2750397Z outputs = self.mobilebert( 2025-09-07T07:11:54.2750670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2750745Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2751021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2751096Z layer_outputs = layer_module( 2025-09-07T07:11:54.2751374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2751458Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2751767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2751837Z self_outputs = self.self( 2025-09-07T07:11:54.2752123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2752189Z self.key(key_tensor) 2025-09-07T07:11:54.2752192Z 2025-09-07T07:11:54.2752273Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2752362Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2752462Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2752661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2752725Z return mod(**inputs) 2025-09-07T07:11:54.2753003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2753081Z outputs = self.mobilebert( 2025-09-07T07:11:54.2753356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2753434Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2753708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2753785Z layer_outputs = layer_module( 2025-09-07T07:11:54.2754061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2754145Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2754423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2754540Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2754843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2754924Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2754928Z 2025-09-07T07:11:54.2755024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2755217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2755278Z return mod(**inputs) 2025-09-07T07:11:54.2755558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2755627Z outputs = self.mobilebert( 2025-09-07T07:11:54.2755926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2756001Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2756287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2756369Z layer_outputs = layer_module( 2025-09-07T07:11:54.2756645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2756735Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2757013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2757131Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2757417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2757543Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2757823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2757948Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2757952Z 2025-09-07T07:11:54.2758056Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2758250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2758311Z return mod(**inputs) 2025-09-07T07:11:54.2758602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2758667Z outputs = self.mobilebert( 2025-09-07T07:11:54.2758944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2759014Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2759291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2759374Z layer_outputs = layer_module( 2025-09-07T07:11:54.2759651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2759754Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2760029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2760148Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2760424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2760503Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2760507Z 2025-09-07T07:11:54.2760613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2760806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2760894Z return mod(**inputs) 2025-09-07T07:11:54.2761176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2761243Z outputs = self.mobilebert( 2025-09-07T07:11:54.2761515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2761584Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2761856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2761922Z layer_outputs = layer_module( 2025-09-07T07:11:54.2762262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2762357Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2762634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2762751Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2763019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2763134Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2763137Z 2025-09-07T07:11:54.2763234Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2763422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2763486Z return mod(**inputs) 2025-09-07T07:11:54.2763753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2763865Z outputs = self.mobilebert( 2025-09-07T07:11:54.2764135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2764206Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2764472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2764541Z layer_outputs = layer_module( 2025-09-07T07:11:54.2764832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2764927Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2765237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2765373Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2765678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2765777Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2765781Z 2025-09-07T07:11:54.2765891Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2766113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2766183Z return mod(**inputs) 2025-09-07T07:11:54.2766491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2766561Z outputs = self.mobilebert( 2025-09-07T07:11:54.2766847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2766928Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2767229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2767311Z layer_outputs = layer_module( 2025-09-07T07:11:54.2767596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2767691Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2767979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2768099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2768373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2768526Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2768808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2768902Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2768906Z 2025-09-07T07:11:54.2769006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2769208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2769270Z return mod(**inputs) 2025-09-07T07:11:54.2769557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2769628Z outputs = self.mobilebert( 2025-09-07T07:11:54.2769907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2769986Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2770264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2770372Z layer_outputs = layer_module( 2025-09-07T07:11:54.2770654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2770752Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2771039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2771151Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2771454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2771532Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2771535Z 2025-09-07T07:11:54.2771643Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2771838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2771906Z return mod(**inputs) 2025-09-07T07:11:54.2772189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2772257Z outputs = self.mobilebert( 2025-09-07T07:11:54.2772542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2772613Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2772905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2772976Z layer_outputs = layer_module( 2025-09-07T07:11:54.2773260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2773362Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2773663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2773781Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2774068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2774177Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2774187Z 2025-09-07T07:11:54.2774289Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2774486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2774557Z return mod(**inputs) 2025-09-07T07:11:54.2774847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2774929Z outputs = self.mobilebert( 2025-09-07T07:11:54.2775207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2775280Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2775563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2775632Z layer_outputs = layer_module( 2025-09-07T07:11:54.2775915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2776009Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2776350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2776483Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2776788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2776880Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2776884Z 2025-09-07T07:11:54.2776984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2777187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2777250Z return mod(**inputs) 2025-09-07T07:11:54.2777522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2777592Z outputs = self.mobilebert( 2025-09-07T07:11:54.2777862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2777938Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2778213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2778285Z layer_outputs = layer_module( 2025-09-07T07:11:54.2778563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2778651Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2778930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2779051Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2779334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2779455Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2779732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2779847Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2779851Z 2025-09-07T07:11:54.2779955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2780163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2780232Z return mod(**inputs) 2025-09-07T07:11:54.2780523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2780599Z outputs = self.mobilebert( 2025-09-07T07:11:54.2780886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2780984Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2781272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2781357Z layer_outputs = layer_module( 2025-09-07T07:11:54.2781640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2781734Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2782027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2782138Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2782430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2782514Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2782517Z 2025-09-07T07:11:54.2782627Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2782827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2782927Z return mod(**inputs) 2025-09-07T07:11:54.2783216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2783288Z outputs = self.mobilebert( 2025-09-07T07:11:54.2783578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2783652Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2783939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2784020Z layer_outputs = layer_module( 2025-09-07T07:11:54.2784308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2784410Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2784703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2784815Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2785107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2785217Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2785221Z 2025-09-07T07:11:54.2785332Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2785541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2785618Z return mod(**inputs) 2025-09-07T07:11:54.2786041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2786126Z outputs = self.mobilebert( 2025-09-07T07:11:54.2786462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2786545Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2786858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2786934Z layer_outputs = layer_module( 2025-09-07T07:11:54.2787313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2787417Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2787722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2787882Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2788190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2788289Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2788293Z 2025-09-07T07:11:54.2788397Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2788597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2788673Z return mod(**inputs) 2025-09-07T07:11:54.2788955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2789034Z outputs = self.mobilebert( 2025-09-07T07:11:54.2789317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2789389Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2789678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2789790Z layer_outputs = layer_module( 2025-09-07T07:11:54.2790098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2790195Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2790517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2790650Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2790953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2791090Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2791398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2791509Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2791513Z 2025-09-07T07:11:54.2791621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2791844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2791914Z return mod(**inputs) 2025-09-07T07:11:54.2792220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2792305Z outputs = self.mobilebert( 2025-09-07T07:11:54.2792617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2792702Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2793009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2793089Z layer_outputs = layer_module( 2025-09-07T07:11:54.2793424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2793557Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2793867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2793956Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2793960Z 2025-09-07T07:11:54.2794075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2794292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2794362Z return mod(**inputs) 2025-09-07T07:11:54.2794701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2794780Z outputs = self.mobilebert( 2025-09-07T07:11:54.2795085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2795163Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2795464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2795547Z layer_outputs = layer_module( 2025-09-07T07:11:54.2795850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2795983Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2796283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2796402Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2796443Z 2025-09-07T07:11:54.2796555Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2796766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2796843Z return mod(**inputs) 2025-09-07T07:11:54.2797139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2797219Z outputs = self.mobilebert( 2025-09-07T07:11:54.2797517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2797595Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2797901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2797975Z layer_outputs = layer_module( 2025-09-07T07:11:54.2798281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2798452Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2798753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2798857Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2798861Z 2025-09-07T07:11:54.2798967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2799201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2799271Z return mod(**inputs) 2025-09-07T07:11:54.2799572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2799654Z outputs = self.mobilebert( 2025-09-07T07:11:54.2799977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2800062Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2800360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2800435Z layer_outputs = layer_module( 2025-09-07T07:11:54.2800742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2800910Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2801233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2801366Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2801678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2801775Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2801779Z 2025-09-07T07:11:54.2801886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2802108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2802177Z return mod(**inputs) 2025-09-07T07:11:54.2802484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2802558Z outputs = self.mobilebert( 2025-09-07T07:11:54.2802859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2802943Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2803247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2803360Z layer_outputs = layer_module( 2025-09-07T07:11:54.2803658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2803837Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2804142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2804277Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2804590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2804686Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2804690Z 2025-09-07T07:11:54.2804808Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2805030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2805102Z return mod(**inputs) 2025-09-07T07:11:54.2805415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2805493Z outputs = self.mobilebert( 2025-09-07T07:11:54.2805813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2805892Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2806278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2806354Z layer_outputs = layer_module( 2025-09-07T07:11:54.2806650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2806850Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2807147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2807284Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2807581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2807719Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2808017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2808133Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2808137Z 2025-09-07T07:11:54.2808256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2808474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2808549Z return mod(**inputs) 2025-09-07T07:11:54.2808851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2808925Z outputs = self.mobilebert( 2025-09-07T07:11:54.2809237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2809316Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2809638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2809715Z layer_outputs = layer_module( 2025-09-07T07:11:54.2810037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2810259Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2810566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2810693Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2811012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2811109Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2811113Z 2025-09-07T07:11:54.2811224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2811440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2811522Z return mod(**inputs) 2025-09-07T07:11:54.2811829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2811917Z outputs = self.mobilebert( 2025-09-07T07:11:54.2812227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2812311Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2812629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2812707Z layer_outputs = layer_module( 2025-09-07T07:11:54.2813026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2813120Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2813438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2813515Z self_outputs = self.self( 2025-09-07T07:11:54.2813841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2813929Z self.value(value_tensor) 2025-09-07T07:11:54.2813932Z 2025-09-07T07:11:54.2814044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2814268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2814340Z return mod(**inputs) 2025-09-07T07:11:54.2814648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2814732Z outputs = self.mobilebert( 2025-09-07T07:11:54.2815080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2815170Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2815483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2815569Z layer_outputs = layer_module( 2025-09-07T07:11:54.2815878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2816055Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2816379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2816499Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2816819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2816912Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2816916Z 2025-09-07T07:11:54.2817070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2817287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2817360Z return mod(**inputs) 2025-09-07T07:11:54.2817674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2817751Z outputs = self.mobilebert( 2025-09-07T07:11:54.2818065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2818144Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2818455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2818542Z layer_outputs = layer_module( 2025-09-07T07:11:54.2818850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2819035Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2819346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2819473Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2819981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2820083Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2820408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2820513Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2820516Z 2025-09-07T07:11:54.2820638Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2820912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2820986Z return mod(**inputs) 2025-09-07T07:11:54.2821304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2821381Z outputs = self.mobilebert( 2025-09-07T07:11:54.2821695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2821774Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2822094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2822172Z layer_outputs = layer_module( 2025-09-07T07:11:54.2822506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2822614Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2822924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2823011Z self_outputs = self.self( 2025-09-07T07:11:54.2823325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2823404Z self.query(query_tensor) 2025-09-07T07:11:54.2823408Z 2025-09-07T07:11:54.2823528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2823745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2823824Z return mod(**inputs) 2025-09-07T07:11:54.2824135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2824214Z outputs = self.mobilebert( 2025-09-07T07:11:54.2824586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2824665Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2824977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2825054Z layer_outputs = layer_module( 2025-09-07T07:11:54.2825368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2825458Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2825818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2825917Z self_outputs = self.self( 2025-09-07T07:11:54.2826226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2826310Z self.key(key_tensor) 2025-09-07T07:11:54.2826314Z 2025-09-07T07:11:54.2826406Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2826494Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2826614Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2826828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2826908Z return mod(**inputs) 2025-09-07T07:11:54.2827215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2827293Z outputs = self.mobilebert( 2025-09-07T07:11:54.2827622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2827702Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2828046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2828126Z layer_outputs = layer_module( 2025-09-07T07:11:54.2828447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2828539Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2828848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2828993Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2829312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2829431Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2829435Z 2025-09-07T07:11:54.2829552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2829772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2829852Z return mod(**inputs) 2025-09-07T07:11:54.2830163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2830250Z outputs = self.mobilebert( 2025-09-07T07:11:54.2830565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2830647Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2830932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2831004Z layer_outputs = layer_module( 2025-09-07T07:11:54.2831309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2831427Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2831711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2831831Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2832105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2832237Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2832512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2832611Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2832616Z 2025-09-07T07:11:54.2832717Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2832930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2832993Z return mod(**inputs) 2025-09-07T07:11:54.2833260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2833339Z outputs = self.mobilebert( 2025-09-07T07:11:54.2833607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2833682Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2833951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2834020Z layer_outputs = layer_module( 2025-09-07T07:11:54.2834303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2834395Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2834695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2834808Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2835091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2835173Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2835177Z 2025-09-07T07:11:54.2835276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2835481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2835544Z return mod(**inputs) 2025-09-07T07:11:54.2835840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2835914Z outputs = self.mobilebert( 2025-09-07T07:11:54.2836193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2836272Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2836548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2836635Z layer_outputs = layer_module( 2025-09-07T07:11:54.2836903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2836998Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2837270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2837379Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2837682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2837788Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2837792Z 2025-09-07T07:11:54.2837896Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2838084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2838147Z return mod(**inputs) 2025-09-07T07:11:54.2838420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2838487Z outputs = self.mobilebert( 2025-09-07T07:11:54.2838762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2838831Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2839108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2839175Z layer_outputs = layer_module( 2025-09-07T07:11:54.2839488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2839587Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2839862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2839991Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2840265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2840350Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2840360Z 2025-09-07T07:11:54.2840459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2840681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2840755Z return mod(**inputs) 2025-09-07T07:11:54.2841035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2841110Z outputs = self.mobilebert( 2025-09-07T07:11:54.2841397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2841466Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2841750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2841817Z layer_outputs = layer_module( 2025-09-07T07:11:54.2842107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2842202Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2842469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2842596Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2842864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2842986Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2843257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2843353Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2843358Z 2025-09-07T07:11:54.2843455Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2843679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2843749Z return mod(**inputs) 2025-09-07T07:11:54.2844018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2844092Z outputs = self.mobilebert( 2025-09-07T07:11:54.2844363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2844431Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2844705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2844773Z layer_outputs = layer_module( 2025-09-07T07:11:54.2845047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2845139Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2845414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2845522Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2845795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2845882Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2845885Z 2025-09-07T07:11:54.2845983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2846182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2846244Z return mod(**inputs) 2025-09-07T07:11:54.2846515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2846593Z outputs = self.mobilebert( 2025-09-07T07:11:54.2846887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2846970Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2847244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2847320Z layer_outputs = layer_module( 2025-09-07T07:11:54.2847597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2847690Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2847989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2848100Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2848387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2848494Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2848497Z 2025-09-07T07:11:54.2848606Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2848801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2848864Z return mod(**inputs) 2025-09-07T07:11:54.2849147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2849217Z outputs = self.mobilebert( 2025-09-07T07:11:54.2849501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2849572Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2849849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2849958Z layer_outputs = layer_module( 2025-09-07T07:11:54.2850236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2850335Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2850612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2850735Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2851023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2851110Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2851114Z 2025-09-07T07:11:54.2851223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2851429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2851504Z return mod(**inputs) 2025-09-07T07:11:54.2851789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2851860Z outputs = self.mobilebert( 2025-09-07T07:11:54.2852152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2852236Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2852521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2852590Z layer_outputs = layer_module( 2025-09-07T07:11:54.2852868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2852988Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2853265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2853395Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2853670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2853801Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2854077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2854168Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2854186Z 2025-09-07T07:11:54.2854295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2854493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2854566Z return mod(**inputs) 2025-09-07T07:11:54.2854840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2854909Z outputs = self.mobilebert( 2025-09-07T07:11:54.2855191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2855261Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2855544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2855614Z layer_outputs = layer_module( 2025-09-07T07:11:54.2855898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2856020Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2856307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2856426Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2856722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2856819Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2856822Z 2025-09-07T07:11:54.2856932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2857150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2857219Z return mod(**inputs) 2025-09-07T07:11:54.2857520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2857604Z outputs = self.mobilebert( 2025-09-07T07:11:54.2857904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2857988Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2858288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2858360Z layer_outputs = layer_module( 2025-09-07T07:11:54.2858650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2858755Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2859038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2859146Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2859445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2859562Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2859565Z 2025-09-07T07:11:54.2859665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2859869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2859938Z return mod(**inputs) 2025-09-07T07:11:54.2860241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2860316Z outputs = self.mobilebert( 2025-09-07T07:11:54.2860632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2860719Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2861024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2861110Z layer_outputs = layer_module( 2025-09-07T07:11:54.2861410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2861510Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2861820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2861956Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2862268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2862360Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2862363Z 2025-09-07T07:11:54.2862481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2862738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2862809Z return mod(**inputs) 2025-09-07T07:11:54.2863115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2863190Z outputs = self.mobilebert( 2025-09-07T07:11:54.2863497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2863574Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2863874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2863958Z layer_outputs = layer_module( 2025-09-07T07:11:54.2864261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2864374Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2864674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2864813Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2865114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2865244Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2865551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2865649Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2865652Z 2025-09-07T07:11:54.2865842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2866064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2866227Z return mod(**inputs) 2025-09-07T07:11:54.2866535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2866614Z outputs = self.mobilebert( 2025-09-07T07:11:54.2866943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2867021Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2867331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2867409Z layer_outputs = layer_module( 2025-09-07T07:11:54.2867726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2867868Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2868168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2868264Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2868268Z 2025-09-07T07:11:54.2868378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2868597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2868666Z return mod(**inputs) 2025-09-07T07:11:54.2868966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2869049Z outputs = self.mobilebert( 2025-09-07T07:11:54.2869351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2869436Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2869773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2869849Z layer_outputs = layer_module( 2025-09-07T07:11:54.2870155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2870283Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2870595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2870713Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2870717Z 2025-09-07T07:11:54.2870832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2871049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2871122Z return mod(**inputs) 2025-09-07T07:11:54.2871427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2871502Z outputs = self.mobilebert( 2025-09-07T07:11:54.2871808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2871885Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2872191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2872275Z layer_outputs = layer_module( 2025-09-07T07:11:54.2872584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2872765Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2873078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2873192Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2873196Z 2025-09-07T07:11:54.2873304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2873518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2873592Z return mod(**inputs) 2025-09-07T07:11:54.2873892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2873981Z outputs = self.mobilebert( 2025-09-07T07:11:54.2874256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2874341Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2874624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2874697Z layer_outputs = layer_module( 2025-09-07T07:11:54.2874978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2875134Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2875419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2875539Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2875813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2875914Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2875917Z 2025-09-07T07:11:54.2876018Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2876255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2876320Z return mod(**inputs) 2025-09-07T07:11:54.2876594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2876670Z outputs = self.mobilebert( 2025-09-07T07:11:54.2876944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2877023Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2877308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2877385Z layer_outputs = layer_module( 2025-09-07T07:11:54.2877676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2877835Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2878116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2878240Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2878535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2878622Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2878625Z 2025-09-07T07:11:54.2878728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2878947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2879013Z return mod(**inputs) 2025-09-07T07:11:54.2879294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2879382Z outputs = self.mobilebert( 2025-09-07T07:11:54.2879666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2879737Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2880013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2880093Z layer_outputs = layer_module( 2025-09-07T07:11:54.2880378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2880543Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2880840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2880971Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2881262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2881385Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2881678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2881772Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2881775Z 2025-09-07T07:11:54.2881886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2882085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2882155Z return mod(**inputs) 2025-09-07T07:11:54.2882444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2882546Z outputs = self.mobilebert( 2025-09-07T07:11:54.2882834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2882906Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2883198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2883279Z layer_outputs = layer_module( 2025-09-07T07:11:54.2883577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2883758Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2884059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2884186Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2884484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2884569Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2884573Z 2025-09-07T07:11:54.2884683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2884881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2884955Z return mod(**inputs) 2025-09-07T07:11:54.2885235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2885306Z outputs = self.mobilebert( 2025-09-07T07:11:54.2885595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2885672Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2885980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2886052Z layer_outputs = layer_module( 2025-09-07T07:11:54.2886340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2886426Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2886707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2886786Z self_outputs = self.self( 2025-09-07T07:11:54.2887086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2887166Z self.value(value_tensor) 2025-09-07T07:11:54.2887169Z 2025-09-07T07:11:54.2887277Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2887485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2887559Z return mod(**inputs) 2025-09-07T07:11:54.2887842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2887920Z outputs = self.mobilebert( 2025-09-07T07:11:54.2888206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2888286Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2888566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2888640Z layer_outputs = layer_module( 2025-09-07T07:11:54.2888933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2889137Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2889430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2889542Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2889841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2889936Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2889940Z 2025-09-07T07:11:54.2890048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2890269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2890338Z return mod(**inputs) 2025-09-07T07:11:54.2890642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2890720Z outputs = self.mobilebert( 2025-09-07T07:11:54.2891019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2891101Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2891409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2891492Z layer_outputs = layer_module( 2025-09-07T07:11:54.2891799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2891969Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2892278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2892415Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2892723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2892816Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2893121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2893220Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2893223Z 2025-09-07T07:11:54.2893330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2893551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2893639Z return mod(**inputs) 2025-09-07T07:11:54.2893949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2894030Z outputs = self.mobilebert( 2025-09-07T07:11:54.2894326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2894411Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2894718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2894801Z layer_outputs = layer_module( 2025-09-07T07:11:54.2895101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2895198Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2895497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2895572Z self_outputs = self.self( 2025-09-07T07:11:54.2895908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2895982Z self.query(query_tensor) 2025-09-07T07:11:54.2895986Z 2025-09-07T07:11:54.2896102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2896310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2896380Z return mod(**inputs) 2025-09-07T07:11:54.2896684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2896760Z outputs = self.mobilebert( 2025-09-07T07:11:54.2897069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2897146Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2897453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2897532Z layer_outputs = layer_module( 2025-09-07T07:11:54.2897830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2897929Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2898226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2898308Z self_outputs = self.self( 2025-09-07T07:11:54.2898602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2898673Z self.key(key_tensor) 2025-09-07T07:11:54.2898677Z 2025-09-07T07:11:54.2898773Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2898859Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2898978Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2899199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2899271Z return mod(**inputs) 2025-09-07T07:11:54.2899587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2899663Z outputs = self.mobilebert( 2025-09-07T07:11:54.2899974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2900049Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2900363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2900457Z layer_outputs = layer_module( 2025-09-07T07:11:54.2900759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2900861Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2901173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2901309Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2901620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2901712Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2901723Z 2025-09-07T07:11:54.2901832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2902056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2902134Z return mod(**inputs) 2025-09-07T07:11:54.2902445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2902569Z outputs = self.mobilebert( 2025-09-07T07:11:54.2902851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2902925Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2903215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2903286Z layer_outputs = layer_module( 2025-09-07T07:11:54.2903576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2903660Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2903944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2904078Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2904363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2904496Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2904781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2904881Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2904885Z 2025-09-07T07:11:54.2904993Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2905217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2905294Z return mod(**inputs) 2025-09-07T07:11:54.2905604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2905690Z outputs = self.mobilebert( 2025-09-07T07:11:54.2906082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2906167Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2906487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2906562Z layer_outputs = layer_module( 2025-09-07T07:11:54.2906880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2906983Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2907374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2907499Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2907818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2907917Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2907921Z 2025-09-07T07:11:54.2908030Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2908260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2908329Z return mod(**inputs) 2025-09-07T07:11:54.2908637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2908723Z outputs = self.mobilebert( 2025-09-07T07:11:54.2909034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2909119Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2909467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2909551Z layer_outputs = layer_module( 2025-09-07T07:11:54.2909860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2909960Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2910267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2910385Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2910691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2910811Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2910815Z 2025-09-07T07:11:54.2910927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2911148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2911218Z return mod(**inputs) 2025-09-07T07:11:54.2911523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2911600Z outputs = self.mobilebert( 2025-09-07T07:11:54.2911901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2911978Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2912287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2912373Z layer_outputs = layer_module( 2025-09-07T07:11:54.2912669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2912796Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2913100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2913236Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2913545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2913640Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2913643Z 2025-09-07T07:11:54.2913760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2913970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2914076Z return mod(**inputs) 2025-09-07T07:11:54.2914375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2914456Z outputs = self.mobilebert( 2025-09-07T07:11:54.2914766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2914843Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2915159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2915233Z layer_outputs = layer_module( 2025-09-07T07:11:54.2915545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2915650Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2915959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2916155Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2916461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2916598Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2916902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2917001Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2917004Z 2025-09-07T07:11:54.2917122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2917336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2917412Z return mod(**inputs) 2025-09-07T07:11:54.2917725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2917803Z outputs = self.mobilebert( 2025-09-07T07:11:54.2918115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2918192Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2918503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2918578Z layer_outputs = layer_module( 2025-09-07T07:11:54.2918894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2918994Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2919308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2919436Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2919945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2920052Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2920056Z 2025-09-07T07:11:54.2920167Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2920383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2920464Z return mod(**inputs) 2025-09-07T07:11:54.2920767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2920850Z outputs = self.mobilebert( 2025-09-07T07:11:54.2921176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2921263Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2921569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2921645Z layer_outputs = layer_module( 2025-09-07T07:11:54.2921952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2922051Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2922359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2922479Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2922782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2923035Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2923039Z 2025-09-07T07:11:54.2923195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2923419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2923489Z return mod(**inputs) 2025-09-07T07:11:54.2923796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2923872Z outputs = self.mobilebert( 2025-09-07T07:11:54.2924171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2924260Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2924562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2924646Z layer_outputs = layer_module( 2025-09-07T07:11:54.2924950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2925054Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2925368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2925503Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2925810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2925900Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2925904Z 2025-09-07T07:11:54.2926020Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2926232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2926303Z return mod(**inputs) 2025-09-07T07:11:54.2926612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2926708Z outputs = self.mobilebert( 2025-09-07T07:11:54.2927015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2927091Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2927389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2927468Z layer_outputs = layer_module( 2025-09-07T07:11:54.2927766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2927872Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2928187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2928340Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2928623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2928745Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2929037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2929131Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2929134Z 2025-09-07T07:11:54.2929243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2929446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2929514Z return mod(**inputs) 2025-09-07T07:11:54.2929811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2929912Z outputs = self.mobilebert( 2025-09-07T07:11:54.2930204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2930278Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2930571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2930643Z layer_outputs = layer_module( 2025-09-07T07:11:54.2930924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2931026Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2931314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2931435Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2931721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2931806Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2931816Z 2025-09-07T07:11:54.2931919Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2932118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2932193Z return mod(**inputs) 2025-09-07T07:11:54.2932476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2932555Z outputs = self.mobilebert( 2025-09-07T07:11:54.2932839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2932911Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2933223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2933294Z layer_outputs = layer_module( 2025-09-07T07:11:54.2933590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2933684Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2933968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2934093Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2934374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2934507Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2934511Z 2025-09-07T07:11:54.2934617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2934830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2934897Z return mod(**inputs) 2025-09-07T07:11:54.2935183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2935265Z outputs = self.mobilebert( 2025-09-07T07:11:54.2935553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2935632Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2935925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2935999Z layer_outputs = layer_module( 2025-09-07T07:11:54.2936292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2936420Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2936712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2936839Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2937130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2937214Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2937217Z 2025-09-07T07:11:54.2937319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2937526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2937593Z return mod(**inputs) 2025-09-07T07:11:54.2937881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2937955Z outputs = self.mobilebert( 2025-09-07T07:11:54.2938236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2938316Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2938597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2938676Z layer_outputs = layer_module( 2025-09-07T07:11:54.2938956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2939056Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2939341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2939465Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2939796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2939927Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2940234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2940332Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2940336Z 2025-09-07T07:11:54.2940449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2940661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2940730Z return mod(**inputs) 2025-09-07T07:11:54.2941055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2941134Z outputs = self.mobilebert( 2025-09-07T07:11:54.2941466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2941542Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2941851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2941936Z layer_outputs = layer_module( 2025-09-07T07:11:54.2942242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2942377Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2942688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2942783Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2942818Z 2025-09-07T07:11:54.2942929Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2943137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2943213Z return mod(**inputs) 2025-09-07T07:11:54.2943521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2943600Z outputs = self.mobilebert( 2025-09-07T07:11:54.2943908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2943982Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2944300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2944376Z layer_outputs = layer_module( 2025-09-07T07:11:54.2944689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.2944821Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.2945159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2945276Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2945280Z 2025-09-07T07:11:54.2945387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2945602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2945671Z return mod(**inputs) 2025-09-07T07:11:54.2946200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2946282Z outputs = self.mobilebert( 2025-09-07T07:11:54.2946621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2946717Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2947044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2947136Z layer_outputs = layer_module( 2025-09-07T07:11:54.2947463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2947640Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2947962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.2948084Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.2948089Z 2025-09-07T07:11:54.2948210Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2948428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2948506Z return mod(**inputs) 2025-09-07T07:11:54.2948820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2948896Z outputs = self.mobilebert( 2025-09-07T07:11:54.2949214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2949292Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2949611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2949686Z layer_outputs = layer_module( 2025-09-07T07:11:54.2949996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2950209Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2950520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.2950660Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.2950961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2951065Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2951069Z 2025-09-07T07:11:54.2951178Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2951390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2951469Z return mod(**inputs) 2025-09-07T07:11:54.2951767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2951853Z outputs = self.mobilebert( 2025-09-07T07:11:54.2952156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2952241Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2952541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2952618Z layer_outputs = layer_module( 2025-09-07T07:11:54.2952926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2953093Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2953407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2953559Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2953860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.2953958Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2953962Z 2025-09-07T07:11:54.2954069Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2954289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2954360Z return mod(**inputs) 2025-09-07T07:11:54.2954665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2954741Z outputs = self.mobilebert( 2025-09-07T07:11:54.2955062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2955144Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2955419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2955496Z layer_outputs = layer_module( 2025-09-07T07:11:54.2955769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.2955925Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.2956229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.2956352Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.2956635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.2956784Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2957068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2957158Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2957161Z 2025-09-07T07:11:54.2957262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2957465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2957530Z return mod(**inputs) 2025-09-07T07:11:54.2957811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2957881Z outputs = self.mobilebert( 2025-09-07T07:11:54.2958157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2958240Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2958515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2958591Z layer_outputs = layer_module( 2025-09-07T07:11:54.2958862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2959027Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2959306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2959417Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2959711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2959795Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2959801Z 2025-09-07T07:11:54.2959924Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2960125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2960193Z return mod(**inputs) 2025-09-07T07:11:54.2960481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2960550Z outputs = self.mobilebert( 2025-09-07T07:11:54.2960842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2960911Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2962067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2962146Z layer_outputs = layer_module( 2025-09-07T07:11:54.2962432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2962528Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2962806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2962885Z self_outputs = self.self( 2025-09-07T07:11:54.2963163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.2963233Z self.value(value_tensor) 2025-09-07T07:11:54.2963243Z 2025-09-07T07:11:54.2963343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2963539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2963611Z return mod(**inputs) 2025-09-07T07:11:54.2963892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2964014Z outputs = self.mobilebert( 2025-09-07T07:11:54.2964289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2964361Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2964644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2964714Z layer_outputs = layer_module( 2025-09-07T07:11:54.2964995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2965156Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2965431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.2965552Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.2965827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.2965918Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.2965922Z 2025-09-07T07:11:54.2966022Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2966220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2966283Z return mod(**inputs) 2025-09-07T07:11:54.2966553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2966631Z outputs = self.mobilebert( 2025-09-07T07:11:54.2966906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2966989Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2967284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2967355Z layer_outputs = layer_module( 2025-09-07T07:11:54.2967641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.2967800Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.2968085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.2968193Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.2968504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.2968594Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.2968871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2968979Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2968982Z 2025-09-07T07:11:54.2969081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2969278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2969341Z return mod(**inputs) 2025-09-07T07:11:54.2969612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2969689Z outputs = self.mobilebert( 2025-09-07T07:11:54.2969972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2970107Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2970397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2970476Z layer_outputs = layer_module( 2025-09-07T07:11:54.2970758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2970845Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2971134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2971206Z self_outputs = self.self( 2025-09-07T07:11:54.2971507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.2971577Z self.query(query_tensor) 2025-09-07T07:11:54.2971580Z 2025-09-07T07:11:54.2971681Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2971885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2971951Z return mod(**inputs) 2025-09-07T07:11:54.2972237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2972305Z outputs = self.mobilebert( 2025-09-07T07:11:54.2972586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2972657Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2972933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2973013Z layer_outputs = layer_module( 2025-09-07T07:11:54.2973290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2973394Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2973665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.2973733Z self_outputs = self.self( 2025-09-07T07:11:54.2974007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.2974072Z self.key(key_tensor) 2025-09-07T07:11:54.2974076Z 2025-09-07T07:11:54.2974162Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2974238Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.2974337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2974557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2974623Z return mod(**inputs) 2025-09-07T07:11:54.2974908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2974978Z outputs = self.mobilebert( 2025-09-07T07:11:54.2975263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2975333Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2975608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2975684Z layer_outputs = layer_module( 2025-09-07T07:11:54.2975963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2976050Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2976328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2976481Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2976766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.2976850Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2976854Z 2025-09-07T07:11:54.2976960Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2977152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2977222Z return mod(**inputs) 2025-09-07T07:11:54.2977499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2977567Z outputs = self.mobilebert( 2025-09-07T07:11:54.2977850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2977922Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2978207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2978277Z layer_outputs = layer_module( 2025-09-07T07:11:54.2978551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.2978641Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.2978917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.2979046Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.2979328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.2979464Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2979762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2979858Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2979862Z 2025-09-07T07:11:54.2979969Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2980170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2980244Z return mod(**inputs) 2025-09-07T07:11:54.2980531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2980605Z outputs = self.mobilebert( 2025-09-07T07:11:54.2980916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2980992Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2981289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2981361Z layer_outputs = layer_module( 2025-09-07T07:11:54.2981658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2981752Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2982029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2982149Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2982435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2982528Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2982559Z 2025-09-07T07:11:54.2982663Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2982865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2982941Z return mod(**inputs) 2025-09-07T07:11:54.2983225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2983303Z outputs = self.mobilebert( 2025-09-07T07:11:54.2983590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2983669Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2984001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2984079Z layer_outputs = layer_module( 2025-09-07T07:11:54.2984385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2984490Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2984809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2984929Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2985278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2985406Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2985410Z 2025-09-07T07:11:54.2985517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2985819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2985898Z return mod(**inputs) 2025-09-07T07:11:54.2986240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2986321Z outputs = self.mobilebert( 2025-09-07T07:11:54.2986631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2986716Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2987024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2987108Z layer_outputs = layer_module( 2025-09-07T07:11:54.2987416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2987517Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2987850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2987995Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2988313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.2988403Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.2988407Z 2025-09-07T07:11:54.2988526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2988740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2988812Z return mod(**inputs) 2025-09-07T07:11:54.2989129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2989206Z outputs = self.mobilebert( 2025-09-07T07:11:54.2989531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2989651Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2989967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2990050Z layer_outputs = layer_module( 2025-09-07T07:11:54.2990362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2990470Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2990783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2990925Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.2991226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.2991356Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.2991681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.2991778Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.2991783Z 2025-09-07T07:11:54.2991897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2992109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2992178Z return mod(**inputs) 2025-09-07T07:11:54.2992495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2992570Z outputs = self.mobilebert( 2025-09-07T07:11:54.2992881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2992962Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2993309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2993386Z layer_outputs = layer_module( 2025-09-07T07:11:54.2993685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2993790Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2994099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2994224Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2994539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.2994631Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.2994642Z 2025-09-07T07:11:54.2994754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2994972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2995050Z return mod(**inputs) 2025-09-07T07:11:54.2995353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2995437Z outputs = self.mobilebert( 2025-09-07T07:11:54.2995736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2995812Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2996121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2996198Z layer_outputs = layer_module( 2025-09-07T07:11:54.2996506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2996640Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2996936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.2997062Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.2997371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.2997499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.2997503Z 2025-09-07T07:11:54.2997609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.2997831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.2997901Z return mod(**inputs) 2025-09-07T07:11:54.2998201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.2998298Z outputs = self.mobilebert( 2025-09-07T07:11:54.2998580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.2998659Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.2998944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.2999014Z layer_outputs = layer_module( 2025-09-07T07:11:54.2999307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.2999401Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.2999709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.2999846Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3000168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3000260Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3000263Z 2025-09-07T07:11:54.3000374Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3000592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3000662Z return mod(**inputs) 2025-09-07T07:11:54.3000972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3001047Z outputs = self.mobilebert( 2025-09-07T07:11:54.3001364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3001460Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3001757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3001836Z layer_outputs = layer_module( 2025-09-07T07:11:54.3002117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3002218Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3002501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3002627Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3002923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3003047Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3003363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3003456Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3003459Z 2025-09-07T07:11:54.3003568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3003765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3003831Z return mod(**inputs) 2025-09-07T07:11:54.3004121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3004192Z outputs = self.mobilebert( 2025-09-07T07:11:54.3004482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3004555Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3004839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3004918Z layer_outputs = layer_module( 2025-09-07T07:11:54.3005199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3005299Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3005580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3005699Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3005979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3006066Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3006069Z 2025-09-07T07:11:54.3006182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3006392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3006467Z return mod(**inputs) 2025-09-07T07:11:54.3006751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3006820Z outputs = self.mobilebert( 2025-09-07T07:11:54.3007108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3007180Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3007471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3007558Z layer_outputs = layer_module( 2025-09-07T07:11:54.3007841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3007947Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3008233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3008350Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3008633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3008754Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3008758Z 2025-09-07T07:11:54.3008860Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3009058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3009132Z return mod(**inputs) 2025-09-07T07:11:54.3009414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3009524Z outputs = self.mobilebert( 2025-09-07T07:11:54.3009811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3009884Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3010180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3010252Z layer_outputs = layer_module( 2025-09-07T07:11:54.3010544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3010636Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3010933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3011060Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3011348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3011440Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3011443Z 2025-09-07T07:11:54.3011544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3011750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3011817Z return mod(**inputs) 2025-09-07T07:11:54.3012111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3012186Z outputs = self.mobilebert( 2025-09-07T07:11:54.3012482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3012572Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3012897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3012984Z layer_outputs = layer_module( 2025-09-07T07:11:54.3013280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3013375Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3013667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3013794Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3014108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3014233Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3014537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3014635Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3014638Z 2025-09-07T07:11:54.3014749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3014972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3015042Z return mod(**inputs) 2025-09-07T07:11:54.3015351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3015426Z outputs = self.mobilebert( 2025-09-07T07:11:54.3015732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3015819Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3016151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3016235Z layer_outputs = layer_module( 2025-09-07T07:11:54.3016539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3016667Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3016947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3017031Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3017034Z 2025-09-07T07:11:54.3017144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3017344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3017420Z return mod(**inputs) 2025-09-07T07:11:54.3017714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3017789Z outputs = self.mobilebert( 2025-09-07T07:11:54.3018095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3018171Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3018485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3018560Z layer_outputs = layer_module( 2025-09-07T07:11:54.3018859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3018995Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3019294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3019451Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3019455Z 2025-09-07T07:11:54.3019753Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3019984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3020055Z return mod(**inputs) 2025-09-07T07:11:54.3020355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3020440Z outputs = self.mobilebert( 2025-09-07T07:11:54.3020739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3020868Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3021183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3021266Z layer_outputs = layer_module( 2025-09-07T07:11:54.3021581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3021760Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3022076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3022193Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3022198Z 2025-09-07T07:11:54.3022316Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3022528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3022602Z return mod(**inputs) 2025-09-07T07:11:54.3022909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3023036Z outputs = self.mobilebert( 2025-09-07T07:11:54.3023353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3023431Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3023757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3023832Z layer_outputs = layer_module( 2025-09-07T07:11:54.3024141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3024318Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3024627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3024771Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3025076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3025173Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3025185Z 2025-09-07T07:11:54.3025295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3025509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3025586Z return mod(**inputs) 2025-09-07T07:11:54.3025944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3026030Z outputs = self.mobilebert( 2025-09-07T07:11:54.3026336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3026417Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3026753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3026831Z layer_outputs = layer_module( 2025-09-07T07:11:54.3027137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3027306Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3027606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3027749Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3028069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3028173Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3028177Z 2025-09-07T07:11:54.3028287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3028506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3028578Z return mod(**inputs) 2025-09-07T07:11:54.3028876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3028958Z outputs = self.mobilebert( 2025-09-07T07:11:54.3029255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3029340Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3029642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3029719Z layer_outputs = layer_module( 2025-09-07T07:11:54.3030061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3030232Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3030525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3030649Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3030946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3031066Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3031342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3031441Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3031447Z 2025-09-07T07:11:54.3031549Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3031752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3031818Z return mod(**inputs) 2025-09-07T07:11:54.3032090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3032169Z outputs = self.mobilebert( 2025-09-07T07:11:54.3032442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3032522Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3032798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3032875Z layer_outputs = layer_module( 2025-09-07T07:11:54.3033173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3033334Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3033618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3033726Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3034008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3034090Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3034094Z 2025-09-07T07:11:54.3034201Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3034411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3034485Z return mod(**inputs) 2025-09-07T07:11:54.3034771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3034840Z outputs = self.mobilebert( 2025-09-07T07:11:54.3035123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3035194Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3035471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3035550Z layer_outputs = layer_module( 2025-09-07T07:11:54.3035825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3035918Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3036194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3036294Z self_outputs = self.self( 2025-09-07T07:11:54.3036589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3036659Z self.value(value_tensor) 2025-09-07T07:11:54.3036662Z 2025-09-07T07:11:54.3036772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3036967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3037039Z return mod(**inputs) 2025-09-07T07:11:54.3037317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3037386Z outputs = self.mobilebert( 2025-09-07T07:11:54.3037677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3037751Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3038044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3038114Z layer_outputs = layer_module( 2025-09-07T07:11:54.3038395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3038565Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3038846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3038963Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3039247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3039340Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3039366Z 2025-09-07T07:11:54.3039469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3039661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3039732Z return mod(**inputs) 2025-09-07T07:11:54.3040013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3040090Z outputs = self.mobilebert( 2025-09-07T07:11:54.3040373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3040446Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3040754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3040826Z layer_outputs = layer_module( 2025-09-07T07:11:54.3041122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3041284Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3041576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3041695Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3041971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3042062Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3042338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3042436Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3042470Z 2025-09-07T07:11:54.3042571Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3042771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3042835Z return mod(**inputs) 2025-09-07T07:11:54.3043112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3043191Z outputs = self.mobilebert( 2025-09-07T07:11:54.3043467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3043546Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3043826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3043896Z layer_outputs = layer_module( 2025-09-07T07:11:54.3044185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3044269Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3044555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3044625Z self_outputs = self.self( 2025-09-07T07:11:54.3044908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3044983Z self.query(query_tensor) 2025-09-07T07:11:54.3044986Z 2025-09-07T07:11:54.3045096Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3045296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3045361Z return mod(**inputs) 2025-09-07T07:11:54.3045636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3045727Z outputs = self.mobilebert( 2025-09-07T07:11:54.3046003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3046082Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3046371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3046448Z layer_outputs = layer_module( 2025-09-07T07:11:54.3046718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3046799Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3047093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3047163Z self_outputs = self.self( 2025-09-07T07:11:54.3047440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3047504Z self.key(key_tensor) 2025-09-07T07:11:54.3047507Z 2025-09-07T07:11:54.3047594Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3047673Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3047772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3047967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3048032Z return mod(**inputs) 2025-09-07T07:11:54.3048310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3048381Z outputs = self.mobilebert( 2025-09-07T07:11:54.3048650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3048757Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3049025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3049100Z layer_outputs = layer_module( 2025-09-07T07:11:54.3049369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3049451Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3049731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3049851Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3050128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3050214Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3050219Z 2025-09-07T07:11:54.3050326Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3050516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3050581Z return mod(**inputs) 2025-09-07T07:11:54.3050858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3050928Z outputs = self.mobilebert( 2025-09-07T07:11:54.3051215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3051283Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3051549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3051623Z layer_outputs = layer_module( 2025-09-07T07:11:54.3051916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3052007Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3052274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3052396Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3052665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3052786Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3053079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3053169Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3053175Z 2025-09-07T07:11:54.3053281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3053472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3053535Z return mod(**inputs) 2025-09-07T07:11:54.3053807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3053876Z outputs = self.mobilebert( 2025-09-07T07:11:54.3054153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3054222Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3054499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3054566Z layer_outputs = layer_module( 2025-09-07T07:11:54.3054835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3054962Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3055235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3055350Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3055615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3055695Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3055706Z 2025-09-07T07:11:54.3055802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3055990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3056061Z return mod(**inputs) 2025-09-07T07:11:54.3056329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3056406Z outputs = self.mobilebert( 2025-09-07T07:11:54.3056673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3056742Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3057017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3057084Z layer_outputs = layer_module( 2025-09-07T07:11:54.3057355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3057444Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3057711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3057845Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3058115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3058229Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3058233Z 2025-09-07T07:11:54.3058329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3058523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3058585Z return mod(**inputs) 2025-09-07T07:11:54.3058849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3058923Z outputs = self.mobilebert( 2025-09-07T07:11:54.3059202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3059282Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3059554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3059625Z layer_outputs = layer_module( 2025-09-07T07:11:54.3059907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3059998Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3060280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3060405Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3060688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3060812Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3060816Z 2025-09-07T07:11:54.3060918Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3061118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3061183Z return mod(**inputs) 2025-09-07T07:11:54.3061468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3061537Z outputs = self.mobilebert( 2025-09-07T07:11:54.3061816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3061895Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3062177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3062254Z layer_outputs = layer_module( 2025-09-07T07:11:54.3062534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3062632Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3062911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3063036Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3063318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3063437Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3063727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3063817Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3063823Z 2025-09-07T07:11:54.3063941Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3064145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3064213Z return mod(**inputs) 2025-09-07T07:11:54.3064496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3064565Z outputs = self.mobilebert( 2025-09-07T07:11:54.3064851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3064924Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3065220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3065298Z layer_outputs = layer_module( 2025-09-07T07:11:54.3065601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3065778Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3066101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3066223Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3066551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3066642Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3066646Z 2025-09-07T07:11:54.3066766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3066995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3067073Z return mod(**inputs) 2025-09-07T07:11:54.3067388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3067501Z outputs = self.mobilebert( 2025-09-07T07:11:54.3067811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3067889Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3068199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3068269Z layer_outputs = layer_module( 2025-09-07T07:11:54.3068545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3068643Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3068921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3069042Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3069318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3069433Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3069436Z 2025-09-07T07:11:54.3069535Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3069731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3069803Z return mod(**inputs) 2025-09-07T07:11:54.3070079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3070156Z outputs = self.mobilebert( 2025-09-07T07:11:54.3070461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3070537Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3070848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3070921Z layer_outputs = layer_module( 2025-09-07T07:11:54.3071213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3071307Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3071598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3071724Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3072031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3072125Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3072132Z 2025-09-07T07:11:54.3072231Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3072432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3072495Z return mod(**inputs) 2025-09-07T07:11:54.3072767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3072842Z outputs = self.mobilebert( 2025-09-07T07:11:54.3073119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3073196Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3073477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3073554Z layer_outputs = layer_module( 2025-09-07T07:11:54.3073894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3073985Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3074264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3074386Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3074671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3074793Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3075073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3075170Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3075176Z 2025-09-07T07:11:54.3075274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3075477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3075541Z return mod(**inputs) 2025-09-07T07:11:54.3075830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3075896Z outputs = self.mobilebert( 2025-09-07T07:11:54.3076157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3076233Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3076501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3076580Z layer_outputs = layer_module( 2025-09-07T07:11:54.3076855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3076963Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3077250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3077357Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3077629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3077708Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3077712Z 2025-09-07T07:11:54.3077815Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3078018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3078083Z return mod(**inputs) 2025-09-07T07:11:54.3078359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3078429Z outputs = self.mobilebert( 2025-09-07T07:11:54.3078705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3078775Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3079043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3079119Z layer_outputs = layer_module( 2025-09-07T07:11:54.3079388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3079488Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3079777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3079928Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3080211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3080326Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3080329Z 2025-09-07T07:11:54.3080441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3080641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3080714Z return mod(**inputs) 2025-09-07T07:11:54.3080998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3081070Z outputs = self.mobilebert( 2025-09-07T07:11:54.3081368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3081439Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3081716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3081784Z layer_outputs = layer_module( 2025-09-07T07:11:54.3082056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3082146Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3082412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3082537Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3082814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3082903Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3082909Z 2025-09-07T07:11:54.3083021Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3083210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3083280Z return mod(**inputs) 2025-09-07T07:11:54.3083545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3083621Z outputs = self.mobilebert( 2025-09-07T07:11:54.3083898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3083977Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3084262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3084332Z layer_outputs = layer_module( 2025-09-07T07:11:54.3084612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3084701Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3084977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3085097Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3085371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3085494Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3085769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3085865Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3085908Z 2025-09-07T07:11:54.3086007Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3086211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3086275Z return mod(**inputs) 2025-09-07T07:11:54.3086551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3086628Z outputs = self.mobilebert( 2025-09-07T07:11:54.3086906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3086984Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3087262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3087333Z layer_outputs = layer_module( 2025-09-07T07:11:54.3087627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3087750Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3088035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3088117Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3088120Z 2025-09-07T07:11:54.3088227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3088423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3088488Z return mod(**inputs) 2025-09-07T07:11:54.3088770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3088840Z outputs = self.mobilebert( 2025-09-07T07:11:54.3089122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3089211Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3089494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3089572Z layer_outputs = layer_module( 2025-09-07T07:11:54.3089856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3089983Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3090266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3090384Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3090402Z 2025-09-07T07:11:54.3090506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3090710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3090788Z return mod(**inputs) 2025-09-07T07:11:54.3091096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3091177Z outputs = self.mobilebert( 2025-09-07T07:11:54.3091484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3091556Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3091847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3091917Z layer_outputs = layer_module( 2025-09-07T07:11:54.3092208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3092401Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3092695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3092792Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3092795Z 2025-09-07T07:11:54.3092900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3093111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3093175Z return mod(**inputs) 2025-09-07T07:11:54.3093473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3093543Z outputs = self.mobilebert( 2025-09-07T07:11:54.3093823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3093906Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3094194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3094271Z layer_outputs = layer_module( 2025-09-07T07:11:54.3094555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3094722Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3095074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3095201Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3095499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3095595Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3095599Z 2025-09-07T07:11:54.3095725Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3095926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3095993Z return mod(**inputs) 2025-09-07T07:11:54.3096284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3096354Z outputs = self.mobilebert( 2025-09-07T07:11:54.3096643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3096717Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3097027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3097101Z layer_outputs = layer_module( 2025-09-07T07:11:54.3097389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3097557Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3097839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3097972Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3098256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3098350Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3098354Z 2025-09-07T07:11:54.3098460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3098659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3098764Z return mod(**inputs) 2025-09-07T07:11:54.3099049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3099127Z outputs = self.mobilebert( 2025-09-07T07:11:54.3099408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3099482Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3099778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3099851Z layer_outputs = layer_module( 2025-09-07T07:11:54.3100167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3100334Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3100643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3100774Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3101072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3101211Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3101523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3101629Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3101632Z 2025-09-07T07:11:54.3101742Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3101955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3102037Z return mod(**inputs) 2025-09-07T07:11:54.3102360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3102443Z outputs = self.mobilebert( 2025-09-07T07:11:54.3102742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3102826Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3103138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3103214Z layer_outputs = layer_module( 2025-09-07T07:11:54.3103523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3103715Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3104030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3104152Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3104454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3104550Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3104554Z 2025-09-07T07:11:54.3104662Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3104881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3104950Z return mod(**inputs) 2025-09-07T07:11:54.3105270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3105346Z outputs = self.mobilebert( 2025-09-07T07:11:54.3105647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3105867Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3106171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3106256Z layer_outputs = layer_module( 2025-09-07T07:11:54.3106558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3106651Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3106971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3107043Z self_outputs = self.self( 2025-09-07T07:11:54.3107340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3107415Z self.value(value_tensor) 2025-09-07T07:11:54.3107420Z 2025-09-07T07:11:54.3107532Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3107732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3107800Z return mod(**inputs) 2025-09-07T07:11:54.3108095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3108167Z outputs = self.mobilebert( 2025-09-07T07:11:54.3108463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3108536Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3108826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3108907Z layer_outputs = layer_module( 2025-09-07T07:11:54.3109223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3109394Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3109679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3109797Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3110080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3110163Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3110167Z 2025-09-07T07:11:54.3110304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3110506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3110584Z return mod(**inputs) 2025-09-07T07:11:54.3110887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3110961Z outputs = self.mobilebert( 2025-09-07T07:11:54.3111271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3111344Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3111633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3111703Z layer_outputs = layer_module( 2025-09-07T07:11:54.3111991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3112151Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3112481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3112596Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3112869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3112962Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3113237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3113327Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3113338Z 2025-09-07T07:11:54.3113438Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3113634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3113706Z return mod(**inputs) 2025-09-07T07:11:54.3113985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3114062Z outputs = self.mobilebert( 2025-09-07T07:11:54.3114332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3114405Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3114689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3114758Z layer_outputs = layer_module( 2025-09-07T07:11:54.3115038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3115126Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3115401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3115500Z self_outputs = self.self( 2025-09-07T07:11:54.3115780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3115857Z self.query(query_tensor) 2025-09-07T07:11:54.3115860Z 2025-09-07T07:11:54.3115962Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3116166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3116233Z return mod(**inputs) 2025-09-07T07:11:54.3116516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3116594Z outputs = self.mobilebert( 2025-09-07T07:11:54.3116896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3116979Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3117261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3117333Z layer_outputs = layer_module( 2025-09-07T07:11:54.3117629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3117710Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3117993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3118063Z self_outputs = self.self( 2025-09-07T07:11:54.3118341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3118413Z self.key(key_tensor) 2025-09-07T07:11:54.3118446Z 2025-09-07T07:11:54.3118528Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3118618Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3118719Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3118923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3118987Z return mod(**inputs) 2025-09-07T07:11:54.3119265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3119343Z outputs = self.mobilebert( 2025-09-07T07:11:54.3119772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3119858Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3120134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3120207Z layer_outputs = layer_module( 2025-09-07T07:11:54.3120492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3120577Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3120858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3120983Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3121300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3121392Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3121397Z 2025-09-07T07:11:54.3121509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3121726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3121801Z return mod(**inputs) 2025-09-07T07:11:54.3122139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3122216Z outputs = self.mobilebert( 2025-09-07T07:11:54.3122525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3122610Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3122929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3123009Z layer_outputs = layer_module( 2025-09-07T07:11:54.3123318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3123401Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3123682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3123804Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3124085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3124213Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3124496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3124586Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3124589Z 2025-09-07T07:11:54.3124690Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3124900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3124965Z return mod(**inputs) 2025-09-07T07:11:54.3125293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3125363Z outputs = self.mobilebert( 2025-09-07T07:11:54.3125632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3125710Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3125987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3126067Z layer_outputs = layer_module( 2025-09-07T07:11:54.3126342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3126446Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3126720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3126835Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3127113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3127197Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3127200Z 2025-09-07T07:11:54.3127308Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3127498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3127569Z return mod(**inputs) 2025-09-07T07:11:54.3127838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3127909Z outputs = self.mobilebert( 2025-09-07T07:11:54.3128186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3128276Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3128559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3128629Z layer_outputs = layer_module( 2025-09-07T07:11:54.3128904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3129006Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3129282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3129399Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3129691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3129813Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3129819Z 2025-09-07T07:11:54.3129919Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3130110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3130183Z return mod(**inputs) 2025-09-07T07:11:54.3130463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3130542Z outputs = self.mobilebert( 2025-09-07T07:11:54.3130820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3130895Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3131187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3131259Z layer_outputs = layer_module( 2025-09-07T07:11:54.3131569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3131663Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3131934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3132068Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3132340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3132430Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3132433Z 2025-09-07T07:11:54.3132533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3132735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3132803Z return mod(**inputs) 2025-09-07T07:11:54.3133078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3133154Z outputs = self.mobilebert( 2025-09-07T07:11:54.3133424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3133502Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3133771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3133842Z layer_outputs = layer_module( 2025-09-07T07:11:54.3134123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3134217Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3134525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3134653Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3134937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3135056Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3135343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3135443Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3135446Z 2025-09-07T07:11:54.3135547Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3135768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3135833Z return mod(**inputs) 2025-09-07T07:11:54.3136126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3136196Z outputs = self.mobilebert( 2025-09-07T07:11:54.3136470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3136547Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3136823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3136897Z layer_outputs = layer_module( 2025-09-07T07:11:54.3137173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3137265Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3137544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3137690Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3137970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3138052Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3138055Z 2025-09-07T07:11:54.3138160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3138353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3138417Z return mod(**inputs) 2025-09-07T07:11:54.3138703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3138770Z outputs = self.mobilebert( 2025-09-07T07:11:54.3139043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3139120Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3139404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3139481Z layer_outputs = layer_module( 2025-09-07T07:11:54.3139765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3139864Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3140149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3140267Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3140556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3140671Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3140675Z 2025-09-07T07:11:54.3140800Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3141001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3141074Z return mod(**inputs) 2025-09-07T07:11:54.3141360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3141432Z outputs = self.mobilebert( 2025-09-07T07:11:54.3141724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3141797Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3142106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3142179Z layer_outputs = layer_module( 2025-09-07T07:11:54.3142465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3142566Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3142849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3142983Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3143268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3143362Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3143366Z 2025-09-07T07:11:54.3143473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3143686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3143798Z return mod(**inputs) 2025-09-07T07:11:54.3144101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3144184Z outputs = self.mobilebert( 2025-09-07T07:11:54.3144488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3144565Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3144884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3144959Z layer_outputs = layer_module( 2025-09-07T07:11:54.3145274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3145374Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3145683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3145883Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3146206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3146343Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3146652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3146759Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3146763Z 2025-09-07T07:11:54.3146871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3147096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3147166Z return mod(**inputs) 2025-09-07T07:11:54.3147487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3147569Z outputs = self.mobilebert( 2025-09-07T07:11:54.3147834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3147911Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3148187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3148256Z layer_outputs = layer_module( 2025-09-07T07:11:54.3148534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3148641Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3148920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3149030Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3149307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3149386Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3149389Z 2025-09-07T07:11:54.3149488Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3149684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3149745Z return mod(**inputs) 2025-09-07T07:11:54.3150017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3150084Z outputs = self.mobilebert( 2025-09-07T07:11:54.3150351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3150464Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3150731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3150807Z layer_outputs = layer_module( 2025-09-07T07:11:54.3151075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3151172Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3151436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3151542Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3151817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3151923Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3151929Z 2025-09-07T07:11:54.3152035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3152223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3152284Z return mod(**inputs) 2025-09-07T07:11:54.3152560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3152628Z outputs = self.mobilebert( 2025-09-07T07:11:54.3152902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3152971Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3153246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3153314Z layer_outputs = layer_module( 2025-09-07T07:11:54.3153595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3153695Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3153962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3154090Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3154357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3154436Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3154439Z 2025-09-07T07:11:54.3154545Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3154749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3154821Z return mod(**inputs) 2025-09-07T07:11:54.3155089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3155165Z outputs = self.mobilebert( 2025-09-07T07:11:54.3155432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3155499Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3155775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3155844Z layer_outputs = layer_module( 2025-09-07T07:11:54.3156117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3156207Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3156476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3156643Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3156914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3157035Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3157302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3157397Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3157400Z 2025-09-07T07:11:54.3157497Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3157688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3157760Z return mod(**inputs) 2025-09-07T07:11:54.3158029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3158107Z outputs = self.mobilebert( 2025-09-07T07:11:54.3158373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3158443Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3158717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3158786Z layer_outputs = layer_module( 2025-09-07T07:11:54.3159062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3159179Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3159455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3159553Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3159557Z 2025-09-07T07:11:54.3159656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3159861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3159925Z return mod(**inputs) 2025-09-07T07:11:54.3160210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3160280Z outputs = self.mobilebert( 2025-09-07T07:11:54.3160556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3160636Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3160928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3161011Z layer_outputs = layer_module( 2025-09-07T07:11:54.3161291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3161415Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3161692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3161802Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3161805Z 2025-09-07T07:11:54.3161914Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3162106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3162176Z return mod(**inputs) 2025-09-07T07:11:54.3162455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3162563Z outputs = self.mobilebert( 2025-09-07T07:11:54.3162847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3162918Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3163200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3163269Z layer_outputs = layer_module( 2025-09-07T07:11:54.3163554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3163714Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3163995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3164097Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3164104Z 2025-09-07T07:11:54.3164205Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3164405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3164469Z return mod(**inputs) 2025-09-07T07:11:54.3164745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3164821Z outputs = self.mobilebert( 2025-09-07T07:11:54.3165099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3165176Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3165453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3165528Z layer_outputs = layer_module( 2025-09-07T07:11:54.3165822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3165982Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3166272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3166395Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3166681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3166772Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3166775Z 2025-09-07T07:11:54.3166874Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3167096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3167164Z return mod(**inputs) 2025-09-07T07:11:54.3167450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3167520Z outputs = self.mobilebert( 2025-09-07T07:11:54.3167803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3167874Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3168154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3168231Z layer_outputs = layer_module( 2025-09-07T07:11:54.3168508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3168669Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3168975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3169096Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3169415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3169495Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3169499Z 2025-09-07T07:11:54.3169605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3169804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3169874Z return mod(**inputs) 2025-09-07T07:11:54.3170147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3170215Z outputs = self.mobilebert( 2025-09-07T07:11:54.3170503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3170574Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3170858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3170928Z layer_outputs = layer_module( 2025-09-07T07:11:54.3171201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3171360Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3171642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3171768Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3172059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3172188Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3172464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3172553Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3172557Z 2025-09-07T07:11:54.3172664Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3172858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3172929Z return mod(**inputs) 2025-09-07T07:11:54.3173248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3173320Z outputs = self.mobilebert( 2025-09-07T07:11:54.3173609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3173684Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3173982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3174054Z layer_outputs = layer_module( 2025-09-07T07:11:54.3174376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3174569Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3174887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3175015Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3175329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3175460Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3175464Z 2025-09-07T07:11:54.3175574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3175793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3175862Z return mod(**inputs) 2025-09-07T07:11:54.3176239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3176331Z outputs = self.mobilebert( 2025-09-07T07:11:54.3176609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3176690Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3176972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3177048Z layer_outputs = layer_module( 2025-09-07T07:11:54.3177339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3177423Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3177708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3177777Z self_outputs = self.self( 2025-09-07T07:11:54.3178056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3178134Z self.value(value_tensor) 2025-09-07T07:11:54.3178138Z 2025-09-07T07:11:54.3178240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3178450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3178515Z return mod(**inputs) 2025-09-07T07:11:54.3178807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3178878Z outputs = self.mobilebert( 2025-09-07T07:11:54.3179144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3179226Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3179510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3179587Z layer_outputs = layer_module( 2025-09-07T07:11:54.3179889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3180053Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3180346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3180458Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3180748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3180831Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3180834Z 2025-09-07T07:11:54.3180944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3181143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3181209Z return mod(**inputs) 2025-09-07T07:11:54.3181501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3181575Z outputs = self.mobilebert( 2025-09-07T07:11:54.3181906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3181980Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3182263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3182342Z layer_outputs = layer_module( 2025-09-07T07:11:54.3182623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3182792Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3183079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3183198Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3183480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3183568Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3183858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3183950Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3183954Z 2025-09-07T07:11:54.3184060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3184258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3184331Z return mod(**inputs) 2025-09-07T07:11:54.3184617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3184690Z outputs = self.mobilebert( 2025-09-07T07:11:54.3184992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3185070Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3185355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3185425Z layer_outputs = layer_module( 2025-09-07T07:11:54.3185770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3185874Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3186157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3186236Z self_outputs = self.self( 2025-09-07T07:11:54.3186600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3186681Z self.query(query_tensor) 2025-09-07T07:11:54.3186692Z 2025-09-07T07:11:54.3186807Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3187031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3187106Z return mod(**inputs) 2025-09-07T07:11:54.3187398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3187475Z outputs = self.mobilebert( 2025-09-07T07:11:54.3187763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3187838Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3188139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3188209Z layer_outputs = layer_module( 2025-09-07T07:11:54.3188531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3188613Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3188886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3188962Z self_outputs = self.self( 2025-09-07T07:11:54.3189243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3189316Z self.key(key_tensor) 2025-09-07T07:11:54.3189319Z 2025-09-07T07:11:54.3189398Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3189484Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3189582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3189772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3189846Z return mod(**inputs) 2025-09-07T07:11:54.3190123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3190200Z outputs = self.mobilebert( 2025-09-07T07:11:54.3190480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3190550Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3190844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3190916Z layer_outputs = layer_module( 2025-09-07T07:11:54.3191211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3191296Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3191601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3191737Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3192018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3192113Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3192116Z 2025-09-07T07:11:54.3192218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3192423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3192488Z return mod(**inputs) 2025-09-07T07:11:54.3192786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3192867Z outputs = self.mobilebert( 2025-09-07T07:11:54.3193165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3193244Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3193519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3193589Z layer_outputs = layer_module( 2025-09-07T07:11:54.3193872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3193953Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3194238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3194357Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3194638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3194792Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3195071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3195171Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3195174Z 2025-09-07T07:11:54.3195270Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3195467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3195531Z return mod(**inputs) 2025-09-07T07:11:54.3195806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3195884Z outputs = self.mobilebert( 2025-09-07T07:11:54.3196163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3196246Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3196523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3196601Z layer_outputs = layer_module( 2025-09-07T07:11:54.3196878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3196970Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3197251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3197360Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3197645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3197729Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3197746Z 2025-09-07T07:11:54.3197847Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3198049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3198114Z return mod(**inputs) 2025-09-07T07:11:54.3198397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3198467Z outputs = self.mobilebert( 2025-09-07T07:11:54.3198754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3198825Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3199121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3199203Z layer_outputs = layer_module( 2025-09-07T07:11:54.3199481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3199585Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3199868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3199983Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3200279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3200393Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3200397Z 2025-09-07T07:11:54.3200507Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3200708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3200811Z return mod(**inputs) 2025-09-07T07:11:54.3201097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3201167Z outputs = self.mobilebert( 2025-09-07T07:11:54.3201457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3201540Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3201823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3201892Z layer_outputs = layer_module( 2025-09-07T07:11:54.3202171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3202270Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3202549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3202681Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3202960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3203051Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3203054Z 2025-09-07T07:11:54.3203154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3203347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3203419Z return mod(**inputs) 2025-09-07T07:11:54.3203697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3203776Z outputs = self.mobilebert( 2025-09-07T07:11:54.3204067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3204140Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3204423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3204493Z layer_outputs = layer_module( 2025-09-07T07:11:54.3204775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3204867Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3205153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3205302Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3205578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3205712Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3205990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3206085Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3206089Z 2025-09-07T07:11:54.3206189Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3206391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3206456Z return mod(**inputs) 2025-09-07T07:11:54.3206731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3206813Z outputs = self.mobilebert( 2025-09-07T07:11:54.3207089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3207200Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3207476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3207545Z layer_outputs = layer_module( 2025-09-07T07:11:54.3207829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3207921Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3208212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3208322Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3208605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3208699Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3208703Z 2025-09-07T07:11:54.3208802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3209003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3209067Z return mod(**inputs) 2025-09-07T07:11:54.3209353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3209422Z outputs = self.mobilebert( 2025-09-07T07:11:54.3209702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3209782Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3210062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3210146Z layer_outputs = layer_module( 2025-09-07T07:11:54.3211123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3211235Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3211523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3211631Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3211922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3212027Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3212030Z 2025-09-07T07:11:54.3212151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3212343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3212411Z return mod(**inputs) 2025-09-07T07:11:54.3212700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3212770Z outputs = self.mobilebert( 2025-09-07T07:11:54.3213056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3213126Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3213404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3213480Z layer_outputs = layer_module( 2025-09-07T07:11:54.3213760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3213861Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3214140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3214309Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3214593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3214679Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3214684Z 2025-09-07T07:11:54.3214796Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3215004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3215077Z return mod(**inputs) 2025-09-07T07:11:54.3215369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3215440Z outputs = self.mobilebert( 2025-09-07T07:11:54.3215740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3215817Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3216136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3216211Z layer_outputs = layer_module( 2025-09-07T07:11:54.3216534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3216634Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3216950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3217092Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3217408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3217569Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3217880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3217987Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3217997Z 2025-09-07T07:11:54.3218101Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3218303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3218375Z return mod(**inputs) 2025-09-07T07:11:54.3218677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3218777Z outputs = self.mobilebert( 2025-09-07T07:11:54.3219085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3219168Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3219481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3219718Z layer_outputs = layer_module( 2025-09-07T07:11:54.3220047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3220147Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3220445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3220571Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3220883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3221038Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3221044Z 2025-09-07T07:11:54.3221152Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3221382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3221450Z return mod(**inputs) 2025-09-07T07:11:54.3221755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3221839Z outputs = self.mobilebert( 2025-09-07T07:11:54.3222136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3222220Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3222531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3222605Z layer_outputs = layer_module( 2025-09-07T07:11:54.3222915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3223014Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3223319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3223436Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3223744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3223860Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3223864Z 2025-09-07T07:11:54.3223971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3224193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3224268Z return mod(**inputs) 2025-09-07T07:11:54.3224599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3224676Z outputs = self.mobilebert( 2025-09-07T07:11:54.3224987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3225069Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3225378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3225462Z layer_outputs = layer_module( 2025-09-07T07:11:54.3225814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3225952Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3226254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3226396Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3226711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3226805Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3226809Z 2025-09-07T07:11:54.3226935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3227132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3227196Z return mod(**inputs) 2025-09-07T07:11:54.3227481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3227550Z outputs = self.mobilebert( 2025-09-07T07:11:54.3227834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3227941Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3228246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3228323Z layer_outputs = layer_module( 2025-09-07T07:11:54.3228635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3228744Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3229054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3229198Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3229509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3229645Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3229952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3230050Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3230054Z 2025-09-07T07:11:54.3230172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3230388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3230466Z return mod(**inputs) 2025-09-07T07:11:54.3230783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3230860Z outputs = self.mobilebert( 2025-09-07T07:11:54.3231166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3231271Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3231577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3231654Z layer_outputs = layer_module( 2025-09-07T07:11:54.3232026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3232163Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3232458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3232555Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3232559Z 2025-09-07T07:11:54.3232682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3232905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3232980Z return mod(**inputs) 2025-09-07T07:11:54.3233278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3233361Z outputs = self.mobilebert( 2025-09-07T07:11:54.3233665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3233750Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3234048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3234123Z layer_outputs = layer_module( 2025-09-07T07:11:54.3234433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3234560Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3234899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3235017Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3235021Z 2025-09-07T07:11:54.3235136Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3235343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3235414Z return mod(**inputs) 2025-09-07T07:11:54.3235715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3235790Z outputs = self.mobilebert( 2025-09-07T07:11:54.3236090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3236165Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3236466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3236547Z layer_outputs = layer_module( 2025-09-07T07:11:54.3236843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3237020Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3237316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3237423Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3237427Z 2025-09-07T07:11:54.3237535Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3237746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3237824Z return mod(**inputs) 2025-09-07T07:11:54.3238135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3238218Z outputs = self.mobilebert( 2025-09-07T07:11:54.3238515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3238591Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3238895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3238970Z layer_outputs = layer_module( 2025-09-07T07:11:54.3239278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3239464Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3239770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3239906Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3240212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3240313Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3240316Z 2025-09-07T07:11:54.3240419Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3240624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3240692Z return mod(**inputs) 2025-09-07T07:11:54.3240986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3241056Z outputs = self.mobilebert( 2025-09-07T07:11:54.3241340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3241458Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3241742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3241819Z layer_outputs = layer_module( 2025-09-07T07:11:54.3242102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3242262Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3242553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3242680Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3242971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3243061Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3243065Z 2025-09-07T07:11:54.3243178Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3243378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3243444Z return mod(**inputs) 2025-09-07T07:11:54.3243732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3243804Z outputs = self.mobilebert( 2025-09-07T07:11:54.3244093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3244167Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3244455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3244553Z layer_outputs = layer_module( 2025-09-07T07:11:54.3244838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3245001Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3245287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3245416Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3245699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3245839Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3246143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3246245Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3246249Z 2025-09-07T07:11:54.3246365Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3246577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3246645Z return mod(**inputs) 2025-09-07T07:11:54.3246928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3246999Z outputs = self.mobilebert( 2025-09-07T07:11:54.3247287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3247361Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3247655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3247762Z layer_outputs = layer_module( 2025-09-07T07:11:54.3248062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3248253Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3248540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3248659Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3248944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3249029Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3249040Z 2025-09-07T07:11:54.3249143Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3249349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3249424Z return mod(**inputs) 2025-09-07T07:11:54.3249708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3249786Z outputs = self.mobilebert( 2025-09-07T07:11:54.3250067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3250140Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3250428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3250499Z layer_outputs = layer_module( 2025-09-07T07:11:54.3250789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3250877Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3251174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3251254Z self_outputs = self.self( 2025-09-07T07:11:54.3251547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3251629Z self.value(value_tensor) 2025-09-07T07:11:54.3251632Z 2025-09-07T07:11:54.3251741Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3251958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3252036Z return mod(**inputs) 2025-09-07T07:11:54.3252332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3252412Z outputs = self.mobilebert( 2025-09-07T07:11:54.3252700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3252779Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3253060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3253131Z layer_outputs = layer_module( 2025-09-07T07:11:54.3253424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3253586Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3253879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3253993Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3254285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3254410Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3254413Z 2025-09-07T07:11:54.3254516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3254723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3254788Z return mod(**inputs) 2025-09-07T07:11:54.3255078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3255148Z outputs = self.mobilebert( 2025-09-07T07:11:54.3255430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3255514Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3255795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3255876Z layer_outputs = layer_module( 2025-09-07T07:11:54.3256156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3256326Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3256609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3256720Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3257008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3257098Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3257388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3257500Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3257503Z 2025-09-07T07:11:54.3257607Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3257816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3257882Z return mod(**inputs) 2025-09-07T07:11:54.3258179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3258250Z outputs = self.mobilebert( 2025-09-07T07:11:54.3258543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3258617Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3258920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3259005Z layer_outputs = layer_module( 2025-09-07T07:11:54.3259292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3259384Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3259672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3259744Z self_outputs = self.self( 2025-09-07T07:11:54.3260046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3260120Z self.query(query_tensor) 2025-09-07T07:11:54.3260123Z 2025-09-07T07:11:54.3260239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3260453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3260564Z return mod(**inputs) 2025-09-07T07:11:54.3260863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3260937Z outputs = self.mobilebert( 2025-09-07T07:11:54.3261244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3261320Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3261627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3261703Z layer_outputs = layer_module( 2025-09-07T07:11:54.3262002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3262104Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3262406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3262489Z self_outputs = self.self( 2025-09-07T07:11:54.3262786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3262862Z self.key(key_tensor) 2025-09-07T07:11:54.3262866Z 2025-09-07T07:11:54.3262953Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3263038Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3263157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3263369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3263445Z return mod(**inputs) 2025-09-07T07:11:54.3263769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3263843Z outputs = self.mobilebert( 2025-09-07T07:11:54.3264174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3264252Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3264558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3264633Z layer_outputs = layer_module( 2025-09-07T07:11:54.3264955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3265052Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3265378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3265534Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3266001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3266120Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3266125Z 2025-09-07T07:11:54.3266240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3266464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3266543Z return mod(**inputs) 2025-09-07T07:11:54.3266866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3266951Z outputs = self.mobilebert( 2025-09-07T07:11:54.3267275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3267355Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3267664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3267791Z layer_outputs = layer_module( 2025-09-07T07:11:54.3268103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3268193Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3268526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3268656Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3268979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3269125Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3269423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3269530Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3269534Z 2025-09-07T07:11:54.3269642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3269852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3269927Z return mod(**inputs) 2025-09-07T07:11:54.3270249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3270331Z outputs = self.mobilebert( 2025-09-07T07:11:54.3270653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3270738Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3271039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3271118Z layer_outputs = layer_module( 2025-09-07T07:11:54.3271450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3271552Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3271856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3271975Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3272299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3272395Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3272399Z 2025-09-07T07:11:54.3272525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3272746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3272820Z return mod(**inputs) 2025-09-07T07:11:54.3273133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3273210Z outputs = self.mobilebert( 2025-09-07T07:11:54.3273508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3273593Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3273893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3273977Z layer_outputs = layer_module( 2025-09-07T07:11:54.3274281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3274382Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3274723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3274844Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3275152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3275273Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3275277Z 2025-09-07T07:11:54.3275402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3275608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3275674Z return mod(**inputs) 2025-09-07T07:11:54.3275966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3276038Z outputs = self.mobilebert( 2025-09-07T07:11:54.3276339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3276411Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3276691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3276770Z layer_outputs = layer_module( 2025-09-07T07:11:54.3277056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3277159Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3277444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3277591Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3277936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3278033Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3278036Z 2025-09-07T07:11:54.3278151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3278363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3278442Z return mod(**inputs) 2025-09-07T07:11:54.3278751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3278825Z outputs = self.mobilebert( 2025-09-07T07:11:54.3279138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3279231Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3279535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3279609Z layer_outputs = layer_module( 2025-09-07T07:11:54.3279899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3279993Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3280275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3280409Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3280690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3280818Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3281101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3281232Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3281245Z 2025-09-07T07:11:54.3281351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3281556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3281631Z return mod(**inputs) 2025-09-07T07:11:54.3281912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3281990Z outputs = self.mobilebert( 2025-09-07T07:11:54.3282278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3282350Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3282643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3282716Z layer_outputs = layer_module( 2025-09-07T07:11:54.3283010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3283105Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3283388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3283509Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3283792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3283884Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3283887Z 2025-09-07T07:11:54.3283991Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3284199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3284268Z return mod(**inputs) 2025-09-07T07:11:54.3284598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3284678Z outputs = self.mobilebert( 2025-09-07T07:11:54.3284957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3285036Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3285318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3285389Z layer_outputs = layer_module( 2025-09-07T07:11:54.3285691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3285785Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3286075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3286185Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3286473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3286585Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3286589Z 2025-09-07T07:11:54.3286690Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3286897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3286961Z return mod(**inputs) 2025-09-07T07:11:54.3287249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3287321Z outputs = self.mobilebert( 2025-09-07T07:11:54.3287633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3287715Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3287993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3288071Z layer_outputs = layer_module( 2025-09-07T07:11:54.3288352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3288452Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3288730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3288858Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3289147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3289237Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3289240Z 2025-09-07T07:11:54.3289348Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3289545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3289610Z return mod(**inputs) 2025-09-07T07:11:54.3289898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3289969Z outputs = self.mobilebert( 2025-09-07T07:11:54.3290261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3290338Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3290641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3290737Z layer_outputs = layer_module( 2025-09-07T07:11:54.3291032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3291138Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3291435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3291575Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3291873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3292003Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3292325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3292428Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3292432Z 2025-09-07T07:11:54.3292548Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3292760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3292835Z return mod(**inputs) 2025-09-07T07:11:54.3293134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3293211Z outputs = self.mobilebert( 2025-09-07T07:11:54.3293522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3293597Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3293907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3294021Z layer_outputs = layer_module( 2025-09-07T07:11:54.3294303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3294402Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3294688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3294806Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3295087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3295177Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3295181Z 2025-09-07T07:11:54.3295284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3295482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3295560Z return mod(**inputs) 2025-09-07T07:11:54.3295844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3295923Z outputs = self.mobilebert( 2025-09-07T07:11:54.3296204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3296279Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3296587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3296661Z layer_outputs = layer_module( 2025-09-07T07:11:54.3296970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3297068Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3297395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3297516Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3297817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3297946Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3297950Z 2025-09-07T07:11:54.3298059Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3298285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3298351Z return mod(**inputs) 2025-09-07T07:11:54.3298652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3298734Z outputs = self.mobilebert( 2025-09-07T07:11:54.3299037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3299120Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3299423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3299504Z layer_outputs = layer_module( 2025-09-07T07:11:54.3299801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3299901Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3300209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3300345Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3300652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3300778Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3300782Z 2025-09-07T07:11:54.3300894Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3301115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3301185Z return mod(**inputs) 2025-09-07T07:11:54.3301502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3301578Z outputs = self.mobilebert( 2025-09-07T07:11:54.3301883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3301961Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3302256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3302341Z layer_outputs = layer_module( 2025-09-07T07:11:54.3302642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3302747Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3303043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3303176Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3303480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3303608Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3303914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3304012Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3304034Z 2025-09-07T07:11:54.3304150Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3304362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3304431Z return mod(**inputs) 2025-09-07T07:11:54.3304737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3304813Z outputs = self.mobilebert( 2025-09-07T07:11:54.3305122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3305199Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3305526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3305614Z layer_outputs = layer_module( 2025-09-07T07:11:54.3306008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3306155Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3306464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3306563Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3306567Z 2025-09-07T07:11:54.3306678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3306904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3306984Z return mod(**inputs) 2025-09-07T07:11:54.3307295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3307418Z outputs = self.mobilebert( 2025-09-07T07:11:54.3307721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3307799Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3308109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3308184Z layer_outputs = layer_module( 2025-09-07T07:11:54.3308494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3308622Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3308943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3309065Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3309071Z 2025-09-07T07:11:54.3309182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3309407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3309479Z return mod(**inputs) 2025-09-07T07:11:54.3309786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3312181Z outputs = self.mobilebert( 2025-09-07T07:11:54.3313662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3313754Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3314073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3314149Z layer_outputs = layer_module( 2025-09-07T07:11:54.3314467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3314644Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3314969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3315073Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3315077Z 2025-09-07T07:11:54.3315241Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3315473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3315544Z return mod(**inputs) 2025-09-07T07:11:54.3315856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3315934Z outputs = self.mobilebert( 2025-09-07T07:11:54.3316243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3316324Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3316626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3316707Z layer_outputs = layer_module( 2025-09-07T07:11:54.3317013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3317192Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3317495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3317630Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3317941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3318074Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3318078Z 2025-09-07T07:11:54.3318197Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3318416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3318492Z return mod(**inputs) 2025-09-07T07:11:54.3318790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3318866Z outputs = self.mobilebert( 2025-09-07T07:11:54.3319172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3319248Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3319726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3319815Z layer_outputs = layer_module( 2025-09-07T07:11:54.3320114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3320291Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3320663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3320805Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3321128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3321228Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3321231Z 2025-09-07T07:11:54.3321336Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3321543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3321618Z return mod(**inputs) 2025-09-07T07:11:54.3321899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3321980Z outputs = self.mobilebert( 2025-09-07T07:11:54.3322267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3322340Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3322636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3322707Z layer_outputs = layer_module( 2025-09-07T07:11:54.3322998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3323163Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3323453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3323578Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3323861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3323994Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3324283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3324386Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3324389Z 2025-09-07T07:11:54.3324493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3324750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3324815Z return mod(**inputs) 2025-09-07T07:11:54.3325097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3325176Z outputs = self.mobilebert( 2025-09-07T07:11:54.3325466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3325552Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3325853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3325928Z layer_outputs = layer_module( 2025-09-07T07:11:54.3326238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3326419Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3326737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3326848Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3327168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3327253Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3327257Z 2025-09-07T07:11:54.3327380Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3327595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3327661Z return mod(**inputs) 2025-09-07T07:11:54.3327955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3328029Z outputs = self.mobilebert( 2025-09-07T07:11:54.3328315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3328398Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3328686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3328766Z layer_outputs = layer_module( 2025-09-07T07:11:54.3329054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3329140Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3329433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3329505Z self_outputs = self.self( 2025-09-07T07:11:54.3329800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3329871Z self.value(value_tensor) 2025-09-07T07:11:54.3329874Z 2025-09-07T07:11:54.3329986Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3330185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3330250Z return mod(**inputs) 2025-09-07T07:11:54.3330543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3330618Z outputs = self.mobilebert( 2025-09-07T07:11:54.3330908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3330982Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3331265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3331373Z layer_outputs = layer_module( 2025-09-07T07:11:54.3331656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3331828Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3332122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3332243Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3332529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3332611Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3332615Z 2025-09-07T07:11:54.3332739Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3332945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3333010Z return mod(**inputs) 2025-09-07T07:11:54.3333300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3333371Z outputs = self.mobilebert( 2025-09-07T07:11:54.3333692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3333763Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3334048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3334126Z layer_outputs = layer_module( 2025-09-07T07:11:54.3334391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3334553Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3334821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3334931Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3335199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3335282Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3335560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3335648Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3335651Z 2025-09-07T07:11:54.3335754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3335943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3336007Z return mod(**inputs) 2025-09-07T07:11:54.3336280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3336347Z outputs = self.mobilebert( 2025-09-07T07:11:54.3336622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3336692Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3336969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3337037Z layer_outputs = layer_module( 2025-09-07T07:11:54.3337305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3337395Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3337695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3337770Z self_outputs = self.self( 2025-09-07T07:11:54.3338040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3338106Z self.query(query_tensor) 2025-09-07T07:11:54.3338111Z 2025-09-07T07:11:54.3338217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3338405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3338473Z return mod(**inputs) 2025-09-07T07:11:54.3338742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3338817Z outputs = self.mobilebert( 2025-09-07T07:11:54.3339091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3339160Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3339452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3339523Z layer_outputs = layer_module( 2025-09-07T07:11:54.3339832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3339934Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3340227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3340302Z self_outputs = self.self( 2025-09-07T07:11:54.3340579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3340654Z self.key(key_tensor) 2025-09-07T07:11:54.3340658Z 2025-09-07T07:11:54.3340736Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3340815Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3340922Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3341115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3341187Z return mod(**inputs) 2025-09-07T07:11:54.3341471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3341546Z outputs = self.mobilebert( 2025-09-07T07:11:54.3341823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3341894Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3342179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3342250Z layer_outputs = layer_module( 2025-09-07T07:11:54.3342533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3342616Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3342892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3343020Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3343295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3343386Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3343389Z 2025-09-07T07:11:54.3343488Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3343720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3343786Z return mod(**inputs) 2025-09-07T07:11:54.3344066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3344143Z outputs = self.mobilebert( 2025-09-07T07:11:54.3344430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3344508Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3344791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3344860Z layer_outputs = layer_module( 2025-09-07T07:11:54.3345153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3345238Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3345545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3345682Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3346101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3346249Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3346587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3346697Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3346701Z 2025-09-07T07:11:54.3346815Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3347037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3347106Z return mod(**inputs) 2025-09-07T07:11:54.3347373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3347452Z outputs = self.mobilebert( 2025-09-07T07:11:54.3347730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3347809Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3348077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3348147Z layer_outputs = layer_module( 2025-09-07T07:11:54.3348424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3348517Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3348800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3348910Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3349189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3349272Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3349275Z 2025-09-07T07:11:54.3349375Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3349574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3349637Z return mod(**inputs) 2025-09-07T07:11:54.3349911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3349981Z outputs = self.mobilebert( 2025-09-07T07:11:54.3350293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3350365Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3350641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3350718Z layer_outputs = layer_module( 2025-09-07T07:11:54.3350992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3351093Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3351368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3351476Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3351758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3351870Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3351873Z 2025-09-07T07:11:54.3351980Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3352172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3352243Z return mod(**inputs) 2025-09-07T07:11:54.3352537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3352623Z outputs = self.mobilebert( 2025-09-07T07:11:54.3352904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3352975Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3353256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3353328Z layer_outputs = layer_module( 2025-09-07T07:11:54.3353601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3353700Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3353979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3354112Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3354390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3354479Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3354482Z 2025-09-07T07:11:54.3354580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3354777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3354848Z return mod(**inputs) 2025-09-07T07:11:54.3355123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3355198Z outputs = self.mobilebert( 2025-09-07T07:11:54.3355477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3355547Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3355829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3355899Z layer_outputs = layer_module( 2025-09-07T07:11:54.3356181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3356313Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3356586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3356715Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3356991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3357118Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3357395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3357491Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3357494Z 2025-09-07T07:11:54.3357595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3357795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3357869Z return mod(**inputs) 2025-09-07T07:11:54.3358141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3358217Z outputs = self.mobilebert( 2025-09-07T07:11:54.3358488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3358584Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3358875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3358945Z layer_outputs = layer_module( 2025-09-07T07:11:54.3359235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3359327Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3359615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3359726Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3360003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3360094Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3360098Z 2025-09-07T07:11:54.3360197Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3360401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3360466Z return mod(**inputs) 2025-09-07T07:11:54.3360756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3360826Z outputs = self.mobilebert( 2025-09-07T07:11:54.3361114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3361196Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3361485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3361561Z layer_outputs = layer_module( 2025-09-07T07:11:54.3361842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3361939Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3362232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3362345Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3362635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3362781Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3362785Z 2025-09-07T07:11:54.3362893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3363091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3363158Z return mod(**inputs) 2025-09-07T07:11:54.3363449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3363522Z outputs = self.mobilebert( 2025-09-07T07:11:54.3363817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3363889Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3364162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3364243Z layer_outputs = layer_module( 2025-09-07T07:11:54.3364516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3364615Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3364908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3365040Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3365328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3365414Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3365417Z 2025-09-07T07:11:54.3365526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3365722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3365792Z return mod(**inputs) 2025-09-07T07:11:54.3366066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3366136Z outputs = self.mobilebert( 2025-09-07T07:11:54.3366418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3366490Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3366774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3366843Z layer_outputs = layer_module( 2025-09-07T07:11:54.3367119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3367219Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3367501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3367628Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3367893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3368019Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3368294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3368385Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3368388Z 2025-09-07T07:11:54.3368497Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3368690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3368795Z return mod(**inputs) 2025-09-07T07:11:54.3369069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3369144Z outputs = self.mobilebert( 2025-09-07T07:11:54.3369424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3369496Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3369782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3369853Z layer_outputs = layer_module( 2025-09-07T07:11:54.3370144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3370237Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3370535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3370650Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3370927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3371015Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3371041Z 2025-09-07T07:11:54.3371142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3371361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3371426Z return mod(**inputs) 2025-09-07T07:11:54.3371699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3371773Z outputs = self.mobilebert( 2025-09-07T07:11:54.3372051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3372127Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3372402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3372471Z layer_outputs = layer_module( 2025-09-07T07:11:54.3372758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3372849Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3373126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3373234Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3373518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3373628Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3373631Z 2025-09-07T07:11:54.3373730Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3373929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3373994Z return mod(**inputs) 2025-09-07T07:11:54.3374278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3374350Z outputs = self.mobilebert( 2025-09-07T07:11:54.3374623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3374701Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3374978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3375084Z layer_outputs = layer_module( 2025-09-07T07:11:54.3375356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3375452Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3375730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3375852Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3376136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3376219Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3376222Z 2025-09-07T07:11:54.3376328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3376527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3376592Z return mod(**inputs) 2025-09-07T07:11:54.3376883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3376954Z outputs = self.mobilebert( 2025-09-07T07:11:54.3377259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3377334Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3377641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3377726Z layer_outputs = layer_module( 2025-09-07T07:11:54.3378006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3378109Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3378388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3378519Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3378793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3378914Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3379200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3379290Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3379293Z 2025-09-07T07:11:54.3379401Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3379599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3379676Z return mod(**inputs) 2025-09-07T07:11:54.3379960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3380032Z outputs = self.mobilebert( 2025-09-07T07:11:54.3380338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3380415Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3380724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3380798Z layer_outputs = layer_module( 2025-09-07T07:11:54.3381094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3381229Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3381603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3381698Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3381702Z 2025-09-07T07:11:54.3381809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3382030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3382100Z return mod(**inputs) 2025-09-07T07:11:54.3382399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3382480Z outputs = self.mobilebert( 2025-09-07T07:11:54.3382776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3382860Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3383160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3383235Z layer_outputs = layer_module( 2025-09-07T07:11:54.3383539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3383666Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3383992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3384129Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3384133Z 2025-09-07T07:11:54.3384253Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3384466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3384537Z return mod(**inputs) 2025-09-07T07:11:54.3384848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3384924Z outputs = self.mobilebert( 2025-09-07T07:11:54.3385233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3385311Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3385613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3385695Z layer_outputs = layer_module( 2025-09-07T07:11:54.3386083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3386263Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3386567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3386681Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3386685Z 2025-09-07T07:11:54.3386797Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3387010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3387089Z return mod(**inputs) 2025-09-07T07:11:54.3387391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3387476Z outputs = self.mobilebert( 2025-09-07T07:11:54.3387775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3387852Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3388163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3388278Z layer_outputs = layer_module( 2025-09-07T07:11:54.3388582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3388751Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3389072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3389205Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3389506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3389609Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3389613Z 2025-09-07T07:11:54.3389723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3389946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3390014Z return mod(**inputs) 2025-09-07T07:11:54.3390320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3390402Z outputs = self.mobilebert( 2025-09-07T07:11:54.3390728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3390811Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3391129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3391211Z layer_outputs = layer_module( 2025-09-07T07:11:54.3391511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3391683Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3391993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3392125Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3392445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3392538Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3392541Z 2025-09-07T07:11:54.3392661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3392875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3392947Z return mod(**inputs) 2025-09-07T07:11:54.3393257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3393337Z outputs = self.mobilebert( 2025-09-07T07:11:54.3393653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3393732Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3394043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3394127Z layer_outputs = layer_module( 2025-09-07T07:11:54.3394431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3394606Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3394925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3395091Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3395395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3395523Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3395840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3395937Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3395941Z 2025-09-07T07:11:54.3396057Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3396266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3396336Z return mod(**inputs) 2025-09-07T07:11:54.3396655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3396732Z outputs = self.mobilebert( 2025-09-07T07:11:54.3397036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3397113Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3397426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3397524Z layer_outputs = layer_module( 2025-09-07T07:11:54.3397842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3398028Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3398342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3398470Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3398782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3398870Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3398873Z 2025-09-07T07:11:54.3398992Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3399206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3399282Z return mod(**inputs) 2025-09-07T07:11:54.3399587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3399670Z outputs = self.mobilebert( 2025-09-07T07:11:54.3399975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3400051Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3400365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3400439Z layer_outputs = layer_module( 2025-09-07T07:11:54.3400746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3400838Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3401139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3401224Z self_outputs = self.self( 2025-09-07T07:11:54.3401522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3401604Z self.value(value_tensor) 2025-09-07T07:11:54.3401608Z 2025-09-07T07:11:54.3401715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3401975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3402045Z return mod(**inputs) 2025-09-07T07:11:54.3402352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3402443Z outputs = self.mobilebert( 2025-09-07T07:11:54.3402732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3402814Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3403107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3403178Z layer_outputs = layer_module( 2025-09-07T07:11:54.3403477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3403642Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3403939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3404052Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3404356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3404447Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3404452Z 2025-09-07T07:11:54.3404576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3404784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3404849Z return mod(**inputs) 2025-09-07T07:11:54.3405141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3405214Z outputs = self.mobilebert( 2025-09-07T07:11:54.3405498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3405580Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3405868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3405945Z layer_outputs = layer_module( 2025-09-07T07:11:54.3406231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3406394Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3406688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3406803Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3407097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3407183Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3407474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3407567Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3407571Z 2025-09-07T07:11:54.3407674Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3407883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3407947Z return mod(**inputs) 2025-09-07T07:11:54.3408239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3408343Z outputs = self.mobilebert( 2025-09-07T07:11:54.3408630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3408704Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3408987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3409070Z layer_outputs = layer_module( 2025-09-07T07:11:54.3409352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3409446Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3409727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3409799Z self_outputs = self.self( 2025-09-07T07:11:54.3410093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3410171Z self.query(query_tensor) 2025-09-07T07:11:54.3410174Z 2025-09-07T07:11:54.3410291Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3410501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3410569Z return mod(**inputs) 2025-09-07T07:11:54.3410900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3410993Z outputs = self.mobilebert( 2025-09-07T07:11:54.3411307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3411385Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3411688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3411768Z layer_outputs = layer_module( 2025-09-07T07:11:54.3412068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3412166Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3412471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3412552Z self_outputs = self.self( 2025-09-07T07:11:54.3412853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3412923Z self.key(key_tensor) 2025-09-07T07:11:54.3412933Z 2025-09-07T07:11:54.3413019Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3413105Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3413221Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3413436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3413506Z return mod(**inputs) 2025-09-07T07:11:54.3413823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3413894Z outputs = self.mobilebert( 2025-09-07T07:11:54.3414185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3414260Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3414552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3414622Z layer_outputs = layer_module( 2025-09-07T07:11:54.3414906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3415029Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3415312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3415445Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3415731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3415817Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3415827Z 2025-09-07T07:11:54.3415932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3416133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3416207Z return mod(**inputs) 2025-09-07T07:11:54.3416490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3416573Z outputs = self.mobilebert( 2025-09-07T07:11:54.3416856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3416929Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3417238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3417311Z layer_outputs = layer_module( 2025-09-07T07:11:54.3417619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3417705Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3417988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3418118Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3418406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3418543Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3418827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3418929Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3418933Z 2025-09-07T07:11:54.3419035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3419241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3419319Z return mod(**inputs) 2025-09-07T07:11:54.3419778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3419872Z outputs = self.mobilebert( 2025-09-07T07:11:54.3420177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3420256Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3420561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3420638Z layer_outputs = layer_module( 2025-09-07T07:11:54.3420947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3421051Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3421356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3421472Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3421815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3421908Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3421912Z 2025-09-07T07:11:54.3422012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3422216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3422285Z return mod(**inputs) 2025-09-07T07:11:54.3422585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3422667Z outputs = self.mobilebert( 2025-09-07T07:11:54.3422965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3423046Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3423345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3423431Z layer_outputs = layer_module( 2025-09-07T07:11:54.3423730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3423831Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3424194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3424354Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3424661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3424782Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3424785Z 2025-09-07T07:11:54.3424900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3425117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3425187Z return mod(**inputs) 2025-09-07T07:11:54.3425496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3425572Z outputs = self.mobilebert( 2025-09-07T07:11:54.3425938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3426020Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3426333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3426415Z layer_outputs = layer_module( 2025-09-07T07:11:54.3426715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3426826Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3427125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3427262Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3427575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3427666Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3427669Z 2025-09-07T07:11:54.3427787Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3427998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3428077Z return mod(**inputs) 2025-09-07T07:11:54.3428376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3428496Z outputs = self.mobilebert( 2025-09-07T07:11:54.3428809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3428887Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3429199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3429275Z layer_outputs = layer_module( 2025-09-07T07:11:54.3429575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3429684Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3429987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3430130Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3430430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3430569Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3430895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3431022Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3431026Z 2025-09-07T07:11:54.3431144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3431374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3431454Z return mod(**inputs) 2025-09-07T07:11:54.3431759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3431837Z outputs = self.mobilebert( 2025-09-07T07:11:54.3432143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3432221Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3432548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3432622Z layer_outputs = layer_module( 2025-09-07T07:11:54.3432938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3433033Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3433313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3433434Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3433720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3433813Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3433817Z 2025-09-07T07:11:54.3433922Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3434121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3434196Z return mod(**inputs) 2025-09-07T07:11:54.3434482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3434564Z outputs = self.mobilebert( 2025-09-07T07:11:54.3434851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3434930Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3435218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3435325Z layer_outputs = layer_module( 2025-09-07T07:11:54.3435625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3435719Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3436019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3436132Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3436424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3436544Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3436547Z 2025-09-07T07:11:54.3436651Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3436866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3436931Z return mod(**inputs) 2025-09-07T07:11:54.3437233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3437304Z outputs = self.mobilebert( 2025-09-07T07:11:54.3437613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3437695Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3437996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3438079Z layer_outputs = layer_module( 2025-09-07T07:11:54.3438369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3438468Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3438756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3438881Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3439172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3439256Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3439260Z 2025-09-07T07:11:54.3439369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3439571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3439634Z return mod(**inputs) 2025-09-07T07:11:54.3439921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3439995Z outputs = self.mobilebert( 2025-09-07T07:11:54.3440283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3440354Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3440641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3440720Z layer_outputs = layer_module( 2025-09-07T07:11:54.3441008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3441108Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3441391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3441523Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3441839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3441961Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3442253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3442346Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3442350Z 2025-09-07T07:11:54.3442463Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3442663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3442731Z return mod(**inputs) 2025-09-07T07:11:54.3443020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3443095Z outputs = self.mobilebert( 2025-09-07T07:11:54.3443385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3443458Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3443751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3443823Z layer_outputs = layer_module( 2025-09-07T07:11:54.3444142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3444260Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3444548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3444669Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3444959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3445046Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3445058Z 2025-09-07T07:11:54.3445161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3445364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3445436Z return mod(**inputs) 2025-09-07T07:11:54.3445730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3445809Z outputs = self.mobilebert( 2025-09-07T07:11:54.3446097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3446168Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3446453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3446527Z layer_outputs = layer_module( 2025-09-07T07:11:54.3446811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3446903Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3447181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3447297Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3447576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3447691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3447695Z 2025-09-07T07:11:54.3447794Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3448023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3448087Z return mod(**inputs) 2025-09-07T07:11:54.3448368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3448446Z outputs = self.mobilebert( 2025-09-07T07:11:54.3448731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3448809Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3449093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3449163Z layer_outputs = layer_module( 2025-09-07T07:11:54.3449455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3449549Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3449842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3449964Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3450285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3450371Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3450374Z 2025-09-07T07:11:54.3450493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3450708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3450772Z return mod(**inputs) 2025-09-07T07:11:54.3451068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3451146Z outputs = self.mobilebert( 2025-09-07T07:11:54.3451438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3451522Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3451841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3451925Z layer_outputs = layer_module( 2025-09-07T07:11:54.3452233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3452336Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3452628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3452754Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3453054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3453176Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3453485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3453575Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3453578Z 2025-09-07T07:11:54.3453684Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3453884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3453950Z return mod(**inputs) 2025-09-07T07:11:54.3454237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3454307Z outputs = self.mobilebert( 2025-09-07T07:11:54.3454629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3454699Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3454976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3455052Z layer_outputs = layer_module( 2025-09-07T07:11:54.3455329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3455458Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3455730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3455813Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3455823Z 2025-09-07T07:11:54.3455924Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3456115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3456186Z return mod(**inputs) 2025-09-07T07:11:54.3456460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3456534Z outputs = self.mobilebert( 2025-09-07T07:11:54.3456825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3456910Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3457199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3457267Z layer_outputs = layer_module( 2025-09-07T07:11:54.3457552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3457673Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3457954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3458070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3458074Z 2025-09-07T07:11:54.3458176Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3458381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3458447Z return mod(**inputs) 2025-09-07T07:11:54.3458733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3458802Z outputs = self.mobilebert( 2025-09-07T07:11:54.3459083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3459166Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3459445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3459522Z layer_outputs = layer_module( 2025-09-07T07:11:54.3459804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3459968Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3460262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3460358Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3460362Z 2025-09-07T07:11:54.3460473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3460707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3460781Z return mod(**inputs) 2025-09-07T07:11:54.3461067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3461137Z outputs = self.mobilebert( 2025-09-07T07:11:54.3461433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3461506Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3461797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3461868Z layer_outputs = layer_module( 2025-09-07T07:11:54.3462157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3462331Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3462616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3462749Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3463051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3463158Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3463162Z 2025-09-07T07:11:54.3463288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3463508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3463585Z return mod(**inputs) 2025-09-07T07:11:54.3463888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3463974Z outputs = self.mobilebert( 2025-09-07T07:11:54.3464274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3464369Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3464651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3464724Z layer_outputs = layer_module( 2025-09-07T07:11:54.3465017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3465177Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3465468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3465608Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3465986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3466091Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3466095Z 2025-09-07T07:11:54.3466208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3466436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3466509Z return mod(**inputs) 2025-09-07T07:11:54.3466828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3466904Z outputs = self.mobilebert( 2025-09-07T07:11:54.3467205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3468180Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3468465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3468545Z layer_outputs = layer_module( 2025-09-07T07:11:54.3468828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3468991Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3469290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3469415Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3469704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3469827Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3470118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3470211Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3470215Z 2025-09-07T07:11:54.3470317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3470541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3470609Z return mod(**inputs) 2025-09-07T07:11:54.3470913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3470986Z outputs = self.mobilebert( 2025-09-07T07:11:54.3471266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3471346Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3471632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3471709Z layer_outputs = layer_module( 2025-09-07T07:11:54.3471991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3472164Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3472450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3472564Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3472855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3472937Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3472944Z 2025-09-07T07:11:54.3473050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3473247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3473313Z return mod(**inputs) 2025-09-07T07:11:54.3473602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3473675Z outputs = self.mobilebert( 2025-09-07T07:11:54.3473962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3474034Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3474322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3474393Z layer_outputs = layer_module( 2025-09-07T07:11:54.3474674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3474800Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3475086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3475164Z self_outputs = self.self( 2025-09-07T07:11:54.3475450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3475520Z self.value(value_tensor) 2025-09-07T07:11:54.3475530Z 2025-09-07T07:11:54.3475635Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3475837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3475910Z return mod(**inputs) 2025-09-07T07:11:54.3476193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3476272Z outputs = self.mobilebert( 2025-09-07T07:11:54.3476556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3476628Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3476944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3477017Z layer_outputs = layer_module( 2025-09-07T07:11:54.3477321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3477485Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3477766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3477889Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3478171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3478263Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3478266Z 2025-09-07T07:11:54.3478811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3479031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3479096Z return mod(**inputs) 2025-09-07T07:11:54.3479371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3479450Z outputs = self.mobilebert( 2025-09-07T07:11:54.3479728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3479814Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3480096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3480168Z layer_outputs = layer_module( 2025-09-07T07:11:54.3480457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3480620Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3480911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3481022Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3481311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3481399Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3481747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3481855Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3481859Z 2025-09-07T07:11:54.3481968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3482201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3482270Z return mod(**inputs) 2025-09-07T07:11:54.3482577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3482654Z outputs = self.mobilebert( 2025-09-07T07:11:54.3482962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3483044Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3483356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3483438Z layer_outputs = layer_module( 2025-09-07T07:11:54.3483736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3483822Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3484136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3484223Z self_outputs = self.self( 2025-09-07T07:11:54.3484518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3484590Z self.query(query_tensor) 2025-09-07T07:11:54.3484593Z 2025-09-07T07:11:54.3484703Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3484904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3484969Z return mod(**inputs) 2025-09-07T07:11:54.3485265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3485337Z outputs = self.mobilebert( 2025-09-07T07:11:54.3485628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3485701Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3485991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3486071Z layer_outputs = layer_module( 2025-09-07T07:11:54.3486354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3486449Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3486740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3486809Z self_outputs = self.self( 2025-09-07T07:11:54.3487094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3487160Z self.key(key_tensor) 2025-09-07T07:11:54.3487163Z 2025-09-07T07:11:54.3487253Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3487336Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3487446Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3487648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3487715Z return mod(**inputs) 2025-09-07T07:11:54.3488006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3488138Z outputs = self.mobilebert( 2025-09-07T07:11:54.3488428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3488500Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3488783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3488860Z layer_outputs = layer_module( 2025-09-07T07:11:54.3489143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3489233Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3489516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3489647Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3489937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3490021Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3490024Z 2025-09-07T07:11:54.3490134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3490348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3490421Z return mod(**inputs) 2025-09-07T07:11:54.3490726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3490800Z outputs = self.mobilebert( 2025-09-07T07:11:54.3491093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3491170Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3491463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3491534Z layer_outputs = layer_module( 2025-09-07T07:11:54.3491818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3491909Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3492193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3492321Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3492605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3492748Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3493030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3493121Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3493124Z 2025-09-07T07:11:54.3493231Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3493423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3493496Z return mod(**inputs) 2025-09-07T07:11:54.3493774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3493844Z outputs = self.mobilebert( 2025-09-07T07:11:54.3494136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3494210Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3494531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3494604Z layer_outputs = layer_module( 2025-09-07T07:11:54.3494892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3494987Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3495269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3495392Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3495671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3495766Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3495769Z 2025-09-07T07:11:54.3495876Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3496097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3496167Z return mod(**inputs) 2025-09-07T07:11:54.3496474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3496557Z outputs = self.mobilebert( 2025-09-07T07:11:54.3496880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3496980Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3497279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3497355Z layer_outputs = layer_module( 2025-09-07T07:11:54.3497664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3497770Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3498077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3498196Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3498505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3498633Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3498637Z 2025-09-07T07:11:54.3498748Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3498968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3499037Z return mod(**inputs) 2025-09-07T07:11:54.3499352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3499432Z outputs = self.mobilebert( 2025-09-07T07:11:54.3499731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3499818Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3500130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3500213Z layer_outputs = layer_module( 2025-09-07T07:11:54.3500512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3500613Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3500921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3501058Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3501396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3501486Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3501489Z 2025-09-07T07:11:54.3501604Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3501816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3501887Z return mod(**inputs) 2025-09-07T07:11:54.3502190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3502264Z outputs = self.mobilebert( 2025-09-07T07:11:54.3502572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3502649Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3502951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3503034Z layer_outputs = layer_module( 2025-09-07T07:11:54.3503331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3503437Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3503752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3503921Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3504225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3504356Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3504668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3504766Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3504769Z 2025-09-07T07:11:54.3504883Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3505095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3505173Z return mod(**inputs) 2025-09-07T07:11:54.3505471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3505546Z outputs = self.mobilebert( 2025-09-07T07:11:54.3505926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3506010Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3506323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3506398Z layer_outputs = layer_module( 2025-09-07T07:11:54.3506700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3506812Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3507112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3507242Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3507546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3507636Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3507648Z 2025-09-07T07:11:54.3507758Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3508011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3508091Z return mod(**inputs) 2025-09-07T07:11:54.3508389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3508470Z outputs = self.mobilebert( 2025-09-07T07:11:54.3508777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3508855Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3509163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3509237Z layer_outputs = layer_module( 2025-09-07T07:11:54.3509545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3509647Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3509946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3510075Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3510391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3510520Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3510523Z 2025-09-07T07:11:54.3510649Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3510870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3510945Z return mod(**inputs) 2025-09-07T07:11:54.3511245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3511333Z outputs = self.mobilebert( 2025-09-07T07:11:54.3511632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3511716Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3512028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3512100Z layer_outputs = layer_module( 2025-09-07T07:11:54.3512390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3512484Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3512792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3512927Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3513238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3513330Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3513335Z 2025-09-07T07:11:54.3513445Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3513666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3513735Z return mod(**inputs) 2025-09-07T07:11:54.3514043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3514118Z outputs = self.mobilebert( 2025-09-07T07:11:54.3514417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3514500Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3514840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3514923Z layer_outputs = layer_module( 2025-09-07T07:11:54.3515235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3515341Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3515650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3515786Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3516094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3516223Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3516537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3516634Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3516638Z 2025-09-07T07:11:54.3516754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3516964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3517051Z return mod(**inputs) 2025-09-07T07:11:54.3517374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3517451Z outputs = self.mobilebert( 2025-09-07T07:11:54.3517753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3517829Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3518138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3518225Z layer_outputs = layer_module( 2025-09-07T07:11:54.3518522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3518626Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3518940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3519059Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3519363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3519451Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3519455Z 2025-09-07T07:11:54.3519764Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3519986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3520074Z return mod(**inputs) 2025-09-07T07:11:54.3520359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3520433Z outputs = self.mobilebert( 2025-09-07T07:11:54.3520728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3520804Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3521116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3521192Z layer_outputs = layer_module( 2025-09-07T07:11:54.3521498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3521664Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3521944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3522063Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3522352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3522469Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3522472Z 2025-09-07T07:11:54.3522575Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3522773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3522848Z return mod(**inputs) 2025-09-07T07:11:54.3523129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3523211Z outputs = self.mobilebert( 2025-09-07T07:11:54.3523491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3523564Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3523852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3523949Z layer_outputs = layer_module( 2025-09-07T07:11:54.3524263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3524358Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3524651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3524780Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3525065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3525156Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3525160Z 2025-09-07T07:11:54.3525263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3525470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3525536Z return mod(**inputs) 2025-09-07T07:11:54.3525817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3525896Z outputs = self.mobilebert( 2025-09-07T07:11:54.3526182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3526262Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3526550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3526628Z layer_outputs = layer_module( 2025-09-07T07:11:54.3526911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3527006Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3527297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3527425Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3527719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3527843Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3528189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3528282Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3528285Z 2025-09-07T07:11:54.3528387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3528597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3528664Z return mod(**inputs) 2025-09-07T07:11:54.3528961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3529032Z outputs = self.mobilebert( 2025-09-07T07:11:54.3529312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3529392Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3529674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3529756Z layer_outputs = layer_module( 2025-09-07T07:11:54.3530037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3530157Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3530481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3530585Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3530589Z 2025-09-07T07:11:54.3530705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3530906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3530982Z return mod(**inputs) 2025-09-07T07:11:54.3531267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3531344Z outputs = self.mobilebert( 2025-09-07T07:11:54.3531642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3531720Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3532015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3532090Z layer_outputs = layer_module( 2025-09-07T07:11:54.3532377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3532508Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3532797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3532923Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3532927Z 2025-09-07T07:11:54.3533035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3533246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3533315Z return mod(**inputs) 2025-09-07T07:11:54.3533612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3533696Z outputs = self.mobilebert( 2025-09-07T07:11:54.3533985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3534069Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3534364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3534468Z layer_outputs = layer_module( 2025-09-07T07:11:54.3534744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3534900Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3535181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3535273Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3535276Z 2025-09-07T07:11:54.3535383Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3535572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3535633Z return mod(**inputs) 2025-09-07T07:11:54.3535914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3535983Z outputs = self.mobilebert( 2025-09-07T07:11:54.3536258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3536326Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3536602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3536686Z layer_outputs = layer_module( 2025-09-07T07:11:54.3536980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3537146Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3537431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3537559Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3537851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3537939Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3537950Z 2025-09-07T07:11:54.3538047Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3538237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3538306Z return mod(**inputs) 2025-09-07T07:11:54.3555729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3555936Z outputs = self.mobilebert( 2025-09-07T07:11:54.3556284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3556367Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3556679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3556755Z layer_outputs = layer_module( 2025-09-07T07:11:54.3557044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3557213Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3557507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3557638Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3557920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3558020Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3558129Z 2025-09-07T07:11:54.3558247Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3558467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3558537Z return mod(**inputs) 2025-09-07T07:11:54.3558819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3558908Z outputs = self.mobilebert( 2025-09-07T07:11:54.3559190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3559276Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3559562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3559645Z layer_outputs = layer_module( 2025-09-07T07:11:54.3559937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3560126Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3560418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3560543Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3560859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3560999Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3561292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3561394Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3561403Z 2025-09-07T07:11:54.3561512Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3561729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3561798Z return mod(**inputs) 2025-09-07T07:11:54.3562099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3562177Z outputs = self.mobilebert( 2025-09-07T07:11:54.3562463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3562550Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3562842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3562921Z layer_outputs = layer_module( 2025-09-07T07:11:54.3563199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3563372Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3563652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3563765Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3564055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3564141Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3564145Z 2025-09-07T07:11:54.3564256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3564456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3564523Z return mod(**inputs) 2025-09-07T07:11:54.3564836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3564907Z outputs = self.mobilebert( 2025-09-07T07:11:54.3565191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3565265Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3565550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3565622Z layer_outputs = layer_module( 2025-09-07T07:11:54.3565899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3565996Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3566269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3566351Z self_outputs = self.self( 2025-09-07T07:11:54.3566625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3566697Z self.value(value_tensor) 2025-09-07T07:11:54.3566710Z 2025-09-07T07:11:54.3566812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3567020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3567092Z return mod(**inputs) 2025-09-07T07:11:54.3567394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3567466Z outputs = self.mobilebert( 2025-09-07T07:11:54.3567757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3567833Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3568114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3568192Z layer_outputs = layer_module( 2025-09-07T07:11:54.3568474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3568644Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3568927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3569046Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3569332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3569414Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3569421Z 2025-09-07T07:11:54.3569531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3569737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3569812Z return mod(**inputs) 2025-09-07T07:11:54.3570101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3570180Z outputs = self.mobilebert( 2025-09-07T07:11:54.3570471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3570546Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3570843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3570914Z layer_outputs = layer_module( 2025-09-07T07:11:54.3571248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3571419Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3571693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3571811Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3572098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3572195Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3572478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3572580Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3572586Z 2025-09-07T07:11:54.3572689Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3572890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3572964Z return mod(**inputs) 2025-09-07T07:11:54.3573248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3573326Z outputs = self.mobilebert( 2025-09-07T07:11:54.3573624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3573714Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3574006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3574078Z layer_outputs = layer_module( 2025-09-07T07:11:54.3574370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3574461Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3574750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3574821Z self_outputs = self.self( 2025-09-07T07:11:54.3575104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3575184Z self.query(query_tensor) 2025-09-07T07:11:54.3575187Z 2025-09-07T07:11:54.3575293Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3575500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3575567Z return mod(**inputs) 2025-09-07T07:11:54.3575851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3575935Z outputs = self.mobilebert( 2025-09-07T07:11:54.3576223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3576305Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3576590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3576663Z layer_outputs = layer_module( 2025-09-07T07:11:54.3576957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3577043Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3577336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3577407Z self_outputs = self.self( 2025-09-07T07:11:54.3577726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3577794Z self.key(key_tensor) 2025-09-07T07:11:54.3577797Z 2025-09-07T07:11:54.3577882Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3577972Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3578076Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3578283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3578351Z return mod(**inputs) 2025-09-07T07:11:54.3578634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3578716Z outputs = self.mobilebert( 2025-09-07T07:11:54.3579005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3579089Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3579369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3579441Z layer_outputs = layer_module( 2025-09-07T07:11:54.3579730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3579829Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3580134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3580262Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3580552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3580639Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3580645Z 2025-09-07T07:11:54.3580749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3580957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3581024Z return mod(**inputs) 2025-09-07T07:11:54.3581313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3581386Z outputs = self.mobilebert( 2025-09-07T07:11:54.3581672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3581755Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3582040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3582119Z layer_outputs = layer_module( 2025-09-07T07:11:54.3582405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3582498Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3582785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3582909Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3583200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3583329Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3583618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3583710Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3583714Z 2025-09-07T07:11:54.3583850Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3584052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3584119Z return mod(**inputs) 2025-09-07T07:11:54.3584410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3584482Z outputs = self.mobilebert( 2025-09-07T07:11:54.3584773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3584848Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3585130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3585210Z layer_outputs = layer_module( 2025-09-07T07:11:54.3585490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3585596Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3585986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3586118Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3586454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3586551Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3586571Z 2025-09-07T07:11:54.3586694Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3586914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3586995Z return mod(**inputs) 2025-09-07T07:11:54.3587311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3587387Z outputs = self.mobilebert( 2025-09-07T07:11:54.3587686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3587759Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3588061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3588135Z layer_outputs = layer_module( 2025-09-07T07:11:54.3588427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3588534Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3588828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3588955Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3589246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3589370Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3589374Z 2025-09-07T07:11:54.3589476Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3589681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3589755Z return mod(**inputs) 2025-09-07T07:11:54.3590050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3590127Z outputs = self.mobilebert( 2025-09-07T07:11:54.3590417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3590526Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3590817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3590889Z layer_outputs = layer_module( 2025-09-07T07:11:54.3591176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3591273Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3591566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3591695Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3591977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3592069Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3592075Z 2025-09-07T07:11:54.3592178Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3592390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3592457Z return mod(**inputs) 2025-09-07T07:11:54.3592742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3592837Z outputs = self.mobilebert( 2025-09-07T07:11:54.3593132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3593215Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3593498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3593578Z layer_outputs = layer_module( 2025-09-07T07:11:54.3593859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3593959Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3594249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3594376Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3594666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3594793Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3595082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3595172Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3595175Z 2025-09-07T07:11:54.3595280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3595487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3595553Z return mod(**inputs) 2025-09-07T07:11:54.3595844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3595913Z outputs = self.mobilebert( 2025-09-07T07:11:54.3596187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3596268Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3596540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3596617Z layer_outputs = layer_module( 2025-09-07T07:11:54.3596890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3597020Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3597303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3597413Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3597701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3597783Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3597787Z 2025-09-07T07:11:54.3597894Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3598093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3598158Z return mod(**inputs) 2025-09-07T07:11:54.3598449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3598521Z outputs = self.mobilebert( 2025-09-07T07:11:54.3598815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3598887Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3599195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3599277Z layer_outputs = layer_module( 2025-09-07T07:11:54.3599585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3599689Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3599967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3600090Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3600377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3600492Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3600495Z 2025-09-07T07:11:54.3600609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3600808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3600880Z return mod(**inputs) 2025-09-07T07:11:54.3601161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3601233Z outputs = self.mobilebert( 2025-09-07T07:11:54.3601520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3601597Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3601885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3601954Z layer_outputs = layer_module( 2025-09-07T07:11:54.3602242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3602337Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3602619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3602754Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3603034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3603126Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3603166Z 2025-09-07T07:11:54.3603279Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3603482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3603546Z return mod(**inputs) 2025-09-07T07:11:54.3603820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3603899Z outputs = self.mobilebert( 2025-09-07T07:11:54.3604177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3604255Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3604530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3604598Z layer_outputs = layer_module( 2025-09-07T07:11:54.3604879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3604975Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3605264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3605390Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3605684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3605855Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3606136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3606234Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3606238Z 2025-09-07T07:11:54.3606340Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3606547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3606614Z return mod(**inputs) 2025-09-07T07:11:54.3606905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3606982Z outputs = self.mobilebert( 2025-09-07T07:11:54.3607257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3607335Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3607607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3607676Z layer_outputs = layer_module( 2025-09-07T07:11:54.3607958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3608056Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3608346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3608460Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3608750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3608834Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3608837Z 2025-09-07T07:11:54.3608941Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3609145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3609213Z return mod(**inputs) 2025-09-07T07:11:54.3609500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3609602Z outputs = self.mobilebert( 2025-09-07T07:11:54.3609885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3609967Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3610261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3610340Z layer_outputs = layer_module( 2025-09-07T07:11:54.3610634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3610737Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3611021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3611134Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3611428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3611541Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3611545Z 2025-09-07T07:11:54.3611655Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3611872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3611947Z return mod(**inputs) 2025-09-07T07:11:54.3612254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3612329Z outputs = self.mobilebert( 2025-09-07T07:11:54.3612628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3612705Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3613009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3613078Z layer_outputs = layer_module( 2025-09-07T07:11:54.3613353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3613456Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3613743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3613877Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3614165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3614257Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3614264Z 2025-09-07T07:11:54.3614368Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3614572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3614649Z return mod(**inputs) 2025-09-07T07:11:54.3614933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3615013Z outputs = self.mobilebert( 2025-09-07T07:11:54.3615299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3615375Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3615677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3615748Z layer_outputs = layer_module( 2025-09-07T07:11:54.3616037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3616172Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3616474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3616616Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3616919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3617058Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3617360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3617464Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3617467Z 2025-09-07T07:11:54.3617577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3617795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3617872Z return mod(**inputs) 2025-09-07T07:11:54.3618174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3618258Z outputs = self.mobilebert( 2025-09-07T07:11:54.3618583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3618686Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3618969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3619041Z layer_outputs = layer_module( 2025-09-07T07:11:54.3619330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3619461Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3619957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3620049Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3620054Z 2025-09-07T07:11:54.3620158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3620374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3620445Z return mod(**inputs) 2025-09-07T07:11:54.3620753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3620828Z outputs = self.mobilebert( 2025-09-07T07:11:54.3621136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3621219Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3621520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3621605Z layer_outputs = layer_module( 2025-09-07T07:11:54.3621905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3622045Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3622343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3622465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3622470Z 2025-09-07T07:11:54.3622588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3622800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3622963Z return mod(**inputs) 2025-09-07T07:11:54.3623262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3623346Z outputs = self.mobilebert( 2025-09-07T07:11:54.3623647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3623726Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3624040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3624117Z layer_outputs = layer_module( 2025-09-07T07:11:54.3624431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3624605Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3624910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3625023Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3625027Z 2025-09-07T07:11:54.3625137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3625384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3625458Z return mod(**inputs) 2025-09-07T07:11:54.3625862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3625947Z outputs = self.mobilebert( 2025-09-07T07:11:54.3626246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3626331Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3626635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3626716Z layer_outputs = layer_module( 2025-09-07T07:11:54.3627017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3627179Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3627473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3627598Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3627887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3627987Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3627993Z 2025-09-07T07:11:54.3628107Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3628306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3628372Z return mod(**inputs) 2025-09-07T07:11:54.3628662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3628737Z outputs = self.mobilebert( 2025-09-07T07:11:54.3629026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3629100Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3629383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3629462Z layer_outputs = layer_module( 2025-09-07T07:11:54.3629746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3629946Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3630226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3630355Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3630638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3630725Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3630729Z 2025-09-07T07:11:54.3630842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3631041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3631114Z return mod(**inputs) 2025-09-07T07:11:54.3631402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3631473Z outputs = self.mobilebert( 2025-09-07T07:11:54.3631763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3631837Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3632144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3632232Z layer_outputs = layer_module( 2025-09-07T07:11:54.3632514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3632676Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3632958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3633092Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3633373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3633501Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3633782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3633876Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3633888Z 2025-09-07T07:11:54.3633990Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3634190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3634264Z return mod(**inputs) 2025-09-07T07:11:54.3634547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3634625Z outputs = self.mobilebert( 2025-09-07T07:11:54.3634903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3634976Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3635274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3635345Z layer_outputs = layer_module( 2025-09-07T07:11:54.3635623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3635783Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3636059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3636205Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3636479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3636567Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3636571Z 2025-09-07T07:11:54.3636671Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3636870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3636935Z return mod(**inputs) 2025-09-07T07:11:54.3637208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3637285Z outputs = self.mobilebert( 2025-09-07T07:11:54.3637558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3637640Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3637926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3637993Z layer_outputs = layer_module( 2025-09-07T07:11:54.3638281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3638369Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3638676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3638750Z self_outputs = self.self( 2025-09-07T07:11:54.3639043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3639120Z self.value(value_tensor) 2025-09-07T07:11:54.3639124Z 2025-09-07T07:11:54.3639238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3639462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3639533Z return mod(**inputs) 2025-09-07T07:11:54.3639858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3639936Z outputs = self.mobilebert( 2025-09-07T07:11:54.3640237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3640324Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3640623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3640719Z layer_outputs = layer_module( 2025-09-07T07:11:54.3641005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3641174Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3641461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3641568Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3641846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3641925Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3641928Z 2025-09-07T07:11:54.3642034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3642223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3642285Z return mod(**inputs) 2025-09-07T07:11:54.3642591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3642661Z outputs = self.mobilebert( 2025-09-07T07:11:54.3642940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3643010Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3643297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3643367Z layer_outputs = layer_module( 2025-09-07T07:11:54.3643638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3643801Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3644079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3644198Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3644478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3644561Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3644858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3644961Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3644964Z 2025-09-07T07:11:54.3645070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3645257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3645329Z return mod(**inputs) 2025-09-07T07:11:54.3645598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3645666Z outputs = self.mobilebert( 2025-09-07T07:11:54.3645943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3646013Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3646290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3646358Z layer_outputs = layer_module( 2025-09-07T07:11:54.3646625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3646719Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3646990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3647067Z self_outputs = self.self( 2025-09-07T07:11:54.3647333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3647409Z self.query(query_tensor) 2025-09-07T07:11:54.3647412Z 2025-09-07T07:11:54.3647511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3647702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3647776Z return mod(**inputs) 2025-09-07T07:11:54.3648041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3648117Z outputs = self.mobilebert( 2025-09-07T07:11:54.3648382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3648479Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3648759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3648826Z layer_outputs = layer_module( 2025-09-07T07:11:54.3649101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3649185Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3649456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3649530Z self_outputs = self.self( 2025-09-07T07:11:54.3649799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3649868Z self.key(key_tensor) 2025-09-07T07:11:54.3649871Z 2025-09-07T07:11:54.3649950Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3650036Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3650133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3650325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3650396Z return mod(**inputs) 2025-09-07T07:11:54.3650692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3650767Z outputs = self.mobilebert( 2025-09-07T07:11:54.3651063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3651133Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3651410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3651475Z layer_outputs = layer_module( 2025-09-07T07:11:54.3651752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3651834Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3652104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3652234Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3652505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3652592Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3652596Z 2025-09-07T07:11:54.3652692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3652886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3652950Z return mod(**inputs) 2025-09-07T07:11:54.3653220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3653296Z outputs = self.mobilebert( 2025-09-07T07:11:54.3653563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3653639Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3653908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3653977Z layer_outputs = layer_module( 2025-09-07T07:11:54.3654255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3654336Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3654613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3654765Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3655041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3655162Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3655436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3655536Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3655539Z 2025-09-07T07:11:54.3655638Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3655837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3655901Z return mod(**inputs) 2025-09-07T07:11:54.3656182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3656254Z outputs = self.mobilebert( 2025-09-07T07:11:54.3656527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3656606Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3656890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3656966Z layer_outputs = layer_module( 2025-09-07T07:11:54.3657254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3657346Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3657626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3657736Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3658016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3658097Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3658100Z 2025-09-07T07:11:54.3658202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3658398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3658462Z return mod(**inputs) 2025-09-07T07:11:54.3658746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3658815Z outputs = self.mobilebert( 2025-09-07T07:11:54.3659094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3659170Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3659443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3659520Z layer_outputs = layer_module( 2025-09-07T07:11:54.3659799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3659903Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3660184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3660298Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3660585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3660696Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3660728Z 2025-09-07T07:11:54.3660839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3661040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3661113Z return mod(**inputs) 2025-09-07T07:11:54.3661399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3661472Z outputs = self.mobilebert( 2025-09-07T07:11:54.3661763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3661836Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3662129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3662201Z layer_outputs = layer_module( 2025-09-07T07:11:54.3662486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3662589Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3662878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3663022Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3663349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3663466Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3663470Z 2025-09-07T07:11:54.3663580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3663791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3663869Z return mod(**inputs) 2025-09-07T07:11:54.3664171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3664253Z outputs = self.mobilebert( 2025-09-07T07:11:54.3664563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3664654Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3664943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3665017Z layer_outputs = layer_module( 2025-09-07T07:11:54.3665305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3665399Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3665690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3665897Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3666211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3666351Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3666680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3666793Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3666797Z 2025-09-07T07:11:54.3666910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3667141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3667214Z return mod(**inputs) 2025-09-07T07:11:54.3667540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3667680Z outputs = self.mobilebert( 2025-09-07T07:11:54.3668007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3668086Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3668373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3668443Z layer_outputs = layer_module( 2025-09-07T07:11:54.3668732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3668826Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3669126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3669239Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3669530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3669614Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3669618Z 2025-09-07T07:11:54.3669718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3669944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3670011Z return mod(**inputs) 2025-09-07T07:11:54.3670308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3670381Z outputs = self.mobilebert( 2025-09-07T07:11:54.3670665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3670751Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3671047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3671125Z layer_outputs = layer_module( 2025-09-07T07:11:54.3671408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3671503Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3671793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3671902Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3672194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3672304Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3672310Z 2025-09-07T07:11:54.3672418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3672616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3672681Z return mod(**inputs) 2025-09-07T07:11:54.3672970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3673041Z outputs = self.mobilebert( 2025-09-07T07:11:54.3673333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3673404Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3673687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3673767Z layer_outputs = layer_module( 2025-09-07T07:11:54.3674074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3674173Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3674449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3674577Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3674853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3674937Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3674941Z 2025-09-07T07:11:54.3675048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3675241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3675312Z return mod(**inputs) 2025-09-07T07:11:54.3675587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3675663Z outputs = self.mobilebert( 2025-09-07T07:11:54.3675935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3676006Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3676304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3676389Z layer_outputs = layer_module( 2025-09-07T07:11:54.3676676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3676766Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3677035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3677167Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3677441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3677569Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3677847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3677944Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3677949Z 2025-09-07T07:11:54.3678049Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3678246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3678319Z return mod(**inputs) 2025-09-07T07:11:54.3678595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3678675Z outputs = self.mobilebert( 2025-09-07T07:11:54.3678952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3679022Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3679308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3679379Z layer_outputs = layer_module( 2025-09-07T07:11:54.3679666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3679759Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3680052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3680197Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3680481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3680572Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3680575Z 2025-09-07T07:11:54.3680676Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3680888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3680953Z return mod(**inputs) 2025-09-07T07:11:54.3681238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3681330Z outputs = self.mobilebert( 2025-09-07T07:11:54.3681604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3681685Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3681960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3682037Z layer_outputs = layer_module( 2025-09-07T07:11:54.3682312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3682421Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3682721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3682831Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3683111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3683221Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3683227Z 2025-09-07T07:11:54.3683327Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3683531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3683596Z return mod(**inputs) 2025-09-07T07:11:54.3683874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3683946Z outputs = self.mobilebert( 2025-09-07T07:11:54.3684227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3684300Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3684590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3684667Z layer_outputs = layer_module( 2025-09-07T07:11:54.3684952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3685052Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3685345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3685470Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3685771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3685857Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3685860Z 2025-09-07T07:11:54.3685959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3686160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3686224Z return mod(**inputs) 2025-09-07T07:11:54.3686540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3686608Z outputs = self.mobilebert( 2025-09-07T07:11:54.3686883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3686952Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3687228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3687304Z layer_outputs = layer_module( 2025-09-07T07:11:54.3687579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3687678Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3687954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3688079Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3688358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3688476Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3688772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3688865Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3688889Z 2025-09-07T07:11:54.3688998Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3689189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3689255Z return mod(**inputs) 2025-09-07T07:11:54.3689542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3689616Z outputs = self.mobilebert( 2025-09-07T07:11:54.3689903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3689975Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3690258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3690336Z layer_outputs = layer_module( 2025-09-07T07:11:54.3690619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3690748Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3691028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3691123Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3691126Z 2025-09-07T07:11:54.3691228Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3691427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3691500Z return mod(**inputs) 2025-09-07T07:11:54.3691777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3691857Z outputs = self.mobilebert( 2025-09-07T07:11:54.3692139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3692212Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3692503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3692574Z layer_outputs = layer_module( 2025-09-07T07:11:54.3692902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3693020Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3693311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3693423Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3693426Z 2025-09-07T07:11:54.3693527Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3693731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3693798Z return mod(**inputs) 2025-09-07T07:11:54.3694085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3694160Z outputs = self.mobilebert( 2025-09-07T07:11:54.3694439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3694520Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3694799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3694877Z layer_outputs = layer_module( 2025-09-07T07:11:54.3695170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3695354Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3695638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3695735Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3695740Z 2025-09-07T07:11:54.3695851Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3696050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3696122Z return mod(**inputs) 2025-09-07T07:11:54.3696406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3696477Z outputs = self.mobilebert( 2025-09-07T07:11:54.3696769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3696842Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3697134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3697207Z layer_outputs = layer_module( 2025-09-07T07:11:54.3697501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3697666Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3697949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3698085Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3698373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3698477Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3698480Z 2025-09-07T07:11:54.3698583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3698790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3698856Z return mod(**inputs) 2025-09-07T07:11:54.3699174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3699252Z outputs = self.mobilebert( 2025-09-07T07:11:54.3699536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3699615Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3699901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3699974Z layer_outputs = layer_module( 2025-09-07T07:11:54.3700271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3700429Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3700721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3700848Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3701135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3701225Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3701228Z 2025-09-07T07:11:54.3701358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3701597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3701669Z return mod(**inputs) 2025-09-07T07:11:54.3701982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3702058Z outputs = self.mobilebert( 2025-09-07T07:11:54.3702357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3702444Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3702745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3702828Z layer_outputs = layer_module( 2025-09-07T07:11:54.3703131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3703304Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3703607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3703738Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3704049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3704180Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3704490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3704589Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3704593Z 2025-09-07T07:11:54.3704703Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3704921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3704992Z return mod(**inputs) 2025-09-07T07:11:54.3705298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3705374Z outputs = self.mobilebert( 2025-09-07T07:11:54.3705678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3705867Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3706174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3706261Z layer_outputs = layer_module( 2025-09-07T07:11:54.3706566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3706747Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3707057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3707176Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3707487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3707580Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3707586Z 2025-09-07T07:11:54.3707702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3707914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3707991Z return mod(**inputs) 2025-09-07T07:11:54.3708349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3708427Z outputs = self.mobilebert( 2025-09-07T07:11:54.3708750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3708828Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3709133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3709211Z layer_outputs = layer_module( 2025-09-07T07:11:54.3709507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3709608Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3709906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3709992Z self_outputs = self.self( 2025-09-07T07:11:54.3710298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3710381Z self.value(value_tensor) 2025-09-07T07:11:54.3710385Z 2025-09-07T07:11:54.3710493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3710703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3710781Z return mod(**inputs) 2025-09-07T07:11:54.3711081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3711165Z outputs = self.mobilebert( 2025-09-07T07:11:54.3711485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3711563Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3711900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3711977Z layer_outputs = layer_module( 2025-09-07T07:11:54.3712300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3712474Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3712806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3712959Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3713280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3713379Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3713383Z 2025-09-07T07:11:54.3713494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3713715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3713785Z return mod(**inputs) 2025-09-07T07:11:54.3714105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3714190Z outputs = self.mobilebert( 2025-09-07T07:11:54.3714516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3714602Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3714926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3715009Z layer_outputs = layer_module( 2025-09-07T07:11:54.3715345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3715543Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3715850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3715967Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3716294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3716389Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3716711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3716816Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3716820Z 2025-09-07T07:11:54.3716929Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3717146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3717219Z return mod(**inputs) 2025-09-07T07:11:54.3717547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3717623Z outputs = self.mobilebert( 2025-09-07T07:11:54.3717944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3718035Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3718357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3718441Z layer_outputs = layer_module( 2025-09-07T07:11:54.3718763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3718857Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3719246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3719321Z self_outputs = self.self( 2025-09-07T07:11:54.3719847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3719928Z self.query(query_tensor) 2025-09-07T07:11:54.3719994Z 2025-09-07T07:11:54.3720116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3720367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3720437Z return mod(**inputs) 2025-09-07T07:11:54.3720760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3720837Z outputs = self.mobilebert( 2025-09-07T07:11:54.3721231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3721321Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3721646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3721728Z layer_outputs = layer_module( 2025-09-07T07:11:54.3722049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3722152Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3722451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3722525Z self_outputs = self.self( 2025-09-07T07:11:54.3722859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3722930Z self.key(key_tensor) 2025-09-07T07:11:54.3722957Z 2025-09-07T07:11:54.3723054Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3723143Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3723261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3723470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3723544Z return mod(**inputs) 2025-09-07T07:11:54.3723850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3723926Z outputs = self.mobilebert( 2025-09-07T07:11:54.3724231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3724308Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3724606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3724688Z layer_outputs = layer_module( 2025-09-07T07:11:54.3724985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3725082Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3725386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3725530Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3725833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3725924Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3725928Z 2025-09-07T07:11:54.3726046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3726259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3726335Z return mod(**inputs) 2025-09-07T07:11:54.3726635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3726718Z outputs = self.mobilebert( 2025-09-07T07:11:54.3727007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3727113Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3727402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3727471Z layer_outputs = layer_module( 2025-09-07T07:11:54.3727754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3727848Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3728130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3728264Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3728545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3728685Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3728985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3729083Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3729087Z 2025-09-07T07:11:54.3729201Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3729427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3729519Z return mod(**inputs) 2025-09-07T07:11:54.3729823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3729917Z outputs = self.mobilebert( 2025-09-07T07:11:54.3730202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3730278Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3730565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3730637Z layer_outputs = layer_module( 2025-09-07T07:11:54.3730924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3731022Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3731308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3731430Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3731711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3731809Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3731812Z 2025-09-07T07:11:54.3731914Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3732118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3732185Z return mod(**inputs) 2025-09-07T07:11:54.3732464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3732546Z outputs = self.mobilebert( 2025-09-07T07:11:54.3732826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3732905Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3733185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3733256Z layer_outputs = layer_module( 2025-09-07T07:11:54.3733578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3733674Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3733966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3734079Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3734369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3734483Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3734487Z 2025-09-07T07:11:54.3734588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3734794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3734862Z return mod(**inputs) 2025-09-07T07:11:54.3735150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3735220Z outputs = self.mobilebert( 2025-09-07T07:11:54.3735502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3735583Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3735884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3735976Z layer_outputs = layer_module( 2025-09-07T07:11:54.3736260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3736356Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3736644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3736775Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3737066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3737151Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3737154Z 2025-09-07T07:11:54.3737266Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3737466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3737534Z return mod(**inputs) 2025-09-07T07:11:54.3737825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3737898Z outputs = self.mobilebert( 2025-09-07T07:11:54.3738187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3738264Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3738542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3738621Z layer_outputs = layer_module( 2025-09-07T07:11:54.3738903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3739008Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3739291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3739424Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3739704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3740285Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3740597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3740694Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3740698Z 2025-09-07T07:11:54.3740813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3741025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3741102Z return mod(**inputs) 2025-09-07T07:11:54.3741402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3741478Z outputs = self.mobilebert( 2025-09-07T07:11:54.3741785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3741865Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3742179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3742254Z layer_outputs = layer_module( 2025-09-07T07:11:54.3742552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3742684Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3743010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3743139Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3743441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3743536Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3743543Z 2025-09-07T07:11:54.3743652Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3743864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3743941Z return mod(**inputs) 2025-09-07T07:11:54.3744239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3744321Z outputs = self.mobilebert( 2025-09-07T07:11:54.3744620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3744696Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3745003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3745079Z layer_outputs = layer_module( 2025-09-07T07:11:54.3745384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3745483Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3745849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3745977Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3746281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3746410Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3746414Z 2025-09-07T07:11:54.3746523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3746741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3746812Z return mod(**inputs) 2025-09-07T07:11:54.3747146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3747230Z outputs = self.mobilebert( 2025-09-07T07:11:54.3747527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3747612Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3747913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3747999Z layer_outputs = layer_module( 2025-09-07T07:11:54.3748297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3748398Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3748706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3748842Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3749150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3749240Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3749245Z 2025-09-07T07:11:54.3749371Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3749594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3749679Z return mod(**inputs) 2025-09-07T07:11:54.3749985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3750061Z outputs = self.mobilebert( 2025-09-07T07:11:54.3750370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3750451Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3750746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3750829Z layer_outputs = layer_module( 2025-09-07T07:11:54.3751135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3751243Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3751542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3751677Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3751985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3752119Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3752427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3752531Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3752535Z 2025-09-07T07:11:54.3752650Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3752859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3752929Z return mod(**inputs) 2025-09-07T07:11:54.3753234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3753309Z outputs = self.mobilebert( 2025-09-07T07:11:54.3753623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3753732Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3754018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3754096Z layer_outputs = layer_module( 2025-09-07T07:11:54.3754379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3754485Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3754772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3754891Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3755172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3755257Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3755264Z 2025-09-07T07:11:54.3755376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3755578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3755652Z return mod(**inputs) 2025-09-07T07:11:54.3755935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3756045Z outputs = self.mobilebert( 2025-09-07T07:11:54.3756362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3756437Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3756726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3756797Z layer_outputs = layer_module( 2025-09-07T07:11:54.3757089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3757182Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3757467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3757585Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3757872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3757991Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3757995Z 2025-09-07T07:11:54.3758098Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3758299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3758372Z return mod(**inputs) 2025-09-07T07:11:54.3758658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3758737Z outputs = self.mobilebert( 2025-09-07T07:11:54.3759021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3759102Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3759388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3759460Z layer_outputs = layer_module( 2025-09-07T07:11:54.3759750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3759844Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3760136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3760293Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3760573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3760665Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3760668Z 2025-09-07T07:11:54.3760773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3760977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3761044Z return mod(**inputs) 2025-09-07T07:11:54.3761332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3761404Z outputs = self.mobilebert( 2025-09-07T07:11:54.3761688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3761769Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3762051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3762128Z layer_outputs = layer_module( 2025-09-07T07:11:54.3762423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3762521Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3762829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3762956Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3763262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3763383Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3763674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3763764Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3763767Z 2025-09-07T07:11:54.3763871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3764083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3764151Z return mod(**inputs) 2025-09-07T07:11:54.3764443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3764516Z outputs = self.mobilebert( 2025-09-07T07:11:54.3764802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3764887Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3765173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3765251Z layer_outputs = layer_module( 2025-09-07T07:11:54.3765536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3765667Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3765955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3766043Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3766046Z 2025-09-07T07:11:54.3766159Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3766360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3766475Z return mod(**inputs) 2025-09-07T07:11:54.3766752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3766821Z outputs = self.mobilebert( 2025-09-07T07:11:54.3767104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3767177Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3767464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3767535Z layer_outputs = layer_module( 2025-09-07T07:11:54.3767815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3767931Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3768207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3768324Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3768328Z 2025-09-07T07:11:54.3768427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3768627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3768705Z return mod(**inputs) 2025-09-07T07:11:54.3768999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3769076Z outputs = self.mobilebert( 2025-09-07T07:11:54.3769351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3769431Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3769711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3769791Z layer_outputs = layer_module( 2025-09-07T07:11:54.3770071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3770231Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3770517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3770616Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3770619Z 2025-09-07T07:11:54.3770729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3770929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3770996Z return mod(**inputs) 2025-09-07T07:11:54.3771294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3771365Z outputs = self.mobilebert( 2025-09-07T07:11:54.3771644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3771715Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3771996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3772065Z layer_outputs = layer_module( 2025-09-07T07:11:54.3772338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3772503Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3772777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3772935Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3773210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3773302Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3773312Z 2025-09-07T07:11:54.3773413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3773609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3773682Z return mod(**inputs) 2025-09-07T07:11:54.3773962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3774037Z outputs = self.mobilebert( 2025-09-07T07:11:54.3774316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3774387Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3774673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3774740Z layer_outputs = layer_module( 2025-09-07T07:11:54.3775041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3775197Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3775488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3775618Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3775890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3775984Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3775988Z 2025-09-07T07:11:54.3776088Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3776286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3776349Z return mod(**inputs) 2025-09-07T07:11:54.3776623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3776699Z outputs = self.mobilebert( 2025-09-07T07:11:54.3776975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3777053Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3777325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3777397Z layer_outputs = layer_module( 2025-09-07T07:11:54.3777680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3777834Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3778127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3778253Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3778543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3778668Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3778954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3779089Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3779093Z 2025-09-07T07:11:54.3779196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3779406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3779473Z return mod(**inputs) 2025-09-07T07:11:54.3779763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3779834Z outputs = self.mobilebert( 2025-09-07T07:11:54.3780116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3780200Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3780479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3780561Z layer_outputs = layer_module( 2025-09-07T07:11:54.3780854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3781031Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3781358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3781478Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3781799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3781890Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3781894Z 2025-09-07T07:11:54.3782010Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3782220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3782295Z return mod(**inputs) 2025-09-07T07:11:54.3782602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3782677Z outputs = self.mobilebert( 2025-09-07T07:11:54.3782984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3783063Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3783366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3783449Z layer_outputs = layer_module( 2025-09-07T07:11:54.3783753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3783853Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3784155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3784237Z self_outputs = self.self( 2025-09-07T07:11:54.3784538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3784613Z self.value(value_tensor) 2025-09-07T07:11:54.3784617Z 2025-09-07T07:11:54.3784735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3784947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3785024Z return mod(**inputs) 2025-09-07T07:11:54.3785325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3785400Z outputs = self.mobilebert( 2025-09-07T07:11:54.3785769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3785896Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3786210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3786285Z layer_outputs = layer_module( 2025-09-07T07:11:54.3786595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3786781Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3787087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3787216Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3787521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3787622Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3787627Z 2025-09-07T07:11:54.3787736Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3787948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3788029Z return mod(**inputs) 2025-09-07T07:11:54.3788362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3788467Z outputs = self.mobilebert( 2025-09-07T07:11:54.3788767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3788844Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3789163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3789243Z layer_outputs = layer_module( 2025-09-07T07:11:54.3789549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3789721Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3790029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3790147Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3790447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3790546Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3790843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3790951Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3790954Z 2025-09-07T07:11:54.3791063Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3791285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3791355Z return mod(**inputs) 2025-09-07T07:11:54.3791656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3791743Z outputs = self.mobilebert( 2025-09-07T07:11:54.3792043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3792128Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3792428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3792552Z layer_outputs = layer_module( 2025-09-07T07:11:54.3792862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3792953Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3793262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3793338Z self_outputs = self.self( 2025-09-07T07:11:54.3793646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3793728Z self.query(query_tensor) 2025-09-07T07:11:54.3793732Z 2025-09-07T07:11:54.3793840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3794061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3794130Z return mod(**inputs) 2025-09-07T07:11:54.3794450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3794528Z outputs = self.mobilebert( 2025-09-07T07:11:54.3794854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3794941Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3795269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3795368Z layer_outputs = layer_module( 2025-09-07T07:11:54.3795681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3795773Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3796090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3796168Z self_outputs = self.self( 2025-09-07T07:11:54.3796492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3796561Z self.key(key_tensor) 2025-09-07T07:11:54.3796565Z 2025-09-07T07:11:54.3796657Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3796746Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3796858Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3797077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3797146Z return mod(**inputs) 2025-09-07T07:11:54.3797453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3797529Z outputs = self.mobilebert( 2025-09-07T07:11:54.3797833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3797919Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3798220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3798301Z layer_outputs = layer_module( 2025-09-07T07:11:54.3798601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3798693Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3799003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3799137Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3799444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3799566Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3799570Z 2025-09-07T07:11:54.3799684Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3799893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3799962Z return mod(**inputs) 2025-09-07T07:11:54.3800268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3800341Z outputs = self.mobilebert( 2025-09-07T07:11:54.3800630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3800702Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3800995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3801080Z layer_outputs = layer_module( 2025-09-07T07:11:54.3801381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3801482Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3801779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3801910Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3802224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3802361Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3802672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3802771Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3802775Z 2025-09-07T07:11:54.3802889Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3803105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3803175Z return mod(**inputs) 2025-09-07T07:11:54.3803487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3803564Z outputs = self.mobilebert( 2025-09-07T07:11:54.3803890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3803962Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3804257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3804330Z layer_outputs = layer_module( 2025-09-07T07:11:54.3804629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3804740Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3805044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3805174Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3805479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3805569Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3805580Z 2025-09-07T07:11:54.3805692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3805905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3806014Z return mod(**inputs) 2025-09-07T07:11:54.3806316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3806398Z outputs = self.mobilebert( 2025-09-07T07:11:54.3806699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3806775Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3807088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3807160Z layer_outputs = layer_module( 2025-09-07T07:11:54.3807456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3807556Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3807869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3808000Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3808301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3808429Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3808433Z 2025-09-07T07:11:54.3808563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3808797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3808868Z return mod(**inputs) 2025-09-07T07:11:54.3809169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3809253Z outputs = self.mobilebert( 2025-09-07T07:11:54.3809562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3809648Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3809945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3810019Z layer_outputs = layer_module( 2025-09-07T07:11:54.3810338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3810437Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3810743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3810878Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3811195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3811284Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3811287Z 2025-09-07T07:11:54.3811391Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3811600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3811665Z return mod(**inputs) 2025-09-07T07:11:54.3811979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3812054Z outputs = self.mobilebert( 2025-09-07T07:11:54.3812358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3812442Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3812744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3812890Z layer_outputs = layer_module( 2025-09-07T07:11:54.3813187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3813295Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3813602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3813737Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3814046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3814178Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3814486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3814587Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3814590Z 2025-09-07T07:11:54.3814707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3814919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3814988Z return mod(**inputs) 2025-09-07T07:11:54.3815314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3815390Z outputs = self.mobilebert( 2025-09-07T07:11:54.3815723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3815804Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3816106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3816192Z layer_outputs = layer_module( 2025-09-07T07:11:54.3816494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3816602Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3816913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3817034Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3817343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3817433Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3817437Z 2025-09-07T07:11:54.3817555Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3817764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3817845Z return mod(**inputs) 2025-09-07T07:11:54.3818143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3818219Z outputs = self.mobilebert( 2025-09-07T07:11:54.3818524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3818601Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3818911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3818992Z layer_outputs = layer_module( 2025-09-07T07:11:54.3819290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3819395Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3819875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3820073Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3820377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3820505Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3820509Z 2025-09-07T07:11:54.3820624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3820841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3820919Z return mod(**inputs) 2025-09-07T07:11:54.3821220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3821304Z outputs = self.mobilebert( 2025-09-07T07:11:54.3821607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3821688Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3822002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3822077Z layer_outputs = layer_module( 2025-09-07T07:11:54.3822417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3822518Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3822854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3822989Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3823286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3823389Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3823393Z 2025-09-07T07:11:54.3823502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3823726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3823797Z return mod(**inputs) 2025-09-07T07:11:54.3824106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3824192Z outputs = self.mobilebert( 2025-09-07T07:11:54.3824499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3824586Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3824893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3824981Z layer_outputs = layer_module( 2025-09-07T07:11:54.3825290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3825400Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3825752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3825903Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3826219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3826351Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3826652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3826792Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3826796Z 2025-09-07T07:11:54.3826904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3827123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3827193Z return mod(**inputs) 2025-09-07T07:11:54.3827509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3827584Z outputs = self.mobilebert( 2025-09-07T07:11:54.3827884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3827970Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3828271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3828356Z layer_outputs = layer_module( 2025-09-07T07:11:54.3828659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3828760Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3829067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3829205Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3829529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3829618Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3829622Z 2025-09-07T07:11:54.3829738Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3829948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3830016Z return mod(**inputs) 2025-09-07T07:11:54.3830301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3830370Z outputs = self.mobilebert( 2025-09-07T07:11:54.3830654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3830725Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3831003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3831082Z layer_outputs = layer_module( 2025-09-07T07:11:54.3831356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3831459Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3831733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3831852Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3832128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3832237Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3832240Z 2025-09-07T07:11:54.3832350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3832549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3832620Z return mod(**inputs) 2025-09-07T07:11:54.3832892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3832962Z outputs = self.mobilebert( 2025-09-07T07:11:54.3833246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3833351Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3833635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3833704Z layer_outputs = layer_module( 2025-09-07T07:11:54.3833991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3834084Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3834361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3834492Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3834766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3834858Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3834862Z 2025-09-07T07:11:54.3834962Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3835164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3835229Z return mod(**inputs) 2025-09-07T07:11:54.3835519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3835600Z outputs = self.mobilebert( 2025-09-07T07:11:54.3835895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3835974Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3836250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3836324Z layer_outputs = layer_module( 2025-09-07T07:11:54.3836610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3836702Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3836988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3837113Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3837394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3837524Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3837807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3837912Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3837915Z 2025-09-07T07:11:54.3838019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3838231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3838296Z return mod(**inputs) 2025-09-07T07:11:54.3838585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3838663Z outputs = self.mobilebert( 2025-09-07T07:11:54.3838958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3839037Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3839310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3839380Z layer_outputs = layer_module( 2025-09-07T07:11:54.3839694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3839812Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3840097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3840180Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3840185Z 2025-09-07T07:11:54.3840294Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3840489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3840553Z return mod(**inputs) 2025-09-07T07:11:54.3840832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3840902Z outputs = self.mobilebert( 2025-09-07T07:11:54.3841186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3841256Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3841538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3841616Z layer_outputs = layer_module( 2025-09-07T07:11:54.3841916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3842060Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3842349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3842468Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3842473Z 2025-09-07T07:11:54.3842578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3842775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3842849Z return mod(**inputs) 2025-09-07T07:11:54.3843134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3843210Z outputs = self.mobilebert( 2025-09-07T07:11:54.3843499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3843573Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3843879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3843948Z layer_outputs = layer_module( 2025-09-07T07:11:54.3844234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3844394Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3844679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3844774Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3844778Z 2025-09-07T07:11:54.3844879Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3845085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3845149Z return mod(**inputs) 2025-09-07T07:11:54.3845439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3845511Z outputs = self.mobilebert( 2025-09-07T07:11:54.3845799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3845913Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3846198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3846277Z layer_outputs = layer_module( 2025-09-07T07:11:54.3846559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3846727Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3847013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3847138Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3847436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3847528Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3847531Z 2025-09-07T07:11:54.3847638Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3847831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3847901Z return mod(**inputs) 2025-09-07T07:11:54.3848192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3848264Z outputs = self.mobilebert( 2025-09-07T07:11:54.3848573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3848650Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3848939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3849013Z layer_outputs = layer_module( 2025-09-07T07:11:54.3849293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3849459Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3849742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3849874Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3850158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3850251Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3850254Z 2025-09-07T07:11:54.3850357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3850555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3850632Z return mod(**inputs) 2025-09-07T07:11:54.3850912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3850991Z outputs = self.mobilebert( 2025-09-07T07:11:54.3851272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3851346Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3851637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3851708Z layer_outputs = layer_module( 2025-09-07T07:11:54.3852000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3852156Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3852472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3852594Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3852876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3853007Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3853288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3853387Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3853391Z 2025-09-07T07:11:54.3853493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3853691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3853768Z return mod(**inputs) 2025-09-07T07:11:54.3854048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3854128Z outputs = self.mobilebert( 2025-09-07T07:11:54.3854407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3854500Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3854802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3854873Z layer_outputs = layer_module( 2025-09-07T07:11:54.3855170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3855335Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3855632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3855744Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3856031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3856124Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3856128Z 2025-09-07T07:11:54.3856232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3856445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3856511Z return mod(**inputs) 2025-09-07T07:11:54.3856802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3856876Z outputs = self.mobilebert( 2025-09-07T07:11:54.3857161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3857242Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3857531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3857609Z layer_outputs = layer_module( 2025-09-07T07:11:54.3857898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3857989Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3858281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3858352Z self_outputs = self.self( 2025-09-07T07:11:54.3858646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:11:54.3858749Z self.value(value_tensor) 2025-09-07T07:11:54.3858752Z 2025-09-07T07:11:54.3858862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3859068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3859133Z return mod(**inputs) 2025-09-07T07:11:54.3859430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3859503Z outputs = self.mobilebert( 2025-09-07T07:11:54.3859796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3859868Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3860159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3860238Z layer_outputs = layer_module( 2025-09-07T07:11:54.3860530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3860698Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3861011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:11:54.3861133Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:11:54.3861434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:11:54.3861517Z layer_input = self.dense(hidden_states) 2025-09-07T07:11:54.3861521Z 2025-09-07T07:11:54.3861632Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3861834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3861905Z return mod(**inputs) 2025-09-07T07:11:54.3862190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3862262Z outputs = self.mobilebert( 2025-09-07T07:11:54.3862557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3862631Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3862943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3863019Z layer_outputs = layer_module( 2025-09-07T07:11:54.3863335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:11:54.3863500Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:11:54.3863786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:11:54.3863907Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:11:54.3864193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:11:54.3864290Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:11:54.3864574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3864666Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3864675Z 2025-09-07T07:11:54.3864778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3864979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3865092Z return mod(**inputs) 2025-09-07T07:11:54.3865385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3865464Z outputs = self.mobilebert( 2025-09-07T07:11:54.3865833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3865917Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3866211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3866286Z layer_outputs = layer_module( 2025-09-07T07:11:54.3866602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3866696Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3867010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3867091Z self_outputs = self.self( 2025-09-07T07:11:54.3867375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:11:54.3867456Z self.query(query_tensor) 2025-09-07T07:11:54.3867460Z 2025-09-07T07:11:54.3867583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3867808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3867877Z return mod(**inputs) 2025-09-07T07:11:54.3868164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3868251Z outputs = self.mobilebert( 2025-09-07T07:11:54.3868562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3868653Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3868952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3869027Z layer_outputs = layer_module( 2025-09-07T07:11:54.3869338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3869429Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3869737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:11:54.3869813Z self_outputs = self.self( 2025-09-07T07:11:54.3870116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:11:54.3870191Z self.key(key_tensor) 2025-09-07T07:11:54.3870194Z 2025-09-07T07:11:54.3870281Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3870373Z cudagraph partition due to non gpu ops 2025-09-07T07:11:54.3870482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3870698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3870768Z return mod(**inputs) 2025-09-07T07:11:54.3871078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3871163Z outputs = self.mobilebert( 2025-09-07T07:11:54.3871462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3871546Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3871842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3871953Z layer_outputs = layer_module( 2025-09-07T07:11:54.3872263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3872353Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3872659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3872793Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3873098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:11:54.3873189Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3873193Z 2025-09-07T07:11:54.3873304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3873521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3873595Z return mod(**inputs) 2025-09-07T07:11:54.3873901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3873976Z outputs = self.mobilebert( 2025-09-07T07:11:54.3874301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3874389Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3874700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3874782Z layer_outputs = layer_module( 2025-09-07T07:11:54.3875083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:11:54.3875180Z self_attention_outputs = self.attention( 2025-09-07T07:11:54.3875481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:11:54.3875613Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:11:54.3875922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:11:54.3876060Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3876366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3876464Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3876467Z 2025-09-07T07:11:54.3876575Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3876802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3876871Z return mod(**inputs) 2025-09-07T07:11:54.3877149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3877218Z outputs = self.mobilebert( 2025-09-07T07:11:54.3877498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3877572Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3877848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3877922Z layer_outputs = layer_module( 2025-09-07T07:11:54.3878196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3878297Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3878607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3878720Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3879009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3879095Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3879100Z 2025-09-07T07:11:54.3879211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3879409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3879482Z return mod(**inputs) 2025-09-07T07:11:54.3879761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3879831Z outputs = self.mobilebert( 2025-09-07T07:11:54.3880125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3880199Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3880500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3880575Z layer_outputs = layer_module( 2025-09-07T07:11:54.3880889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3881029Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3881312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3881432Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3881716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3881842Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3881845Z 2025-09-07T07:11:54.3881950Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3882150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3882224Z return mod(**inputs) 2025-09-07T07:11:54.3882509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3882589Z outputs = self.mobilebert( 2025-09-07T07:11:54.3882882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3882954Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3883257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3883331Z layer_outputs = layer_module( 2025-09-07T07:11:54.3883623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3883718Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3884009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3884139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3884423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3884518Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3884522Z 2025-09-07T07:11:54.3884624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3884831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3884941Z return mod(**inputs) 2025-09-07T07:11:54.3885233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3885313Z outputs = self.mobilebert( 2025-09-07T07:11:54.3885606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3885687Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3885979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3886069Z layer_outputs = layer_module( 2025-09-07T07:11:54.3886354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3886447Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3886749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3886877Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3887174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3887314Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3887611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3887713Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3887717Z 2025-09-07T07:11:54.3887821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3888028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3888097Z return mod(**inputs) 2025-09-07T07:11:54.3888385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3888457Z outputs = self.mobilebert( 2025-09-07T07:11:54.3888738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3888821Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3889106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3889185Z layer_outputs = layer_module( 2025-09-07T07:11:54.3889469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3889563Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3889857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3889971Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3890261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3890350Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3890353Z 2025-09-07T07:11:54.3890469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3890681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3890752Z return mod(**inputs) 2025-09-07T07:11:54.3891054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3891128Z outputs = self.mobilebert( 2025-09-07T07:11:54.3891429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3891552Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3891852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3891935Z layer_outputs = layer_module( 2025-09-07T07:11:54.3892230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3892328Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3892605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3892723Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3892998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3893111Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3893114Z 2025-09-07T07:11:54.3893219Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3893411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3893482Z return mod(**inputs) 2025-09-07T07:11:54.3893771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3893844Z outputs = self.mobilebert( 2025-09-07T07:11:54.3894152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3894227Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3894518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3894591Z layer_outputs = layer_module( 2025-09-07T07:11:54.3894883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3894980Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3895264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3895399Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3895684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3895776Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3895779Z 2025-09-07T07:11:54.3895882Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3896082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3896159Z return mod(**inputs) 2025-09-07T07:11:54.3896442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3896523Z outputs = self.mobilebert( 2025-09-07T07:11:54.3896807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3896887Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3897175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3897246Z layer_outputs = layer_module( 2025-09-07T07:11:54.3897536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3897630Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3897950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3898076Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3898358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3898490Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3898777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3898878Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3898882Z 2025-09-07T07:11:54.3898986Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3899194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3899262Z return mod(**inputs) 2025-09-07T07:11:54.3899561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3899645Z outputs = self.mobilebert( 2025-09-07T07:11:54.3899939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3900043Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3900355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3900432Z layer_outputs = layer_module( 2025-09-07T07:11:54.3900739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3900839Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3901144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3901268Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3901573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3901663Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3901666Z 2025-09-07T07:11:54.3901776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3901998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3902068Z return mod(**inputs) 2025-09-07T07:11:54.3902371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3902448Z outputs = self.mobilebert( 2025-09-07T07:11:54.3902746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3902834Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3903134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3903215Z layer_outputs = layer_module( 2025-09-07T07:11:54.3903517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3903622Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3903927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:11:54.3904046Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:11:54.3904351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3904502Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3904506Z 2025-09-07T07:11:54.3904622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3904834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3904905Z return mod(**inputs) 2025-09-07T07:11:54.3905212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3905289Z outputs = self.mobilebert( 2025-09-07T07:11:54.3905597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3905673Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3906048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3906134Z layer_outputs = layer_module( 2025-09-07T07:11:54.3906432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3906542Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3906839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3907002Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3907316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:11:54.3907410Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3907423Z 2025-09-07T07:11:54.3907534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3907747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3907828Z return mod(**inputs) 2025-09-07T07:11:54.3908135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3908217Z outputs = self.mobilebert( 2025-09-07T07:11:54.3908524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3908600Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3908913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3908988Z layer_outputs = layer_module( 2025-09-07T07:11:54.3909301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:11:54.3909399Z attention_output = ffn_module(attention_output) 2025-09-07T07:11:54.3909707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:11:54.3909847Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:11:54.3910161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:11:54.3910299Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3910608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3910712Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3910715Z 2025-09-07T07:11:54.3910822Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3911039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3911157Z return mod(**inputs) 2025-09-07T07:11:54.3911456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3911538Z outputs = self.mobilebert( 2025-09-07T07:11:54.3911840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3911917Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3912227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3912303Z layer_outputs = layer_module( 2025-09-07T07:11:54.3912613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3912744Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3913054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:11:54.3913148Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3913152Z 2025-09-07T07:11:54.3913261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3913483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3913553Z return mod(**inputs) 2025-09-07T07:11:54.3913880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3913982Z outputs = self.mobilebert( 2025-09-07T07:11:54.3914292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3914378Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3914696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3914781Z layer_outputs = layer_module( 2025-09-07T07:11:54.3915082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:11:54.3915216Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:11:54.3915526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:11:54.3915645Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:11:54.3915650Z 2025-09-07T07:11:54.3915769Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3915982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3916057Z return mod(**inputs) 2025-09-07T07:11:54.3916365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3916446Z outputs = self.mobilebert( 2025-09-07T07:11:54.3916754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3916832Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3917142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3917217Z layer_outputs = layer_module( 2025-09-07T07:11:54.3917527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3917696Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3917998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:11:54.3918167Z layer_output = self.dense(intermediate_states) 2025-09-07T07:11:54.3918171Z 2025-09-07T07:11:54.3918284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3918507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3918588Z return mod(**inputs) 2025-09-07T07:11:54.3918900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3918984Z outputs = self.mobilebert( 2025-09-07T07:11:54.3919283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3919368Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3919843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3919938Z layer_outputs = layer_module( 2025-09-07T07:11:54.3920241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3920413Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3920771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:11:54.3920907Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:11:54.3921236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3921337Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3921341Z 2025-09-07T07:11:54.3921457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3921669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3921744Z return mod(**inputs) 2025-09-07T07:11:54.3922049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3922125Z outputs = self.mobilebert( 2025-09-07T07:11:54.3922431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3922510Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3922810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3922893Z layer_outputs = layer_module( 2025-09-07T07:11:54.3923190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3923365Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3923666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3923797Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3924104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:11:54.3924198Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:11:54.3924202Z 2025-09-07T07:11:54.3924322Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3924532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3924611Z return mod(**inputs) 2025-09-07T07:11:54.3924908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-09-07T07:11:54.3925029Z outputs = self.mobilebert( 2025-09-07T07:11:54.3925343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:11:54.3925421Z encoder_outputs = self.encoder( 2025-09-07T07:11:54.3925735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:11:54.3925811Z layer_outputs = layer_module( 2025-09-07T07:11:54.3926122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:11:54.3926300Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:11:54.3926607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:11:54.3926748Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:11:54.3927060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:11:54.3927196Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:11:54.3927505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:11:54.3927624Z return input_tensor * self.weight + self.bias 2025-09-07T07:11:54.3927628Z 2025-09-07T07:11:54.3927746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3927973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3928051Z return mod(**inputs) 2025-09-07T07:11:54.3928352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-09-07T07:11:54.3928464Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:11:54.3928765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-09-07T07:11:54.3928885Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:11:54.3929196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-09-07T07:11:54.3929295Z hidden_states = self.transform(hidden_states) 2025-09-07T07:11:54.3929606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-09-07T07:11:54.3929696Z hidden_states = self.dense(hidden_states) 2025-09-07T07:11:54.3929700Z 2025-09-07T07:11:54.3929810Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3930030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3930103Z return mod(**inputs) 2025-09-07T07:11:54.3930416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-09-07T07:11:54.3930508Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:11:54.3930809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-09-07T07:11:54.3930922Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:11:54.3931222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-09-07T07:11:54.3931455Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-09-07T07:11:54.3931459Z 2025-09-07T07:11:54.3931568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3931796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3931896Z return mod(**inputs) 2025-09-07T07:11:54.3932190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-09-07T07:11:54.3932290Z prediction_scores = self.cls(sequence_output) 2025-09-07T07:11:54.3932582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-09-07T07:11:54.3932703Z prediction_scores = self.predictions(sequence_output) 2025-09-07T07:11:54.3932999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-09-07T07:11:54.3933089Z hidden_states += self.decoder.bias 2025-09-07T07:11:54.3933093Z 2025-09-07T07:11:54.3933195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:11:54.3933399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:11:54.3933478Z return mod(**inputs) 2025-09-07T07:11:54.3933772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-09-07T07:11:54.3933979Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:11:54.3933982Z 2025-09-07T07:12:10.0885917Z Compilation time (from dynamo_timed): 43.202686331 2025-09-07T07:12:10.0886266Z pass 2025-09-07T07:12:10.0886659Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:12:10.0887524Z TIMING: _recursive_pre_grad_passes:0.02558 _recursive_joint_graph_passes:1.4649 _recursive_post_grad_passes:0.23614 async_compile.wait:0.77248 code_gen:12.39826 inductor_compile:17.39704 backend_compile:30.72188 gc:0.00119 entire_frame_compile:43.20269 total_wall_time:43.20269 2025-09-07T07:12:10.0888526Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:56770 | FakeTensor.__torch_dispatch__:15340 | ProxyTorchDispatchMode.__torch_dispatch__:21632 2025-09-07T07:12:10.0889096Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-09-07T07:12:13.7420536Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:12:13.7421462Z import pynvml # type: ignore[import] 2025-09-07T07:12:16.5433355Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:12:16.5434368Z from pkg_resources import resource_filename 2025-09-07T07:12:17.2030592Z 2025-09-07T07:12:17.6199409Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:12:17.6199891Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:12:17.6266115Z cpu eval MobileBertForQuestionAnswering 2025-09-07T07:12:17.8386934Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:12:17.9762269Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:12:18.1120116Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:12:45.3935373Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3935851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3936222Z return mod(**inputs) 2025-09-07T07:12:45.3936660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3937448Z outputs = self.mobilebert( 2025-09-07T07:12:45.3937874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-09-07T07:12:45.3938320Z embedding_output = self.embeddings( 2025-09-07T07:12:45.3938756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-09-07T07:12:45.3939185Z inputs_embeds = torch.cat( 2025-09-07T07:12:45.3939315Z 2025-09-07T07:12:45.3939436Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3939870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3940241Z return mod(**inputs) 2025-09-07T07:12:45.3940676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3941148Z outputs = self.mobilebert( 2025-09-07T07:12:45.3941592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-09-07T07:12:45.3942059Z embedding_output = self.embeddings( 2025-09-07T07:12:45.3942518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-09-07T07:12:45.3943069Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-09-07T07:12:45.3943279Z 2025-09-07T07:12:45.3943437Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3943853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3944221Z return mod(**inputs) 2025-09-07T07:12:45.3944664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3945135Z outputs = self.mobilebert( 2025-09-07T07:12:45.3945580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-09-07T07:12:45.3946311Z embedding_output = self.embeddings( 2025-09-07T07:12:45.3946788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-09-07T07:12:45.3947244Z embeddings = self.LayerNorm(embeddings) 2025-09-07T07:12:45.3947683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.3948142Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.3948293Z 2025-09-07T07:12:45.3948407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3948768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3949092Z return mod(**inputs) 2025-09-07T07:12:45.3949494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3949909Z outputs = self.mobilebert( 2025-09-07T07:12:45.3950312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3950735Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3951155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3951578Z layer_outputs = layer_module( 2025-09-07T07:12:45.3951999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.3952514Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.3953084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.3953541Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.3953990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.3954422Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.3954575Z 2025-09-07T07:12:45.3954687Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3955040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3955366Z return mod(**inputs) 2025-09-07T07:12:45.3955761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3956182Z outputs = self.mobilebert( 2025-09-07T07:12:45.3956798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3957221Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3957643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3958070Z layer_outputs = layer_module( 2025-09-07T07:12:45.3958512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.3959035Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.3959553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.3960024Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.3960492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.3960941Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.3961381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.3961830Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.3961988Z 2025-09-07T07:12:45.3962099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3962477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3962815Z return mod(**inputs) 2025-09-07T07:12:45.3963223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3963647Z outputs = self.mobilebert( 2025-09-07T07:12:45.3964069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3964503Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3964928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3965351Z layer_outputs = layer_module( 2025-09-07T07:12:45.3965780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.3966222Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.3966664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.3967095Z self_outputs = self.self( 2025-09-07T07:12:45.3967507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.3967970Z self.query(query_tensor) 2025-09-07T07:12:45.3968095Z 2025-09-07T07:12:45.3968201Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3968573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3968904Z return mod(**inputs) 2025-09-07T07:12:45.3969302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3969727Z outputs = self.mobilebert( 2025-09-07T07:12:45.3970137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3970562Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3970970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3971392Z layer_outputs = layer_module( 2025-09-07T07:12:45.3971816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.3972309Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.3972792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.3973209Z self_outputs = self.self( 2025-09-07T07:12:45.3973656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.3974132Z self.key(key_tensor) 2025-09-07T07:12:45.3974253Z 2025-09-07T07:12:45.3974378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3974784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3975143Z return mod(**inputs) 2025-09-07T07:12:45.3975587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3976035Z outputs = self.mobilebert( 2025-09-07T07:12:45.3976450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3976871Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3977284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3977722Z layer_outputs = layer_module( 2025-09-07T07:12:45.3978173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.3978642Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.3979099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.3979562Z self_outputs = self.self( 2025-09-07T07:12:45.3980001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.3980456Z self.value(value_tensor) 2025-09-07T07:12:45.3980582Z 2025-09-07T07:12:45.3980680Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.3980912Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.3981174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3981569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3981929Z return mod(**inputs) 2025-09-07T07:12:45.3982349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3982821Z outputs = self.mobilebert( 2025-09-07T07:12:45.3983267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3983810Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3984267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3984719Z layer_outputs = layer_module( 2025-09-07T07:12:45.3985198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.3985686Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.3986253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.3986784Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.3987296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.3987767Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.3987928Z 2025-09-07T07:12:45.3988035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3988402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3988724Z return mod(**inputs) 2025-09-07T07:12:45.3989143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3989569Z outputs = self.mobilebert( 2025-09-07T07:12:45.3990004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3990431Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3990844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3991294Z layer_outputs = layer_module( 2025-09-07T07:12:45.3991733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.3992277Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.3992832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.3993321Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.3993819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.3994280Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.3994429Z 2025-09-07T07:12:45.3994551Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.3994943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.3995291Z return mod(**inputs) 2025-09-07T07:12:45.3995713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.3996157Z outputs = self.mobilebert( 2025-09-07T07:12:45.3996588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.3997047Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.3997490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.3997933Z layer_outputs = layer_module( 2025-09-07T07:12:45.3998374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.3998912Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.3999439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.3999944Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4000441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4000953Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4001463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4001929Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4002099Z 2025-09-07T07:12:45.4002215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4002607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4002969Z return mod(**inputs) 2025-09-07T07:12:45.4003401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4003848Z outputs = self.mobilebert( 2025-09-07T07:12:45.4004282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4004730Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4005229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4005702Z layer_outputs = layer_module( 2025-09-07T07:12:45.4006139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4006610Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4007083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4007587Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4008073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4008555Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4008718Z 2025-09-07T07:12:45.4008834Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4009227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4009579Z return mod(**inputs) 2025-09-07T07:12:45.4009997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4010445Z outputs = self.mobilebert( 2025-09-07T07:12:45.4010880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4011335Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4011783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4012202Z layer_outputs = layer_module( 2025-09-07T07:12:45.4012624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4013082Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4013523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4013978Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4014425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4014925Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4015100Z 2025-09-07T07:12:45.4015208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4015574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4015910Z return mod(**inputs) 2025-09-07T07:12:45.4016343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4016787Z outputs = self.mobilebert( 2025-09-07T07:12:45.4017220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4017668Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4018106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4018534Z layer_outputs = layer_module( 2025-09-07T07:12:45.4018950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4019404Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4020066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4020673Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4021223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4021699Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4021854Z 2025-09-07T07:12:45.4021975Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4022378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4022747Z return mod(**inputs) 2025-09-07T07:12:45.4023199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4023659Z outputs = self.mobilebert( 2025-09-07T07:12:45.4024101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4024551Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4025030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4025497Z layer_outputs = layer_module( 2025-09-07T07:12:45.4026013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4026520Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4027010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4027510Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4027971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4028432Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4028892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4029318Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4029476Z 2025-09-07T07:12:45.4029579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4029940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4030354Z return mod(**inputs) 2025-09-07T07:12:45.4030745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4031152Z outputs = self.mobilebert( 2025-09-07T07:12:45.4031555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4031977Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4032390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4032801Z layer_outputs = layer_module( 2025-09-07T07:12:45.4033209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4033715Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4034153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4034606Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4035050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4035478Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4035627Z 2025-09-07T07:12:45.4035752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4036137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4036503Z return mod(**inputs) 2025-09-07T07:12:45.4036914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4037428Z outputs = self.mobilebert( 2025-09-07T07:12:45.4037826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4038279Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4038730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4039177Z layer_outputs = layer_module( 2025-09-07T07:12:45.4039628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4040096Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4040566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4041044Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4041534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4042032Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4042219Z 2025-09-07T07:12:45.4042332Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4042727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4043071Z return mod(**inputs) 2025-09-07T07:12:45.4043502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4043949Z outputs = self.mobilebert( 2025-09-07T07:12:45.4044384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4044831Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4045267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4045760Z layer_outputs = layer_module( 2025-09-07T07:12:45.4046199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4046678Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4047151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4047648Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4048151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4048615Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4048767Z 2025-09-07T07:12:45.4048886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4049277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4049607Z return mod(**inputs) 2025-09-07T07:12:45.4049996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4050413Z outputs = self.mobilebert( 2025-09-07T07:12:45.4050812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4051231Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4051659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4052072Z layer_outputs = layer_module( 2025-09-07T07:12:45.4052471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4052906Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4053336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4053800Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4054262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4054727Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4055191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4055656Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4055827Z 2025-09-07T07:12:45.4055943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4056331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4056684Z return mod(**inputs) 2025-09-07T07:12:45.4057119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4057585Z outputs = self.mobilebert( 2025-09-07T07:12:45.4058028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4058492Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4058947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4059389Z layer_outputs = layer_module( 2025-09-07T07:12:45.4059824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4060296Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4060766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4062266Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4062773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4063256Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4063425Z 2025-09-07T07:12:45.4063545Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4063965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4064338Z return mod(**inputs) 2025-09-07T07:12:45.4064774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4065239Z outputs = self.mobilebert( 2025-09-07T07:12:45.4065686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4066254Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4066723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4067177Z layer_outputs = layer_module( 2025-09-07T07:12:45.4067669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4068157Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4068664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4069174Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4069665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4070172Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4070363Z 2025-09-07T07:12:45.4070481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4070884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4071252Z return mod(**inputs) 2025-09-07T07:12:45.4071689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4072162Z outputs = self.mobilebert( 2025-09-07T07:12:45.4072609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4073075Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4073518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4073985Z layer_outputs = layer_module( 2025-09-07T07:12:45.4074434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4074913Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4075404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4075920Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4076442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4076915Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4077070Z 2025-09-07T07:12:45.4077195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4077597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4077992Z return mod(**inputs) 2025-09-07T07:12:45.4078438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4078921Z outputs = self.mobilebert( 2025-09-07T07:12:45.4079364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4079812Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4080265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4080729Z layer_outputs = layer_module( 2025-09-07T07:12:45.4081176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4081659Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4082156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4082693Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4083231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4083838Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4084404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4084917Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4085096Z 2025-09-07T07:12:45.4085210Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4085629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4086005Z return mod(**inputs) 2025-09-07T07:12:45.4086457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4087005Z outputs = self.mobilebert( 2025-09-07T07:12:45.4087520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4088014Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4088477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4088931Z layer_outputs = layer_module( 2025-09-07T07:12:45.4089344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4089818Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4090293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4090754Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4090906Z 2025-09-07T07:12:45.4091019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4091408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4091740Z return mod(**inputs) 2025-09-07T07:12:45.4092145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4092598Z outputs = self.mobilebert( 2025-09-07T07:12:45.4093031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4093487Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4093936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4094385Z layer_outputs = layer_module( 2025-09-07T07:12:45.4094804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4095293Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4095800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4096292Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4096470Z 2025-09-07T07:12:45.4096593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4096987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4097355Z return mod(**inputs) 2025-09-07T07:12:45.4097788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4098264Z outputs = self.mobilebert( 2025-09-07T07:12:45.4098708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4099161Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4099642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4100090Z layer_outputs = layer_module( 2025-09-07T07:12:45.4100545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4101084Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4101616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4102096Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4102264Z 2025-09-07T07:12:45.4102377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4102763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4103136Z return mod(**inputs) 2025-09-07T07:12:45.4103555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4104002Z outputs = self.mobilebert( 2025-09-07T07:12:45.4104437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4104882Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4105336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4105882Z layer_outputs = layer_module( 2025-09-07T07:12:45.4106346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4106909Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4107460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4107953Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4108453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4108943Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4109117Z 2025-09-07T07:12:45.4109238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4109641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4110036Z return mod(**inputs) 2025-09-07T07:12:45.4110471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4110937Z outputs = self.mobilebert( 2025-09-07T07:12:45.4111382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4111850Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4112307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4112766Z layer_outputs = layer_module( 2025-09-07T07:12:45.4113215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4113774Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4114333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4114841Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4115355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4115849Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4116006Z 2025-09-07T07:12:45.4116131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4116561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4116919Z return mod(**inputs) 2025-09-07T07:12:45.4117360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4117825Z outputs = self.mobilebert( 2025-09-07T07:12:45.4118274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4118728Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4119183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4119828Z layer_outputs = layer_module( 2025-09-07T07:12:45.4120293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4120860Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4121393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4121907Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4122419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4122936Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4123445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4123923Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4124094Z 2025-09-07T07:12:45.4124211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4124599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4124950Z return mod(**inputs) 2025-09-07T07:12:45.4125377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4125825Z outputs = self.mobilebert( 2025-09-07T07:12:45.4126339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4126803Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4127259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4127726Z layer_outputs = layer_module( 2025-09-07T07:12:45.4128177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4128746Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4129299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4129801Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4130288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4130738Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4130895Z 2025-09-07T07:12:45.4131011Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4131401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4131782Z return mod(**inputs) 2025-09-07T07:12:45.4132228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4132679Z outputs = self.mobilebert( 2025-09-07T07:12:45.4133114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4133559Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4134000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4134439Z layer_outputs = layer_module( 2025-09-07T07:12:45.4134888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4135424Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4135978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4136471Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4136945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4137406Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4137863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4138341Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4138500Z 2025-09-07T07:12:45.4138617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4138995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4139343Z return mod(**inputs) 2025-09-07T07:12:45.4139766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4140216Z outputs = self.mobilebert( 2025-09-07T07:12:45.4140642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4141096Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4141553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4142034Z layer_outputs = layer_module( 2025-09-07T07:12:45.4142468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4142924Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4143398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4143866Z self_outputs = self.self( 2025-09-07T07:12:45.4144322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4144782Z self.query(query_tensor) 2025-09-07T07:12:45.4144910Z 2025-09-07T07:12:45.4145026Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4145428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4145865Z return mod(**inputs) 2025-09-07T07:12:45.4146315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4146829Z outputs = self.mobilebert( 2025-09-07T07:12:45.4147276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4147764Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4148246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4148709Z layer_outputs = layer_module( 2025-09-07T07:12:45.4149156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4149647Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4150132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4150602Z self_outputs = self.self( 2025-09-07T07:12:45.4151051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4151511Z self.key(key_tensor) 2025-09-07T07:12:45.4151637Z 2025-09-07T07:12:45.4151755Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4152160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4152536Z return mod(**inputs) 2025-09-07T07:12:45.4152974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4153441Z outputs = self.mobilebert( 2025-09-07T07:12:45.4153887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4154360Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4154809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4155228Z layer_outputs = layer_module( 2025-09-07T07:12:45.4155640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4156075Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4156510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4156931Z self_outputs = self.self( 2025-09-07T07:12:45.4157336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4157829Z self.value(value_tensor) 2025-09-07T07:12:45.4157961Z 2025-09-07T07:12:45.4158050Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4158291Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4158544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4158935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4159296Z return mod(**inputs) 2025-09-07T07:12:45.4159702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4160133Z outputs = self.mobilebert( 2025-09-07T07:12:45.4160537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4160959Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4161380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4161807Z layer_outputs = layer_module( 2025-09-07T07:12:45.4162225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4162681Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4163159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4163662Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4164183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4164643Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4164805Z 2025-09-07T07:12:45.4164917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4165308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4165661Z return mod(**inputs) 2025-09-07T07:12:45.4166097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4166565Z outputs = self.mobilebert( 2025-09-07T07:12:45.4167034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4167503Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4167959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4168414Z layer_outputs = layer_module( 2025-09-07T07:12:45.4168844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4169384Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4169937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4170427Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4170912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4171375Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4171531Z 2025-09-07T07:12:45.4171645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4172035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4172437Z return mod(**inputs) 2025-09-07T07:12:45.4172864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4173363Z outputs = self.mobilebert( 2025-09-07T07:12:45.4173798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4174262Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4174704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4175164Z layer_outputs = layer_module( 2025-09-07T07:12:45.4175612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4176083Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4176546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4177068Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4177599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4178113Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4178615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4179088Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4179267Z 2025-09-07T07:12:45.4179389Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4179797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4180159Z return mod(**inputs) 2025-09-07T07:12:45.4180594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4181067Z outputs = self.mobilebert( 2025-09-07T07:12:45.4181516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4181962Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4182405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4182855Z layer_outputs = layer_module( 2025-09-07T07:12:45.4183309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4183791Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4184281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4184793Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4185304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4185882Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4186045Z 2025-09-07T07:12:45.4186163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4186577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4186943Z return mod(**inputs) 2025-09-07T07:12:45.4187387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4187859Z outputs = self.mobilebert( 2025-09-07T07:12:45.4188310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4188770Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4189233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4189736Z layer_outputs = layer_module( 2025-09-07T07:12:45.4190184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4190669Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4191156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4191663Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4192162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4192669Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4192855Z 2025-09-07T07:12:45.4192972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4193374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4193741Z return mod(**inputs) 2025-09-07T07:12:45.4194187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4194640Z outputs = self.mobilebert( 2025-09-07T07:12:45.4195113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4195580Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4196073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4196525Z layer_outputs = layer_module( 2025-09-07T07:12:45.4196963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4197449Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4197938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4198480Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4199004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4199490Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4199655Z 2025-09-07T07:12:45.4199770Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4200169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4200534Z return mod(**inputs) 2025-09-07T07:12:45.4200972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4201430Z outputs = self.mobilebert( 2025-09-07T07:12:45.4201876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4202349Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4202809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4203266Z layer_outputs = layer_module( 2025-09-07T07:12:45.4203699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4204178Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4204655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4205187Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4205737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4206268Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4206780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4207281Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4207446Z 2025-09-07T07:12:45.4207568Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4207966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4208333Z return mod(**inputs) 2025-09-07T07:12:45.4208775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4209246Z outputs = self.mobilebert( 2025-09-07T07:12:45.4209699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4210151Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4210615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4211072Z layer_outputs = layer_module( 2025-09-07T07:12:45.4211543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4212050Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4212534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4213049Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4213551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4214033Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4214194Z 2025-09-07T07:12:45.4214321Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4214720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4215087Z return mod(**inputs) 2025-09-07T07:12:45.4215529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4215996Z outputs = self.mobilebert( 2025-09-07T07:12:45.4216443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4216904Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4217359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4217822Z layer_outputs = layer_module( 2025-09-07T07:12:45.4218281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4218743Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4219216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4219863Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4220370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4220875Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4221059Z 2025-09-07T07:12:45.4221176Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4221580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4222023Z return mod(**inputs) 2025-09-07T07:12:45.4222463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4222914Z outputs = self.mobilebert( 2025-09-07T07:12:45.4223367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4223832Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4224291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4224753Z layer_outputs = layer_module( 2025-09-07T07:12:45.4225195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4225745Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4226248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4226782Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4227300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4227828Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4227998Z 2025-09-07T07:12:45.4228117Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4228550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4228920Z return mod(**inputs) 2025-09-07T07:12:45.4229372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4229849Z outputs = self.mobilebert( 2025-09-07T07:12:45.4230280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4230694Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4231108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4231514Z layer_outputs = layer_module( 2025-09-07T07:12:45.4231923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4232361Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4232808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4233285Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4233760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4234261Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4234724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4235167Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4235322Z 2025-09-07T07:12:45.4235434Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4235796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4236136Z return mod(**inputs) 2025-09-07T07:12:45.4236569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4237016Z outputs = self.mobilebert( 2025-09-07T07:12:45.4237457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4237876Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4238331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4238788Z layer_outputs = layer_module( 2025-09-07T07:12:45.4239229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4239696Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4240159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4240624Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4241088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4241524Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4241667Z 2025-09-07T07:12:45.4241774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4242146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4242477Z return mod(**inputs) 2025-09-07T07:12:45.4242895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4243318Z outputs = self.mobilebert( 2025-09-07T07:12:45.4243738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4244162Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4244575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4244998Z layer_outputs = layer_module( 2025-09-07T07:12:45.4245409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4245845Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4246291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4246758Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4247225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4247723Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4247909Z 2025-09-07T07:12:45.4248017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4248387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4248724Z return mod(**inputs) 2025-09-07T07:12:45.4249129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4249559Z outputs = self.mobilebert( 2025-09-07T07:12:45.4249973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4250411Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4250832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4251256Z layer_outputs = layer_module( 2025-09-07T07:12:45.4251670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4252116Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4252602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4253079Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4253552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4253981Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4254132Z 2025-09-07T07:12:45.4254238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4254609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4254952Z return mod(**inputs) 2025-09-07T07:12:45.4255351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4255777Z outputs = self.mobilebert( 2025-09-07T07:12:45.4256194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4256624Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4257040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4257454Z layer_outputs = layer_module( 2025-09-07T07:12:45.4257905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4258371Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4258818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4259295Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4259767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4260243Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4260723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4261178Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4261330Z 2025-09-07T07:12:45.4261442Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4261804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4262140Z return mod(**inputs) 2025-09-07T07:12:45.4262547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4262977Z outputs = self.mobilebert( 2025-09-07T07:12:45.4263386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4263811Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4264227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4264671Z layer_outputs = layer_module( 2025-09-07T07:12:45.4265113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4265613Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4266221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4266702Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4266859Z 2025-09-07T07:12:45.4266984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4267412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4267741Z return mod(**inputs) 2025-09-07T07:12:45.4268154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4268630Z outputs = self.mobilebert( 2025-09-07T07:12:45.4269081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4269555Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4270016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4270490Z layer_outputs = layer_module( 2025-09-07T07:12:45.4270939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4271465Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4271975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4272491Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4272683Z 2025-09-07T07:12:45.4272801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4273219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4273578Z return mod(**inputs) 2025-09-07T07:12:45.4274033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4274509Z outputs = self.mobilebert( 2025-09-07T07:12:45.4274954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4275420Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4275839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4276263Z layer_outputs = layer_module( 2025-09-07T07:12:45.4276699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4277226Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4277742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4278187Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4278342Z 2025-09-07T07:12:45.4278448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4278814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4279150Z return mod(**inputs) 2025-09-07T07:12:45.4279551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4279969Z outputs = self.mobilebert( 2025-09-07T07:12:45.4280375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4280799Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4281219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4281627Z layer_outputs = layer_module( 2025-09-07T07:12:45.4282017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4282512Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4283070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4283536Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4284005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4284442Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4284607Z 2025-09-07T07:12:45.4284718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4285109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4285462Z return mod(**inputs) 2025-09-07T07:12:45.4285884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4286332Z outputs = self.mobilebert( 2025-09-07T07:12:45.4286771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4287200Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4287618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4288048Z layer_outputs = layer_module( 2025-09-07T07:12:45.4288480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4288991Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4289526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4290028Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4290532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4290965Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4291115Z 2025-09-07T07:12:45.4291224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4291588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4291923Z return mod(**inputs) 2025-09-07T07:12:45.4292315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4292742Z outputs = self.mobilebert( 2025-09-07T07:12:45.4293152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4293580Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4294000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4294414Z layer_outputs = layer_module( 2025-09-07T07:12:45.4294829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4295338Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4295850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4296325Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4296789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4297265Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4297770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4298216Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4298368Z 2025-09-07T07:12:45.4298481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4298843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4299177Z return mod(**inputs) 2025-09-07T07:12:45.4299584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4300010Z outputs = self.mobilebert( 2025-09-07T07:12:45.4300415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4300836Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4301251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4301678Z layer_outputs = layer_module( 2025-09-07T07:12:45.4302093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4302598Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4303131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4303606Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4304086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4304545Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4304695Z 2025-09-07T07:12:45.4304812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4305201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4305553Z return mod(**inputs) 2025-09-07T07:12:45.4306070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4306540Z outputs = self.mobilebert( 2025-09-07T07:12:45.4306989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4307459Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4307904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4308370Z layer_outputs = layer_module( 2025-09-07T07:12:45.4308778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4309297Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4309833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4310327Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4310821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4311276Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4311742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4312219Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4312379Z 2025-09-07T07:12:45.4312500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4312929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4313279Z return mod(**inputs) 2025-09-07T07:12:45.4313716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4314176Z outputs = self.mobilebert( 2025-09-07T07:12:45.4314619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4315089Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4315534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4316000Z layer_outputs = layer_module( 2025-09-07T07:12:45.4316450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4316936Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4317398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4317852Z self_outputs = self.self( 2025-09-07T07:12:45.4318295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4318782Z self.query(query_tensor) 2025-09-07T07:12:45.4318908Z 2025-09-07T07:12:45.4319028Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4319425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4319955Z return mod(**inputs) 2025-09-07T07:12:45.4320386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4320839Z outputs = self.mobilebert( 2025-09-07T07:12:45.4321281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4321752Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4322198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4322654Z layer_outputs = layer_module( 2025-09-07T07:12:45.4323092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4323561Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4324022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4324478Z self_outputs = self.self( 2025-09-07T07:12:45.4324886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4325318Z self.key(key_tensor) 2025-09-07T07:12:45.4325433Z 2025-09-07T07:12:45.4325545Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4325938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4326289Z return mod(**inputs) 2025-09-07T07:12:45.4326712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4327164Z outputs = self.mobilebert( 2025-09-07T07:12:45.4327591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4328042Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4328481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4328972Z layer_outputs = layer_module( 2025-09-07T07:12:45.4329379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4329815Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4330243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4330664Z self_outputs = self.self( 2025-09-07T07:12:45.4331076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4331497Z self.value(value_tensor) 2025-09-07T07:12:45.4331627Z 2025-09-07T07:12:45.4331713Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4331938Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4332182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4332551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4332885Z return mod(**inputs) 2025-09-07T07:12:45.4333286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4333709Z outputs = self.mobilebert( 2025-09-07T07:12:45.4334154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4334576Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4335026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4335477Z layer_outputs = layer_module( 2025-09-07T07:12:45.4335913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4336400Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4336850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4337370Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4337877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4338341Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4338485Z 2025-09-07T07:12:45.4338601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4338960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4339293Z return mod(**inputs) 2025-09-07T07:12:45.4339694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4340119Z outputs = self.mobilebert( 2025-09-07T07:12:45.4340526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4340949Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4341368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4341792Z layer_outputs = layer_module( 2025-09-07T07:12:45.4342216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4342769Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4343313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4343813Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4344336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4344793Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4344941Z 2025-09-07T07:12:45.4345054Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4345447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4345867Z return mod(**inputs) 2025-09-07T07:12:45.4346322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4346787Z outputs = self.mobilebert( 2025-09-07T07:12:45.4347239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4347696Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4348148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4348596Z layer_outputs = layer_module( 2025-09-07T07:12:45.4349027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4349491Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4349989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4350519Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4351019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4351517Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4352027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4352496Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4352656Z 2025-09-07T07:12:45.4352777Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4353165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4353514Z return mod(**inputs) 2025-09-07T07:12:45.4353941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4354494Z outputs = self.mobilebert( 2025-09-07T07:12:45.4354891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4355305Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4355713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4356144Z layer_outputs = layer_module( 2025-09-07T07:12:45.4356559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4357015Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4357443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4357899Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4358346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4358771Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4358910Z 2025-09-07T07:12:45.4359020Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4359396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4359720Z return mod(**inputs) 2025-09-07T07:12:45.4360113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4360529Z outputs = self.mobilebert( 2025-09-07T07:12:45.4360931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4361334Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4361740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4362149Z layer_outputs = layer_module( 2025-09-07T07:12:45.4362549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4362990Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4363417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4363872Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4364335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4364791Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4364957Z 2025-09-07T07:12:45.4365084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4365443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4365779Z return mod(**inputs) 2025-09-07T07:12:45.4366185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4366610Z outputs = self.mobilebert( 2025-09-07T07:12:45.4367018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4367431Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4367836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4368244Z layer_outputs = layer_module( 2025-09-07T07:12:45.4368647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4369068Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4369499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4369959Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4370423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4370843Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4370983Z 2025-09-07T07:12:45.4371086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4371444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4371770Z return mod(**inputs) 2025-09-07T07:12:45.4372162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4372566Z outputs = self.mobilebert( 2025-09-07T07:12:45.4372963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4373371Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4373804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4374202Z layer_outputs = layer_module( 2025-09-07T07:12:45.4374588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4375010Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4375444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4375913Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4376378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4376832Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4377299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4377731Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4377882Z 2025-09-07T07:12:45.4377992Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4378357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4378689Z return mod(**inputs) 2025-09-07T07:12:45.4379088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4379494Z outputs = self.mobilebert( 2025-09-07T07:12:45.4379881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4380277Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4380674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4381080Z layer_outputs = layer_module( 2025-09-07T07:12:45.4381477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4381906Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4382335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4382459Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4382749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4382843Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4382847Z 2025-09-07T07:12:45.4382952Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4383166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4383234Z return mod(**inputs) 2025-09-07T07:12:45.4383526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4383608Z outputs = self.mobilebert( 2025-09-07T07:12:45.4383896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4383979Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4384277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4384353Z layer_outputs = layer_module( 2025-09-07T07:12:45.4384666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4384810Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4385117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4385238Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4385545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4385671Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4385674Z 2025-09-07T07:12:45.4385871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4386099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4386172Z return mod(**inputs) 2025-09-07T07:12:45.4386485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4386564Z outputs = self.mobilebert( 2025-09-07T07:12:45.4386864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4386951Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4387253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4387360Z layer_outputs = layer_module( 2025-09-07T07:12:45.4387677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4387789Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4388096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4388224Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4388522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4388610Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4388614Z 2025-09-07T07:12:45.4388726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4388929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4388998Z return mod(**inputs) 2025-09-07T07:12:45.4389299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4389371Z outputs = self.mobilebert( 2025-09-07T07:12:45.4389662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4389734Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4390029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4390102Z layer_outputs = layer_module( 2025-09-07T07:12:45.4390383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4390486Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4390773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4390908Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4391191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4391315Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4391613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4391748Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4391752Z 2025-09-07T07:12:45.4391871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4392084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4392162Z return mod(**inputs) 2025-09-07T07:12:45.4392472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4392549Z outputs = self.mobilebert( 2025-09-07T07:12:45.4392865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4392938Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4393229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4393304Z layer_outputs = layer_module( 2025-09-07T07:12:45.4393592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4393693Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4393996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4394120Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4394423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4394518Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4394522Z 2025-09-07T07:12:45.4394626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4394832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4394906Z return mod(**inputs) 2025-09-07T07:12:45.4395192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4395271Z outputs = self.mobilebert( 2025-09-07T07:12:45.4395556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4395632Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4395921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4395993Z layer_outputs = layer_module( 2025-09-07T07:12:45.4396282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4396381Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4396688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4396809Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4397119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4397245Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4397248Z 2025-09-07T07:12:45.4397355Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4397565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4397629Z return mod(**inputs) 2025-09-07T07:12:45.4397917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4398029Z outputs = self.mobilebert( 2025-09-07T07:12:45.4398314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4398394Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4398682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4398765Z layer_outputs = layer_module( 2025-09-07T07:12:45.4399050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4399147Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4399438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4399567Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4399862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4399949Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4399952Z 2025-09-07T07:12:45.4400058Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4400269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4400352Z return mod(**inputs) 2025-09-07T07:12:45.4400676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4400749Z outputs = self.mobilebert( 2025-09-07T07:12:45.4401043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4401116Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4401403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4401483Z layer_outputs = layer_module( 2025-09-07T07:12:45.4401770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4401873Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4402161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4402289Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4402585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4402709Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4403003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4403102Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4403106Z 2025-09-07T07:12:45.4403215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4403416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4403484Z return mod(**inputs) 2025-09-07T07:12:45.4403782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4403854Z outputs = self.mobilebert( 2025-09-07T07:12:45.4404145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4404218Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4404504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4404616Z layer_outputs = layer_module( 2025-09-07T07:12:45.4404906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4405040Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4405333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4405426Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4405431Z 2025-09-07T07:12:45.4405534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4405736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4405812Z return mod(**inputs) 2025-09-07T07:12:45.4406111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4406195Z outputs = self.mobilebert( 2025-09-07T07:12:45.4406513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4406589Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4406918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4406992Z layer_outputs = layer_module( 2025-09-07T07:12:45.4407306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4407430Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4407725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4407843Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4407847Z 2025-09-07T07:12:45.4407951Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4408163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4408229Z return mod(**inputs) 2025-09-07T07:12:45.4408529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4408599Z outputs = self.mobilebert( 2025-09-07T07:12:45.4408891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4408973Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4409260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4409343Z layer_outputs = layer_module( 2025-09-07T07:12:45.4409631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4409804Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4410091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4410189Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4410192Z 2025-09-07T07:12:45.4410305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4410508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4410581Z return mod(**inputs) 2025-09-07T07:12:45.4410870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4410987Z outputs = self.mobilebert( 2025-09-07T07:12:45.4411280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4411352Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4411650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4411723Z layer_outputs = layer_module( 2025-09-07T07:12:45.4412014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4412181Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4412465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4412598Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4412884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4412986Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4412990Z 2025-09-07T07:12:45.4413092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4413308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4413383Z return mod(**inputs) 2025-09-07T07:12:45.4413687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4413766Z outputs = self.mobilebert( 2025-09-07T07:12:45.4414052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4414132Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4414419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4414490Z layer_outputs = layer_module( 2025-09-07T07:12:45.4414784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4414950Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4415243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4415370Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4415654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4415748Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4415754Z 2025-09-07T07:12:45.4415858Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4416083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4416156Z return mod(**inputs) 2025-09-07T07:12:45.4416473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4416549Z outputs = self.mobilebert( 2025-09-07T07:12:45.4416852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4416937Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4417241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4417325Z layer_outputs = layer_module( 2025-09-07T07:12:45.4417624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4417830Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4418140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4418273Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4418586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4418719Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4419027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4419126Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4419129Z 2025-09-07T07:12:45.4419240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4419459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4421396Z return mod(**inputs) 2025-09-07T07:12:45.4421707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4421792Z outputs = self.mobilebert( 2025-09-07T07:12:45.4422147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4422256Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4422563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4422639Z layer_outputs = layer_module( 2025-09-07T07:12:45.4422948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4423128Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4423438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4423559Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4423880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4423980Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4423984Z 2025-09-07T07:12:45.4424094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4424331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4424401Z return mod(**inputs) 2025-09-07T07:12:45.4424725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4424804Z outputs = self.mobilebert( 2025-09-07T07:12:45.4425117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4425204Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4425519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4425603Z layer_outputs = layer_module( 2025-09-07T07:12:45.4425982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4426165Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4426492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4426673Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4427015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4427109Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4427430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4427531Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4427536Z 2025-09-07T07:12:45.4427648Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4427869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4427940Z return mod(**inputs) 2025-09-07T07:12:45.4428258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4428336Z outputs = self.mobilebert( 2025-09-07T07:12:45.4428637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4428723Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4429072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4429156Z layer_outputs = layer_module( 2025-09-07T07:12:45.4429471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4429573Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4429873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4429952Z self_outputs = self.self( 2025-09-07T07:12:45.4430256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4460588Z self.query(query_tensor) 2025-09-07T07:12:45.4460625Z 2025-09-07T07:12:45.4460862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4461118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4461230Z return mod(**inputs) 2025-09-07T07:12:45.4461590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4461689Z outputs = self.mobilebert( 2025-09-07T07:12:45.4462005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4462100Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4462414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4462498Z layer_outputs = layer_module( 2025-09-07T07:12:45.4462811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4462913Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4463236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4463317Z self_outputs = self.self( 2025-09-07T07:12:45.4463634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4463719Z self.key(key_tensor) 2025-09-07T07:12:45.4463724Z 2025-09-07T07:12:45.4463846Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4464205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4464277Z return mod(**inputs) 2025-09-07T07:12:45.4464590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4464679Z outputs = self.mobilebert( 2025-09-07T07:12:45.4464994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4465089Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4465401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4465486Z layer_outputs = layer_module( 2025-09-07T07:12:45.4465892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4466002Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4466318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4466405Z self_outputs = self.self( 2025-09-07T07:12:45.4466721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4466884Z self.value(value_tensor) 2025-09-07T07:12:45.4466888Z 2025-09-07T07:12:45.4466983Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4467097Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4467227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4467453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4467533Z return mod(**inputs) 2025-09-07T07:12:45.4467852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4467945Z outputs = self.mobilebert( 2025-09-07T07:12:45.4468249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4468324Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4468620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4468692Z layer_outputs = layer_module( 2025-09-07T07:12:45.4468998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4469087Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4469372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4469514Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4469800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4469899Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4469902Z 2025-09-07T07:12:45.4470012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4470229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4470296Z return mod(**inputs) 2025-09-07T07:12:45.4470592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4470674Z outputs = self.mobilebert( 2025-09-07T07:12:45.4470960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4471097Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4471387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4471459Z layer_outputs = layer_module( 2025-09-07T07:12:45.4471756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4471932Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4472231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4472349Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4472642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4472728Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4472735Z 2025-09-07T07:12:45.4472840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4473055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4473122Z return mod(**inputs) 2025-09-07T07:12:45.4473420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4473517Z outputs = self.mobilebert( 2025-09-07T07:12:45.4473817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4473901Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4474186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4474257Z layer_outputs = layer_module( 2025-09-07T07:12:45.4474552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4474643Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4474934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4475058Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4475353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4475486Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4475771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4475880Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4475884Z 2025-09-07T07:12:45.4475998Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4476220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4476292Z return mod(**inputs) 2025-09-07T07:12:45.4476609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4476694Z outputs = self.mobilebert( 2025-09-07T07:12:45.4477009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4477097Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4477398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4477480Z layer_outputs = layer_module( 2025-09-07T07:12:45.4477786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4477917Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4478208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4478325Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4478619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4478707Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4478711Z 2025-09-07T07:12:45.4478816Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4479024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4479091Z return mod(**inputs) 2025-09-07T07:12:45.4479385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4479462Z outputs = self.mobilebert( 2025-09-07T07:12:45.4479749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4479822Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4480119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4480209Z layer_outputs = layer_module( 2025-09-07T07:12:45.4480500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4480603Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4480887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4481008Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4481305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4481421Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4481424Z 2025-09-07T07:12:45.4481537Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4481746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4481819Z return mod(**inputs) 2025-09-07T07:12:45.4482113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4482187Z outputs = self.mobilebert( 2025-09-07T07:12:45.4482484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4482561Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4482863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4482936Z layer_outputs = layer_module( 2025-09-07T07:12:45.4483224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4483330Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4483622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4483760Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4484051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4484146Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4484181Z 2025-09-07T07:12:45.4484287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4484487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4484562Z return mod(**inputs) 2025-09-07T07:12:45.4484854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4484938Z outputs = self.mobilebert( 2025-09-07T07:12:45.4485225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4485299Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4485596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4485671Z layer_outputs = layer_module( 2025-09-07T07:12:45.4485978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4486082Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4486383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4486519Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4486819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4486965Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4487253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4487356Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4487359Z 2025-09-07T07:12:45.4487467Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4487673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4487748Z return mod(**inputs) 2025-09-07T07:12:45.4488034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4488115Z outputs = self.mobilebert( 2025-09-07T07:12:45.4488403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4488485Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4488768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4488839Z layer_outputs = layer_module( 2025-09-07T07:12:45.4489134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4489236Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4489546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4489671Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4489979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4490074Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4490077Z 2025-09-07T07:12:45.4490185Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4490394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4490460Z return mod(**inputs) 2025-09-07T07:12:45.4490759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4490862Z outputs = self.mobilebert( 2025-09-07T07:12:45.4491150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4491230Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4491516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4491594Z layer_outputs = layer_module( 2025-09-07T07:12:45.4491879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4491973Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4492268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4492385Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4492677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4492791Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4492794Z 2025-09-07T07:12:45.4492906Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4493120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4493187Z return mod(**inputs) 2025-09-07T07:12:45.4493511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4493584Z outputs = self.mobilebert( 2025-09-07T07:12:45.4493881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4493958Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4494247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4494327Z layer_outputs = layer_module( 2025-09-07T07:12:45.4494613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4494714Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4495000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4495134Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4495421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4495507Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4495514Z 2025-09-07T07:12:45.4495626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4495828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4495900Z return mod(**inputs) 2025-09-07T07:12:45.4496192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4496269Z outputs = self.mobilebert( 2025-09-07T07:12:45.4496577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4496656Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4496971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4497042Z layer_outputs = layer_module( 2025-09-07T07:12:45.4497335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4497466Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4497747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4497880Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4498162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4498295Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4498575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4498668Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4498680Z 2025-09-07T07:12:45.4498782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4498983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4499058Z return mod(**inputs) 2025-09-07T07:12:45.4499344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4499424Z outputs = self.mobilebert( 2025-09-07T07:12:45.4499722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4499812Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4500105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4500177Z layer_outputs = layer_module( 2025-09-07T07:12:45.4500473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4500570Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4500856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4500976Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4501271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4501370Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4501373Z 2025-09-07T07:12:45.4501487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4501706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4501775Z return mod(**inputs) 2025-09-07T07:12:45.4502077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4502164Z outputs = self.mobilebert( 2025-09-07T07:12:45.4502463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4502550Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4502852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4502928Z layer_outputs = layer_module( 2025-09-07T07:12:45.4503237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4503335Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4503648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4503761Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4504083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4504195Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4504199Z 2025-09-07T07:12:45.4504303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4504513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4504579Z return mod(**inputs) 2025-09-07T07:12:45.4504881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4504959Z outputs = self.mobilebert( 2025-09-07T07:12:45.4505257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4505347Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4505645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4505972Z layer_outputs = layer_module( 2025-09-07T07:12:45.4506282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4506412Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4506736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4506873Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4507186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4507279Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4507285Z 2025-09-07T07:12:45.4507408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4507627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4507698Z return mod(**inputs) 2025-09-07T07:12:45.4507998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4508077Z outputs = self.mobilebert( 2025-09-07T07:12:45.4508374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4508454Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4508747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4508822Z layer_outputs = layer_module( 2025-09-07T07:12:45.4509108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4509216Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4509502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4509639Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4509929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4510060Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4510355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4510452Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4510456Z 2025-09-07T07:12:45.4510569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4510809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4510879Z return mod(**inputs) 2025-09-07T07:12:45.4511163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4511234Z outputs = self.mobilebert( 2025-09-07T07:12:45.4511526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4511601Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4511891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4511962Z layer_outputs = layer_module( 2025-09-07T07:12:45.4512241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4512377Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4512659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4512751Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4512755Z 2025-09-07T07:12:45.4512858Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4513081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4513175Z return mod(**inputs) 2025-09-07T07:12:45.4513454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4513532Z outputs = self.mobilebert( 2025-09-07T07:12:45.4513811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4513894Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4514170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4514240Z layer_outputs = layer_module( 2025-09-07T07:12:45.4514524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4514646Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4514933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4515045Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4515048Z 2025-09-07T07:12:45.4515156Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4515353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4515423Z return mod(**inputs) 2025-09-07T07:12:45.4515716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4515792Z outputs = self.mobilebert( 2025-09-07T07:12:45.4516084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4516163Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4516451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4516532Z layer_outputs = layer_module( 2025-09-07T07:12:45.4516817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4516991Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4517314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4517418Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4517422Z 2025-09-07T07:12:45.4517536Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4517735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4517807Z return mod(**inputs) 2025-09-07T07:12:45.4518090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4518168Z outputs = self.mobilebert( 2025-09-07T07:12:45.4518445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4518515Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4518799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4518870Z layer_outputs = layer_module( 2025-09-07T07:12:45.4519152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4519336Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4519824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4519957Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4520237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4520342Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4520348Z 2025-09-07T07:12:45.4520450Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4520655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4520721Z return mod(**inputs) 2025-09-07T07:12:45.4521000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4521080Z outputs = self.mobilebert( 2025-09-07T07:12:45.4521362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4521444Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4521723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4521802Z layer_outputs = layer_module( 2025-09-07T07:12:45.4522082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4522244Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4522537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4522661Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4522951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4523038Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4523042Z 2025-09-07T07:12:45.4523152Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4523353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4523419Z return mod(**inputs) 2025-09-07T07:12:45.4523757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4523827Z outputs = self.mobilebert( 2025-09-07T07:12:45.4524159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4524232Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4524509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4524589Z layer_outputs = layer_module( 2025-09-07T07:12:45.4524870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4525036Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4525320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4525454Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4525763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4525894Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4526225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4526340Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4526344Z 2025-09-07T07:12:45.4526462Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4526686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4526757Z return mod(**inputs) 2025-09-07T07:12:45.4527082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4527157Z outputs = self.mobilebert( 2025-09-07T07:12:45.4527459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4527533Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4527826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4527906Z layer_outputs = layer_module( 2025-09-07T07:12:45.4528179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4528350Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4528627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4528749Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4529024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4529106Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4529109Z 2025-09-07T07:12:45.4529217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4529412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4529486Z return mod(**inputs) 2025-09-07T07:12:45.4529764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4529840Z outputs = self.mobilebert( 2025-09-07T07:12:45.4530117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4530222Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4530506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4530575Z layer_outputs = layer_module( 2025-09-07T07:12:45.4530863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4531021Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4531300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4531417Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4531696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4531791Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4532071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4532167Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4532170Z 2025-09-07T07:12:45.4532271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4532484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4532561Z return mod(**inputs) 2025-09-07T07:12:45.4533535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4533621Z outputs = self.mobilebert( 2025-09-07T07:12:45.4533904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4533982Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4534271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4534343Z layer_outputs = layer_module( 2025-09-07T07:12:45.4534631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4534724Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4535014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4535087Z self_outputs = self.self( 2025-09-07T07:12:45.4535383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4535467Z self.query(query_tensor) 2025-09-07T07:12:45.4535475Z 2025-09-07T07:12:45.4535590Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4535809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4535880Z return mod(**inputs) 2025-09-07T07:12:45.4536181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4536269Z outputs = self.mobilebert( 2025-09-07T07:12:45.4536567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4536651Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4536933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4537012Z layer_outputs = layer_module( 2025-09-07T07:12:45.4537293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4537417Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4537719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4537794Z self_outputs = self.self( 2025-09-07T07:12:45.4538103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4538174Z self.key(key_tensor) 2025-09-07T07:12:45.4538177Z 2025-09-07T07:12:45.4538289Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4538507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4538577Z return mod(**inputs) 2025-09-07T07:12:45.4538887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4538968Z outputs = self.mobilebert( 2025-09-07T07:12:45.4539267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4539354Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4539668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4539754Z layer_outputs = layer_module( 2025-09-07T07:12:45.4540070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4540161Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4540467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4540541Z self_outputs = self.self( 2025-09-07T07:12:45.4540853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4540926Z self.value(value_tensor) 2025-09-07T07:12:45.4540930Z 2025-09-07T07:12:45.4541022Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4541108Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4541218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4541438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4541507Z return mod(**inputs) 2025-09-07T07:12:45.4541815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4541892Z outputs = self.mobilebert( 2025-09-07T07:12:45.4542192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4542280Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4542579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4542661Z layer_outputs = layer_module( 2025-09-07T07:12:45.4542967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4543067Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4543369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4543501Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4543806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4543896Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4543937Z 2025-09-07T07:12:45.4544055Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4544267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4544336Z return mod(**inputs) 2025-09-07T07:12:45.4544649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4544727Z outputs = self.mobilebert( 2025-09-07T07:12:45.4545040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4545117Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4545427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4545510Z layer_outputs = layer_module( 2025-09-07T07:12:45.4545893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4546085Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4546410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4546566Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4546902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4546994Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4546999Z 2025-09-07T07:12:45.4547118Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4547345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4547429Z return mod(**inputs) 2025-09-07T07:12:45.4547746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4547833Z outputs = self.mobilebert( 2025-09-07T07:12:45.4548144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4548224Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4548539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4548618Z layer_outputs = layer_module( 2025-09-07T07:12:45.4548934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4549025Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4549337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4549481Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4549861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4549996Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4550276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4550379Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4550383Z 2025-09-07T07:12:45.4550483Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4550678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4550753Z return mod(**inputs) 2025-09-07T07:12:45.4551084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4551202Z outputs = self.mobilebert( 2025-09-07T07:12:45.4551490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4551563Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4551860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4551931Z layer_outputs = layer_module( 2025-09-07T07:12:45.4552215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4552309Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4552594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4552710Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4552984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4553076Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4553079Z 2025-09-07T07:12:45.4553180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4553394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4553460Z return mod(**inputs) 2025-09-07T07:12:45.4553753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4553834Z outputs = self.mobilebert( 2025-09-07T07:12:45.4554111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4554195Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4554471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4554547Z layer_outputs = layer_module( 2025-09-07T07:12:45.4554827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4554924Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4555217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4555331Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4555623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4555741Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4555747Z 2025-09-07T07:12:45.4555850Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4556060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4556127Z return mod(**inputs) 2025-09-07T07:12:45.4556428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4556501Z outputs = self.mobilebert( 2025-09-07T07:12:45.4556800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4556872Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4557148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4557226Z layer_outputs = layer_module( 2025-09-07T07:12:45.4557565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4557665Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4557939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4558068Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4558353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4558437Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4558440Z 2025-09-07T07:12:45.4558544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4558736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4558808Z return mod(**inputs) 2025-09-07T07:12:45.4559086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4559157Z outputs = self.mobilebert( 2025-09-07T07:12:45.4559436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4559506Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4559805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4559890Z layer_outputs = layer_module( 2025-09-07T07:12:45.4560166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4560268Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4560543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4560680Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4560956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4561083Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4561433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4561529Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4561533Z 2025-09-07T07:12:45.4561640Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4561836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4561905Z return mod(**inputs) 2025-09-07T07:12:45.4562187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4562260Z outputs = self.mobilebert( 2025-09-07T07:12:45.4562545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4562618Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4562903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4562973Z layer_outputs = layer_module( 2025-09-07T07:12:45.4563261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4563355Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4563630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4563786Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4564067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4564156Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4564160Z 2025-09-07T07:12:45.4564260Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4564459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4564533Z return mod(**inputs) 2025-09-07T07:12:45.4564815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4564897Z outputs = self.mobilebert( 2025-09-07T07:12:45.4565183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4565269Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4565560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4565637Z layer_outputs = layer_module( 2025-09-07T07:12:45.4565951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4566069Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4566392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4566514Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4566814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4566948Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4566955Z 2025-09-07T07:12:45.4567054Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4567253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4567317Z return mod(**inputs) 2025-09-07T07:12:45.4567605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4567679Z outputs = self.mobilebert( 2025-09-07T07:12:45.4567961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4568042Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4568326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4568407Z layer_outputs = layer_module( 2025-09-07T07:12:45.4568689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4568784Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4569074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4569200Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4569490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4569575Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4569579Z 2025-09-07T07:12:45.4569687Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4569885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4569950Z return mod(**inputs) 2025-09-07T07:12:45.4570281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4570352Z outputs = self.mobilebert( 2025-09-07T07:12:45.4570640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4570714Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4570997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4571079Z layer_outputs = layer_module( 2025-09-07T07:12:45.4571359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4571460Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4571739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4571875Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4572155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4572277Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4572584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4572692Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4572696Z 2025-09-07T07:12:45.4572807Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4573010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4573078Z return mod(**inputs) 2025-09-07T07:12:45.4573371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4573446Z outputs = self.mobilebert( 2025-09-07T07:12:45.4573739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4573811Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4574105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4574179Z layer_outputs = layer_module( 2025-09-07T07:12:45.4574465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4574567Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4574851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4574977Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4575260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4575345Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4575356Z 2025-09-07T07:12:45.4575459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4575666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4575737Z return mod(**inputs) 2025-09-07T07:12:45.4576034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4576115Z outputs = self.mobilebert( 2025-09-07T07:12:45.4576417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4576531Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4576851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4576926Z layer_outputs = layer_module( 2025-09-07T07:12:45.4577257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4577353Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4577642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4577764Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4578052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4578172Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4578178Z 2025-09-07T07:12:45.4578282Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4578495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4578561Z return mod(**inputs) 2025-09-07T07:12:45.4578852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4578952Z outputs = self.mobilebert( 2025-09-07T07:12:45.4579253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4579335Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4579625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4579694Z layer_outputs = layer_module( 2025-09-07T07:12:45.4579979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4580072Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4580351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4580473Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4580755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4580841Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4580845Z 2025-09-07T07:12:45.4580946Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4581155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4581221Z return mod(**inputs) 2025-09-07T07:12:45.4581517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4581588Z outputs = self.mobilebert( 2025-09-07T07:12:45.4581874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4581957Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4582241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4582323Z layer_outputs = layer_module( 2025-09-07T07:12:45.4582605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4582709Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4582991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4583152Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4583445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4583569Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4583864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4583958Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4583962Z 2025-09-07T07:12:45.4584067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4584281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4584347Z return mod(**inputs) 2025-09-07T07:12:45.4584642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4584715Z outputs = self.mobilebert( 2025-09-07T07:12:45.4585005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4585078Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4585378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4585459Z layer_outputs = layer_module( 2025-09-07T07:12:45.4585849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4585997Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4586302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4586395Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4586408Z 2025-09-07T07:12:45.4586522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4586741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4586823Z return mod(**inputs) 2025-09-07T07:12:45.4587145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4587230Z outputs = self.mobilebert( 2025-09-07T07:12:45.4587541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4587620Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4587935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4588010Z layer_outputs = layer_module( 2025-09-07T07:12:45.4588302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4588436Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4588713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4588832Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4588836Z 2025-09-07T07:12:45.4588937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4589139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4589203Z return mod(**inputs) 2025-09-07T07:12:45.4589487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4589590Z outputs = self.mobilebert( 2025-09-07T07:12:45.4589865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4589945Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4590222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4590302Z layer_outputs = layer_module( 2025-09-07T07:12:45.4590577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4590735Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4591023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4591121Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4591127Z 2025-09-07T07:12:45.4591239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4591439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4591513Z return mod(**inputs) 2025-09-07T07:12:45.4591802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4591889Z outputs = self.mobilebert( 2025-09-07T07:12:45.4592205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4592281Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4592573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4592644Z layer_outputs = layer_module( 2025-09-07T07:12:45.4592928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4593095Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4593376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4593509Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4593792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4593890Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4593894Z 2025-09-07T07:12:45.4593996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4594207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4594284Z return mod(**inputs) 2025-09-07T07:12:45.4594567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4594638Z outputs = self.mobilebert( 2025-09-07T07:12:45.4594924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4594999Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4595291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4595364Z layer_outputs = layer_module( 2025-09-07T07:12:45.4595647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4595816Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4596113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4596288Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4596589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4596687Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4596692Z 2025-09-07T07:12:45.4596802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4597017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4597096Z return mod(**inputs) 2025-09-07T07:12:45.4597405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4597492Z outputs = self.mobilebert( 2025-09-07T07:12:45.4597810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4597897Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4598195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4598272Z layer_outputs = layer_module( 2025-09-07T07:12:45.4598593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4598776Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4599093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4599218Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4599504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4599640Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4599925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4600025Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4600029Z 2025-09-07T07:12:45.4600134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4600342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4600410Z return mod(**inputs) 2025-09-07T07:12:45.4600696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4600775Z outputs = self.mobilebert( 2025-09-07T07:12:45.4601062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4601146Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4601430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4601501Z layer_outputs = layer_module( 2025-09-07T07:12:45.4601793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4601959Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4602249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4602365Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4602656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4602771Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4602775Z 2025-09-07T07:12:45.4602878Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4603085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4603148Z return mod(**inputs) 2025-09-07T07:12:45.4603444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4603516Z outputs = self.mobilebert( 2025-09-07T07:12:45.4603801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4603882Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4604166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4604249Z layer_outputs = layer_module( 2025-09-07T07:12:45.4604532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4604700Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4605000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4605113Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4605418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4605509Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4605798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4605893Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4605896Z 2025-09-07T07:12:45.4606006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4606207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4606271Z return mod(**inputs) 2025-09-07T07:12:45.4606568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4606642Z outputs = self.mobilebert( 2025-09-07T07:12:45.4606936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4607009Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4607294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4607377Z layer_outputs = layer_module( 2025-09-07T07:12:45.4607658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4607751Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4608035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4608109Z self_outputs = self.self( 2025-09-07T07:12:45.4608405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4608477Z self.query(query_tensor) 2025-09-07T07:12:45.4608481Z 2025-09-07T07:12:45.4608590Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4608792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4608866Z return mod(**inputs) 2025-09-07T07:12:45.4609180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4609252Z outputs = self.mobilebert( 2025-09-07T07:12:45.4609540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4609614Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4609905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4609978Z layer_outputs = layer_module( 2025-09-07T07:12:45.4610263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4610357Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4610638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4610719Z self_outputs = self.self( 2025-09-07T07:12:45.4611048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4611122Z self.key(key_tensor) 2025-09-07T07:12:45.4611125Z 2025-09-07T07:12:45.4611224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4611428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4611504Z return mod(**inputs) 2025-09-07T07:12:45.4611805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4611882Z outputs = self.mobilebert( 2025-09-07T07:12:45.4612173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4612247Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4612533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4612602Z layer_outputs = layer_module( 2025-09-07T07:12:45.4612886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4612971Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4613252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4613328Z self_outputs = self.self( 2025-09-07T07:12:45.4613607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4613687Z self.value(value_tensor) 2025-09-07T07:12:45.4613690Z 2025-09-07T07:12:45.4613775Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4613860Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4613963Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4614158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4614230Z return mod(**inputs) 2025-09-07T07:12:45.4614509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4614586Z outputs = self.mobilebert( 2025-09-07T07:12:45.4614865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4614936Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4615220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4615325Z layer_outputs = layer_module( 2025-09-07T07:12:45.4615609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4615691Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4615974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4616097Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4616372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4616464Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4616468Z 2025-09-07T07:12:45.4616569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4616778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4616850Z return mod(**inputs) 2025-09-07T07:12:45.4617136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4617216Z outputs = self.mobilebert( 2025-09-07T07:12:45.4617495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4617602Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4617902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4617975Z layer_outputs = layer_module( 2025-09-07T07:12:45.4618276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4618434Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4618724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4618835Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4619118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4619200Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4619205Z 2025-09-07T07:12:45.4619304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4619509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4619786Z return mod(**inputs) 2025-09-07T07:12:45.4620090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4620161Z outputs = self.mobilebert( 2025-09-07T07:12:45.4620447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4620520Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4620795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4620874Z layer_outputs = layer_module( 2025-09-07T07:12:45.4621152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4621244Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4621520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4621645Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4621929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4622116Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4622398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4622490Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4622494Z 2025-09-07T07:12:45.4622603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4622798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4622865Z return mod(**inputs) 2025-09-07T07:12:45.4623150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4623222Z outputs = self.mobilebert( 2025-09-07T07:12:45.4623513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4623590Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4623867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4623947Z layer_outputs = layer_module( 2025-09-07T07:12:45.4624248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4624352Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4624651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4624775Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4625057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4625146Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4625151Z 2025-09-07T07:12:45.4625262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4625462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4625537Z return mod(**inputs) 2025-09-07T07:12:45.4625873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4625949Z outputs = self.mobilebert( 2025-09-07T07:12:45.4626242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4626315Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4626602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4626679Z layer_outputs = layer_module( 2025-09-07T07:12:45.4626968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4627065Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4627352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4627471Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4627746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4627865Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4627869Z 2025-09-07T07:12:45.4627970Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4628163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4628271Z return mod(**inputs) 2025-09-07T07:12:45.4628564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4628644Z outputs = self.mobilebert( 2025-09-07T07:12:45.4628932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4629014Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4629308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4629378Z layer_outputs = layer_module( 2025-09-07T07:12:45.4629670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4629766Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4630061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4630190Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4630479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4630572Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4630592Z 2025-09-07T07:12:45.4630697Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4630920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4630988Z return mod(**inputs) 2025-09-07T07:12:45.4631278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4631351Z outputs = self.mobilebert( 2025-09-07T07:12:45.4631641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4631722Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4632005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4632083Z layer_outputs = layer_module( 2025-09-07T07:12:45.4632369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4632465Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4632755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4632879Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4633168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4633294Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4633586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4633680Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4633684Z 2025-09-07T07:12:45.4633787Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4633994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4634063Z return mod(**inputs) 2025-09-07T07:12:45.4634358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4634429Z outputs = self.mobilebert( 2025-09-07T07:12:45.4634713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4634824Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4635111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4635189Z layer_outputs = layer_module( 2025-09-07T07:12:45.4635476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4635577Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4635870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4635984Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4636273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4636360Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4636364Z 2025-09-07T07:12:45.4636472Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4636670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4636735Z return mod(**inputs) 2025-09-07T07:12:45.4637046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4637120Z outputs = self.mobilebert( 2025-09-07T07:12:45.4637431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4637507Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4637797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4637871Z layer_outputs = layer_module( 2025-09-07T07:12:45.4638155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4638256Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4638540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4638660Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4638945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4639058Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4639069Z 2025-09-07T07:12:45.4639184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4639379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4639452Z return mod(**inputs) 2025-09-07T07:12:45.4639733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4639813Z outputs = self.mobilebert( 2025-09-07T07:12:45.4640095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4640167Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4640454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4640523Z layer_outputs = layer_module( 2025-09-07T07:12:45.4640806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4640900Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4641177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4641357Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4641633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4641722Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4641725Z 2025-09-07T07:12:45.4641829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4642034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4642098Z return mod(**inputs) 2025-09-07T07:12:45.4642375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4642452Z outputs = self.mobilebert( 2025-09-07T07:12:45.4642723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4642805Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4643082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4643150Z layer_outputs = layer_module( 2025-09-07T07:12:45.4643453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4643546Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4643841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4643967Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4644248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4644371Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4644647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4644746Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4644749Z 2025-09-07T07:12:45.4644851Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4645057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4645127Z return mod(**inputs) 2025-09-07T07:12:45.4645413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4645491Z outputs = self.mobilebert( 2025-09-07T07:12:45.4645777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4645859Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4646145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4646224Z layer_outputs = layer_module( 2025-09-07T07:12:45.4646513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4646608Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4646901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4647015Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4647303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4647418Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4647421Z 2025-09-07T07:12:45.4647524Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4647734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4647800Z return mod(**inputs) 2025-09-07T07:12:45.4648101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4648172Z outputs = self.mobilebert( 2025-09-07T07:12:45.4648471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4648543Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4648815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4648896Z layer_outputs = layer_module( 2025-09-07T07:12:45.4649167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4649267Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4649543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4649668Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4649968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4650080Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4650083Z 2025-09-07T07:12:45.4650189Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4650381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4650454Z return mod(**inputs) 2025-09-07T07:12:45.4650733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4650802Z outputs = self.mobilebert( 2025-09-07T07:12:45.4651082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4651153Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4651443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4651512Z layer_outputs = layer_module( 2025-09-07T07:12:45.4651788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4651888Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4652161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4652295Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4652570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4652661Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4652665Z 2025-09-07T07:12:45.4652766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4652963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4653034Z return mod(**inputs) 2025-09-07T07:12:45.4653314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4653392Z outputs = self.mobilebert( 2025-09-07T07:12:45.4653674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4653775Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4654048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4654116Z layer_outputs = layer_module( 2025-09-07T07:12:45.4654392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4654482Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4654759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4654881Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4655153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4655286Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4655561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4655659Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4655663Z 2025-09-07T07:12:45.4655780Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4655981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4656105Z return mod(**inputs) 2025-09-07T07:12:45.4656392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4656472Z outputs = self.mobilebert( 2025-09-07T07:12:45.4656756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4656840Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4657120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4657191Z layer_outputs = layer_module( 2025-09-07T07:12:45.4657487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4657610Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4657911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4657995Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4657999Z 2025-09-07T07:12:45.4658098Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4658313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4658379Z return mod(**inputs) 2025-09-07T07:12:45.4658656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4658732Z outputs = self.mobilebert( 2025-09-07T07:12:45.4659012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4659085Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4659356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4659431Z layer_outputs = layer_module( 2025-09-07T07:12:45.4659699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4659821Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4660118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4660223Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4660236Z 2025-09-07T07:12:45.4660333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4660521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4660591Z return mod(**inputs) 2025-09-07T07:12:45.4660864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4660940Z outputs = self.mobilebert( 2025-09-07T07:12:45.4661204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4661274Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4661557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4661626Z layer_outputs = layer_module( 2025-09-07T07:12:45.4661908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4662082Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4662369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4662474Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4662478Z 2025-09-07T07:12:45.4662580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4662789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4662856Z return mod(**inputs) 2025-09-07T07:12:45.4663145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4663216Z outputs = self.mobilebert( 2025-09-07T07:12:45.4663492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4663572Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4663858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4663939Z layer_outputs = layer_module( 2025-09-07T07:12:45.4664224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4664388Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4664682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4664802Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4665089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4665184Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4665189Z 2025-09-07T07:12:45.4665299Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4665503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4665570Z return mod(**inputs) 2025-09-07T07:12:45.4665950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4666031Z outputs = self.mobilebert( 2025-09-07T07:12:45.4666381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4666459Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4666759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4666843Z layer_outputs = layer_module( 2025-09-07T07:12:45.4667141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4667306Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4667578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4667708Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4667982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4668069Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4668072Z 2025-09-07T07:12:45.4668183Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4668375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4668447Z return mod(**inputs) 2025-09-07T07:12:45.4668739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4668824Z outputs = self.mobilebert( 2025-09-07T07:12:45.4669107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4669177Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4669460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4669531Z layer_outputs = layer_module( 2025-09-07T07:12:45.4669815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4669971Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4670250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4670383Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4670661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4670788Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4671064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4671167Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4671170Z 2025-09-07T07:12:45.4671273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4671468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4671541Z return mod(**inputs) 2025-09-07T07:12:45.4671822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4671904Z outputs = self.mobilebert( 2025-09-07T07:12:45.4672183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4672255Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4672540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4672659Z layer_outputs = layer_module( 2025-09-07T07:12:45.4672947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4673108Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4673397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4673509Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4673789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4673879Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4673882Z 2025-09-07T07:12:45.4673984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4674187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4674250Z return mod(**inputs) 2025-09-07T07:12:45.4674531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4674608Z outputs = self.mobilebert( 2025-09-07T07:12:45.4674902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4674983Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4675280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4675367Z layer_outputs = layer_module( 2025-09-07T07:12:45.4675661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4675829Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4676130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4676242Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4676533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4676622Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4676919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4677017Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4677020Z 2025-09-07T07:12:45.4677121Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4677323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4677390Z return mod(**inputs) 2025-09-07T07:12:45.4677677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4677749Z outputs = self.mobilebert( 2025-09-07T07:12:45.4678029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4678107Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4678386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4678464Z layer_outputs = layer_module( 2025-09-07T07:12:45.4678742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4678827Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4679146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4679216Z self_outputs = self.self( 2025-09-07T07:12:45.4679500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4679569Z self.query(query_tensor) 2025-09-07T07:12:45.4679573Z 2025-09-07T07:12:45.4679683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4679883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4679948Z return mod(**inputs) 2025-09-07T07:12:45.4680235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4680303Z outputs = self.mobilebert( 2025-09-07T07:12:45.4680589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4680663Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4680940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4681019Z layer_outputs = layer_module( 2025-09-07T07:12:45.4681309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4681404Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4681694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4681764Z self_outputs = self.self( 2025-09-07T07:12:45.4682051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4682119Z self.key(key_tensor) 2025-09-07T07:12:45.4682122Z 2025-09-07T07:12:45.4682229Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4682422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4682491Z return mod(**inputs) 2025-09-07T07:12:45.4682772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4682841Z outputs = self.mobilebert( 2025-09-07T07:12:45.4683126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4683197Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4683476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4683545Z layer_outputs = layer_module( 2025-09-07T07:12:45.4683822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4683909Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4684184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4684258Z self_outputs = self.self( 2025-09-07T07:12:45.4684536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4684613Z self.value(value_tensor) 2025-09-07T07:12:45.4684616Z 2025-09-07T07:12:45.4684697Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4684775Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4684884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4685077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4685177Z return mod(**inputs) 2025-09-07T07:12:45.4685469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4685543Z outputs = self.mobilebert( 2025-09-07T07:12:45.4685857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4685934Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4686243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4686315Z layer_outputs = layer_module( 2025-09-07T07:12:45.4686608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4686708Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4687017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4687159Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4687465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4687583Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4687587Z 2025-09-07T07:12:45.4687698Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4687934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4688013Z return mod(**inputs) 2025-09-07T07:12:45.4688320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4688404Z outputs = self.mobilebert( 2025-09-07T07:12:45.4688710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4688789Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4689101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4689177Z layer_outputs = layer_module( 2025-09-07T07:12:45.4689489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4689667Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4689981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4690103Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4690414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4690510Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4690514Z 2025-09-07T07:12:45.4690625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4690846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4690917Z return mod(**inputs) 2025-09-07T07:12:45.4691226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4691310Z outputs = self.mobilebert( 2025-09-07T07:12:45.4691610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4691697Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4692001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4692120Z layer_outputs = layer_module( 2025-09-07T07:12:45.4692423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4692513Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4692822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4692954Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4693264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4693400Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4693702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4693812Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4693816Z 2025-09-07T07:12:45.4693925Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4694148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4694220Z return mod(**inputs) 2025-09-07T07:12:45.4694547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4694642Z outputs = self.mobilebert( 2025-09-07T07:12:45.4694941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4695027Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4695329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4695417Z layer_outputs = layer_module( 2025-09-07T07:12:45.4695757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4695861Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4696170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4696292Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4696607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4696697Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4696701Z 2025-09-07T07:12:45.4696816Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4697027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4697108Z return mod(**inputs) 2025-09-07T07:12:45.4697404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4697475Z outputs = self.mobilebert( 2025-09-07T07:12:45.4697774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4697845Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4698123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4698201Z layer_outputs = layer_module( 2025-09-07T07:12:45.4698474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4698578Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4698885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4699003Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4699277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4699388Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4699392Z 2025-09-07T07:12:45.4699498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4699694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4699764Z return mod(**inputs) 2025-09-07T07:12:45.4700043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4700111Z outputs = self.mobilebert( 2025-09-07T07:12:45.4700394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4700465Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4700749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4700818Z layer_outputs = layer_module( 2025-09-07T07:12:45.4701116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4701225Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4701499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4701632Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4701910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4702002Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4702006Z 2025-09-07T07:12:45.4702108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4702314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4702379Z return mod(**inputs) 2025-09-07T07:12:45.4702666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4702748Z outputs = self.mobilebert( 2025-09-07T07:12:45.4703031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4703111Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4703398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4703472Z layer_outputs = layer_module( 2025-09-07T07:12:45.4703765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4703861Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4704153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4704279Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4704562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4704694Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4704973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4705111Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4705115Z 2025-09-07T07:12:45.4705224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4705448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4705518Z return mod(**inputs) 2025-09-07T07:12:45.4705903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4705993Z outputs = self.mobilebert( 2025-09-07T07:12:45.4706300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4706387Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4706692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4706773Z layer_outputs = layer_module( 2025-09-07T07:12:45.4707084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4707186Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4707519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4707649Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4707951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4708038Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4708042Z 2025-09-07T07:12:45.4708144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4708351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4708419Z return mod(**inputs) 2025-09-07T07:12:45.4708713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4708783Z outputs = self.mobilebert( 2025-09-07T07:12:45.4709060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4709140Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4709420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4709498Z layer_outputs = layer_module( 2025-09-07T07:12:45.4709784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4709881Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4710162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4710273Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4710554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4710664Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4710667Z 2025-09-07T07:12:45.4710774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4710968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4711032Z return mod(**inputs) 2025-09-07T07:12:45.4711317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4711386Z outputs = self.mobilebert( 2025-09-07T07:12:45.4711707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4711778Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4712061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4712131Z layer_outputs = layer_module( 2025-09-07T07:12:45.4712408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4712510Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4712788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4712918Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4713194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4713279Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4713290Z 2025-09-07T07:12:45.4713390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4713584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4713654Z return mod(**inputs) 2025-09-07T07:12:45.4713955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4714047Z outputs = self.mobilebert( 2025-09-07T07:12:45.4714328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4714399Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4714684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4714758Z layer_outputs = layer_module( 2025-09-07T07:12:45.4715043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4715134Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4715411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4715544Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4715830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4715965Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4716266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4716373Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4716376Z 2025-09-07T07:12:45.4716483Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4716693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4716770Z return mod(**inputs) 2025-09-07T07:12:45.4717076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4717160Z outputs = self.mobilebert( 2025-09-07T07:12:45.4717465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4717537Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4717832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4717943Z layer_outputs = layer_module( 2025-09-07T07:12:45.4718242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4718337Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4718627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4718742Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4719024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4719117Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4719120Z 2025-09-07T07:12:45.4719222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4719431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4719500Z return mod(**inputs) 2025-09-07T07:12:45.4719906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4719990Z outputs = self.mobilebert( 2025-09-07T07:12:45.4720273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4720399Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4720702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4720785Z layer_outputs = layer_module( 2025-09-07T07:12:45.4721073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4721168Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4721465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4721578Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4721869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4721981Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4721985Z 2025-09-07T07:12:45.4722099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4722303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4722370Z return mod(**inputs) 2025-09-07T07:12:45.4722671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4722745Z outputs = self.mobilebert( 2025-09-07T07:12:45.4723046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4723122Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4723412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4723492Z layer_outputs = layer_module( 2025-09-07T07:12:45.4723781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4723885Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4724179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4724309Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4724600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4724734Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4724737Z 2025-09-07T07:12:45.4724847Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4725050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4725124Z return mod(**inputs) 2025-09-07T07:12:45.4725410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4725484Z outputs = self.mobilebert( 2025-09-07T07:12:45.4725773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4725847Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4726134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4726209Z layer_outputs = layer_module( 2025-09-07T07:12:45.4726488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4726593Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4726890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4727023Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4727325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4727458Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4727738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4727834Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4727837Z 2025-09-07T07:12:45.4727949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4728148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4728221Z return mod(**inputs) 2025-09-07T07:12:45.4728508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4728586Z outputs = self.mobilebert( 2025-09-07T07:12:45.4728865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4728938Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4729227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4729302Z layer_outputs = layer_module( 2025-09-07T07:12:45.4729586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4729707Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4729987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4730074Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4730077Z 2025-09-07T07:12:45.4730174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4730368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4730430Z return mod(**inputs) 2025-09-07T07:12:45.4730704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4730803Z outputs = self.mobilebert( 2025-09-07T07:12:45.4731070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4731148Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4731418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4731493Z layer_outputs = layer_module( 2025-09-07T07:12:45.4731761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4731873Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4732149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4732258Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4732264Z 2025-09-07T07:12:45.4732367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4732557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4732625Z return mod(**inputs) 2025-09-07T07:12:45.4732899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4732981Z outputs = self.mobilebert( 2025-09-07T07:12:45.4733273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4733342Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4733624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4733693Z layer_outputs = layer_module( 2025-09-07T07:12:45.4733964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4734126Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4734394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4734493Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4734498Z 2025-09-07T07:12:45.4734595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4734793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4734855Z return mod(**inputs) 2025-09-07T07:12:45.4735135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4735218Z outputs = self.mobilebert( 2025-09-07T07:12:45.4735506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4735587Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4735872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4735944Z layer_outputs = layer_module( 2025-09-07T07:12:45.4736250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4736413Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4736706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4736831Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4737164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4737255Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4737259Z 2025-09-07T07:12:45.4737357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4737561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4737626Z return mod(**inputs) 2025-09-07T07:12:45.4737915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4737985Z outputs = self.mobilebert( 2025-09-07T07:12:45.4738262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4738342Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4738616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4738695Z layer_outputs = layer_module( 2025-09-07T07:12:45.4738972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4739135Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4739425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4739563Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4739846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4739932Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4739935Z 2025-09-07T07:12:45.4740041Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4740242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4740306Z return mod(**inputs) 2025-09-07T07:12:45.4740592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4740661Z outputs = self.mobilebert( 2025-09-07T07:12:45.4740947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4741019Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4741304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4741373Z layer_outputs = layer_module( 2025-09-07T07:12:45.4741650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4741815Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4742094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4742222Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4742497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4742619Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4742901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4742993Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4742997Z 2025-09-07T07:12:45.4743108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4743340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4743413Z return mod(**inputs) 2025-09-07T07:12:45.4743707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4743776Z outputs = self.mobilebert( 2025-09-07T07:12:45.4744060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4744133Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4744427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4744496Z layer_outputs = layer_module( 2025-09-07T07:12:45.4744775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4744951Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4745236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4745356Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4745660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4745814Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4745820Z 2025-09-07T07:12:45.4745944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4746149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4746222Z return mod(**inputs) 2025-09-07T07:12:45.4746516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4746599Z outputs = self.mobilebert( 2025-09-07T07:12:45.4746905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4746984Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4747309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4747381Z layer_outputs = layer_module( 2025-09-07T07:12:45.4747679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4747844Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4748145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4748259Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4748550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4748648Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4748941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4749045Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4749048Z 2025-09-07T07:12:45.4749153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4749362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4749429Z return mod(**inputs) 2025-09-07T07:12:45.4749721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4749840Z outputs = self.mobilebert( 2025-09-07T07:12:45.4750126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4750206Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4750491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4750565Z layer_outputs = layer_module( 2025-09-07T07:12:45.4750863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4750955Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4751247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4751320Z self_outputs = self.self( 2025-09-07T07:12:45.4751603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4751687Z self.query(query_tensor) 2025-09-07T07:12:45.4751690Z 2025-09-07T07:12:45.4751795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4752005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4752073Z return mod(**inputs) 2025-09-07T07:12:45.4752388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4752476Z outputs = self.mobilebert( 2025-09-07T07:12:45.4752758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4752841Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4753122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4753207Z layer_outputs = layer_module( 2025-09-07T07:12:45.4753490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4753577Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4753870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4753941Z self_outputs = self.self( 2025-09-07T07:12:45.4754237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4754304Z self.key(key_tensor) 2025-09-07T07:12:45.4754308Z 2025-09-07T07:12:45.4754416Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4754615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4754684Z return mod(**inputs) 2025-09-07T07:12:45.4754977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4755048Z outputs = self.mobilebert( 2025-09-07T07:12:45.4755347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4755428Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4755736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4755825Z layer_outputs = layer_module( 2025-09-07T07:12:45.4756146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4756245Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4756586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4756668Z self_outputs = self.self( 2025-09-07T07:12:45.4756969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4757042Z self.value(value_tensor) 2025-09-07T07:12:45.4757046Z 2025-09-07T07:12:45.4757142Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4757227Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4757345Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4757561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4757627Z return mod(**inputs) 2025-09-07T07:12:45.4757924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4757998Z outputs = self.mobilebert( 2025-09-07T07:12:45.4758290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4758362Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4758644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4758777Z layer_outputs = layer_module( 2025-09-07T07:12:45.4759077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4759171Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4759455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4759587Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4759872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4759954Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4759958Z 2025-09-07T07:12:45.4760063Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4760253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4760327Z return mod(**inputs) 2025-09-07T07:12:45.4760607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4760676Z outputs = self.mobilebert( 2025-09-07T07:12:45.4760958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4761030Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4761312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4761384Z layer_outputs = layer_module( 2025-09-07T07:12:45.4761674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4761831Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4762102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4762220Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4762487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4762573Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4762576Z 2025-09-07T07:12:45.4762714Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4762905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4762975Z return mod(**inputs) 2025-09-07T07:12:45.4763247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4763321Z outputs = self.mobilebert( 2025-09-07T07:12:45.4763591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4763669Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4763939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4764006Z layer_outputs = layer_module( 2025-09-07T07:12:45.4764285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4764368Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4764646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4764762Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4765046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4765178Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4765467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4765568Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4765571Z 2025-09-07T07:12:45.4765674Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4765885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4765953Z return mod(**inputs) 2025-09-07T07:12:45.4766241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4766333Z outputs = self.mobilebert( 2025-09-07T07:12:45.4766604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4766686Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4766961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4767033Z layer_outputs = layer_module( 2025-09-07T07:12:45.4767311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4767409Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4767690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4767804Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4768091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4768179Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4768182Z 2025-09-07T07:12:45.4768286Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4768496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4768568Z return mod(**inputs) 2025-09-07T07:12:45.4768861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4768964Z outputs = self.mobilebert( 2025-09-07T07:12:45.4769239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4769317Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4769594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4769683Z layer_outputs = layer_module( 2025-09-07T07:12:45.4769953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4770050Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4770321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4770429Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4770717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4770825Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4770829Z 2025-09-07T07:12:45.4770934Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4771122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4771207Z return mod(**inputs) 2025-09-07T07:12:45.4771517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4771585Z outputs = self.mobilebert( 2025-09-07T07:12:45.4771861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4771930Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4772211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4772279Z layer_outputs = layer_module( 2025-09-07T07:12:45.4772544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4772645Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4772916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4773051Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4773330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4773412Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4773422Z 2025-09-07T07:12:45.4773519Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4773709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4773779Z return mod(**inputs) 2025-09-07T07:12:45.4774047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4774122Z outputs = self.mobilebert( 2025-09-07T07:12:45.4774394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4774467Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4774749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4774820Z layer_outputs = layer_module( 2025-09-07T07:12:45.4775100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4775226Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4775499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4775634Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4775912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4776039Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4776324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4776425Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4776428Z 2025-09-07T07:12:45.4776531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4776735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4776810Z return mod(**inputs) 2025-09-07T07:12:45.4777094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4777182Z outputs = self.mobilebert( 2025-09-07T07:12:45.4777484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4777560Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4777868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4777941Z layer_outputs = layer_module( 2025-09-07T07:12:45.4778231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4778327Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4778616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4778729Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4779008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4779101Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4779104Z 2025-09-07T07:12:45.4779216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4779414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4779478Z return mod(**inputs) 2025-09-07T07:12:45.4779752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4779831Z outputs = self.mobilebert( 2025-09-07T07:12:45.4780103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4780181Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4780459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4780539Z layer_outputs = layer_module( 2025-09-07T07:12:45.4780821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4780914Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4781200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4781311Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4781638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4781748Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4781752Z 2025-09-07T07:12:45.4781853Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4782058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4782126Z return mod(**inputs) 2025-09-07T07:12:45.4782419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4782489Z outputs = self.mobilebert( 2025-09-07T07:12:45.4782778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4782850Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4783135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4783216Z layer_outputs = layer_module( 2025-09-07T07:12:45.4783497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4783600Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4783915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4784065Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4784357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4784445Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4784448Z 2025-09-07T07:12:45.4784559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4784762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4784838Z return mod(**inputs) 2025-09-07T07:12:45.4785126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4785198Z outputs = self.mobilebert( 2025-09-07T07:12:45.4785495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4785572Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4785939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4786017Z layer_outputs = layer_module( 2025-09-07T07:12:45.4786301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4786413Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4786709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4786853Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4787156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4787297Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4787600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4787695Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4787698Z 2025-09-07T07:12:45.4787813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4788069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4788145Z return mod(**inputs) 2025-09-07T07:12:45.4788431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4788501Z outputs = self.mobilebert( 2025-09-07T07:12:45.4788792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4788864Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4789151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4789220Z layer_outputs = layer_module( 2025-09-07T07:12:45.4789503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4789598Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4789874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4789991Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4790269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4790380Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4790384Z 2025-09-07T07:12:45.4790500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4790693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4790763Z return mod(**inputs) 2025-09-07T07:12:45.4791040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4791120Z outputs = self.mobilebert( 2025-09-07T07:12:45.4791395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4791472Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4791749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4791821Z layer_outputs = layer_module( 2025-09-07T07:12:45.4792110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4792202Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4792484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4792593Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4792878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4792996Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4792999Z 2025-09-07T07:12:45.4793099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4793301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4793366Z return mod(**inputs) 2025-09-07T07:12:45.4793655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4793726Z outputs = self.mobilebert( 2025-09-07T07:12:45.4794002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4794082Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4794358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4794468Z layer_outputs = layer_module( 2025-09-07T07:12:45.4794744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4794835Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4795123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4795248Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4795533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4795616Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4795619Z 2025-09-07T07:12:45.4795727Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4795926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4795990Z return mod(**inputs) 2025-09-07T07:12:45.4796278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4796346Z outputs = self.mobilebert( 2025-09-07T07:12:45.4796650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4796724Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4797013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4797093Z layer_outputs = layer_module( 2025-09-07T07:12:45.4797367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4797469Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4797746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4797877Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4798155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4798275Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4798557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4798647Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4798650Z 2025-09-07T07:12:45.4798757Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4798956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4799021Z return mod(**inputs) 2025-09-07T07:12:45.4799314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4799381Z outputs = self.mobilebert( 2025-09-07T07:12:45.4799658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4799729Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4800005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4800073Z layer_outputs = layer_module( 2025-09-07T07:12:45.4800341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4800498Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4800768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4800858Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4800862Z 2025-09-07T07:12:45.4800961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4801156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4801231Z return mod(**inputs) 2025-09-07T07:12:45.4801510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4801587Z outputs = self.mobilebert( 2025-09-07T07:12:45.4801858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4801939Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4802211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4802282Z layer_outputs = layer_module( 2025-09-07T07:12:45.4802560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4802691Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4802981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4803088Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4803091Z 2025-09-07T07:12:45.4803198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4803385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4803453Z return mod(**inputs) 2025-09-07T07:12:45.4803729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4803795Z outputs = self.mobilebert( 2025-09-07T07:12:45.4804070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4804140Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4804407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4804485Z layer_outputs = layer_module( 2025-09-07T07:12:45.4804752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4804916Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4805184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4805280Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4805290Z 2025-09-07T07:12:45.4805393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4805591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4805665Z return mod(**inputs) 2025-09-07T07:12:45.4805952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4806029Z outputs = self.mobilebert( 2025-09-07T07:12:45.4806312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4806385Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4806685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4806792Z layer_outputs = layer_module( 2025-09-07T07:12:45.4807071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4807229Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4807508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4807639Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4807916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4808014Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4808017Z 2025-09-07T07:12:45.4808124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4808320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4808384Z return mod(**inputs) 2025-09-07T07:12:45.4808663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4808738Z outputs = self.mobilebert( 2025-09-07T07:12:45.4809045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4809141Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4809417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4809486Z layer_outputs = layer_module( 2025-09-07T07:12:45.4809768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4809926Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4810206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4810326Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4810611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4810700Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4810704Z 2025-09-07T07:12:45.4810803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4811007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4811071Z return mod(**inputs) 2025-09-07T07:12:45.4811358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4811430Z outputs = self.mobilebert( 2025-09-07T07:12:45.4811709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4811782Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4812069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4812147Z layer_outputs = layer_module( 2025-09-07T07:12:45.4812421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4812584Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4812859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4813010Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4813291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4813408Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4813697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4813787Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4813792Z 2025-09-07T07:12:45.4813899Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4814097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4814161Z return mod(**inputs) 2025-09-07T07:12:45.4814450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4814522Z outputs = self.mobilebert( 2025-09-07T07:12:45.4814809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4814880Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4815173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4815252Z layer_outputs = layer_module( 2025-09-07T07:12:45.4815543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4815711Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4815995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4816118Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4816407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4816492Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4816495Z 2025-09-07T07:12:45.4816607Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4816811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4816884Z return mod(**inputs) 2025-09-07T07:12:45.4817184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4817254Z outputs = self.mobilebert( 2025-09-07T07:12:45.4817547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4817621Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4817915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4817984Z layer_outputs = layer_module( 2025-09-07T07:12:45.4818273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4818437Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4818721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4818839Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4819125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4819250Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4819525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4819820Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4819832Z 2025-09-07T07:12:45.4819940Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4820140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4820216Z return mod(**inputs) 2025-09-07T07:12:45.4820502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4820585Z outputs = self.mobilebert( 2025-09-07T07:12:45.4820864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4820938Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4821231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4821303Z layer_outputs = layer_module( 2025-09-07T07:12:45.4821589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4821676Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4821997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4822098Z self_outputs = self.self( 2025-09-07T07:12:45.4822382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4822461Z self.query(query_tensor) 2025-09-07T07:12:45.4822464Z 2025-09-07T07:12:45.4822566Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4822775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4822840Z return mod(**inputs) 2025-09-07T07:12:45.4823130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4823210Z outputs = self.mobilebert( 2025-09-07T07:12:45.4823502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4823583Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4823872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4823945Z layer_outputs = layer_module( 2025-09-07T07:12:45.4824241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4824332Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4824624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4824696Z self_outputs = self.self( 2025-09-07T07:12:45.4824987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4825057Z self.key(key_tensor) 2025-09-07T07:12:45.4825060Z 2025-09-07T07:12:45.4825163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4825372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4825438Z return mod(**inputs) 2025-09-07T07:12:45.4825773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4825900Z outputs = self.mobilebert( 2025-09-07T07:12:45.4826187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4826269Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4826561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4826640Z layer_outputs = layer_module( 2025-09-07T07:12:45.4826933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4827020Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4827314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4827383Z self_outputs = self.self( 2025-09-07T07:12:45.4827680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4827758Z self.value(value_tensor) 2025-09-07T07:12:45.4827762Z 2025-09-07T07:12:45.4827860Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4827941Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4828044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4828265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4828332Z return mod(**inputs) 2025-09-07T07:12:45.4828631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4828704Z outputs = self.mobilebert( 2025-09-07T07:12:45.4828985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4829066Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4829346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4829421Z layer_outputs = layer_module( 2025-09-07T07:12:45.4829695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4829776Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4830058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4830181Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4830464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4830549Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4830555Z 2025-09-07T07:12:45.4830661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4830854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4830918Z return mod(**inputs) 2025-09-07T07:12:45.4831205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4831275Z outputs = self.mobilebert( 2025-09-07T07:12:45.4831561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4831634Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4831908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4831986Z layer_outputs = layer_module( 2025-09-07T07:12:45.4832262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4832473Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4832751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4832870Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4833150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4833234Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4833237Z 2025-09-07T07:12:45.4833345Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4833541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4833613Z return mod(**inputs) 2025-09-07T07:12:45.4833893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4833976Z outputs = self.mobilebert( 2025-09-07T07:12:45.4834252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4834324Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4834626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4834697Z layer_outputs = layer_module( 2025-09-07T07:12:45.4834998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4835082Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4835361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4835493Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4835768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4835899Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4836175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4836275Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4836279Z 2025-09-07T07:12:45.4836379Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4836573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4836645Z return mod(**inputs) 2025-09-07T07:12:45.4836924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4837003Z outputs = self.mobilebert( 2025-09-07T07:12:45.4837280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4837351Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4837634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4837704Z layer_outputs = layer_module( 2025-09-07T07:12:45.4837986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4838080Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4838354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4838506Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4838794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4838882Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4838885Z 2025-09-07T07:12:45.4838982Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4839180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4839242Z return mod(**inputs) 2025-09-07T07:12:45.4839518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4839594Z outputs = self.mobilebert( 2025-09-07T07:12:45.4839864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4839943Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4840227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4840298Z layer_outputs = layer_module( 2025-09-07T07:12:45.4840591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4840682Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4840974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4841098Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4841378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4841486Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4841492Z 2025-09-07T07:12:45.4841591Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4841795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4841860Z return mod(**inputs) 2025-09-07T07:12:45.4842146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4842214Z outputs = self.mobilebert( 2025-09-07T07:12:45.4842499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4842578Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4842846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4842922Z layer_outputs = layer_module( 2025-09-07T07:12:45.4843191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4843290Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4843558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4843679Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4843959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4844043Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4844046Z 2025-09-07T07:12:45.4844150Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4844337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4844405Z return mod(**inputs) 2025-09-07T07:12:45.4844676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4844778Z outputs = self.mobilebert( 2025-09-07T07:12:45.4845052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4845121Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4845396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4845464Z layer_outputs = layer_module( 2025-09-07T07:12:45.4845735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4845831Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4846101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4846233Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4846506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4846635Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4846928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4847030Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4847034Z 2025-09-07T07:12:45.4847154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4847344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4847416Z return mod(**inputs) 2025-09-07T07:12:45.4847688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4847758Z outputs = self.mobilebert( 2025-09-07T07:12:45.4848034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4848106Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4848384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4848453Z layer_outputs = layer_module( 2025-09-07T07:12:45.4848726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4848816Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4849087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4849205Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4849475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4849564Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4849567Z 2025-09-07T07:12:45.4849665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4849855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4849928Z return mod(**inputs) 2025-09-07T07:12:45.4850208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4850285Z outputs = self.mobilebert( 2025-09-07T07:12:45.4850571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4850647Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4850947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4851018Z layer_outputs = layer_module( 2025-09-07T07:12:45.4851294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4851385Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4851661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4851777Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4852046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4852158Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4852164Z 2025-09-07T07:12:45.4852262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4852460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4852522Z return mod(**inputs) 2025-09-07T07:12:45.4852806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4852873Z outputs = self.mobilebert( 2025-09-07T07:12:45.4853156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4853249Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4853519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4853596Z layer_outputs = layer_module( 2025-09-07T07:12:45.4853863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4853956Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4854232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4854353Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4854630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4854712Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4854716Z 2025-09-07T07:12:45.4854820Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4855010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4855074Z return mod(**inputs) 2025-09-07T07:12:45.4855353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4855423Z outputs = self.mobilebert( 2025-09-07T07:12:45.4855696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4855766Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4856040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4856119Z layer_outputs = layer_module( 2025-09-07T07:12:45.4856396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4856494Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4856769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4856971Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4857248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4857365Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4857648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4857739Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4857742Z 2025-09-07T07:12:45.4857851Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4858044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4858108Z return mod(**inputs) 2025-09-07T07:12:45.4858392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4858465Z outputs = self.mobilebert( 2025-09-07T07:12:45.4858746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4858817Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4859114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4859186Z layer_outputs = layer_module( 2025-09-07T07:12:45.4859479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4859580Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4859855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4859975Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4860251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4860335Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4860338Z 2025-09-07T07:12:45.4860444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4860640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4860711Z return mod(**inputs) 2025-09-07T07:12:45.4860989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4861067Z outputs = self.mobilebert( 2025-09-07T07:12:45.4861341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4861412Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4861699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4861770Z layer_outputs = layer_module( 2025-09-07T07:12:45.4862052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4862146Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4862424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4862544Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4862823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4862939Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4862943Z 2025-09-07T07:12:45.4863076Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4863280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4863344Z return mod(**inputs) 2025-09-07T07:12:45.4863624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4863703Z outputs = self.mobilebert( 2025-09-07T07:12:45.4863984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4864065Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4864347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4864420Z layer_outputs = layer_module( 2025-09-07T07:12:45.4864712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4864809Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4865100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4865228Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4865535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4865624Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4865642Z 2025-09-07T07:12:45.4865823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4866042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4866113Z return mod(**inputs) 2025-09-07T07:12:45.4866423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4866506Z outputs = self.mobilebert( 2025-09-07T07:12:45.4866817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4866901Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4867188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4867272Z layer_outputs = layer_module( 2025-09-07T07:12:45.4867557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4867660Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4867946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4868077Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4868374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4868495Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4868783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4868874Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4868878Z 2025-09-07T07:12:45.4868980Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4869184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4869251Z return mod(**inputs) 2025-09-07T07:12:45.4869538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4869643Z outputs = self.mobilebert( 2025-09-07T07:12:45.4869933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4870005Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4870286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4870366Z layer_outputs = layer_module( 2025-09-07T07:12:45.4870649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4870779Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4871063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4871147Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4871160Z 2025-09-07T07:12:45.4871262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4871461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4871530Z return mod(**inputs) 2025-09-07T07:12:45.4871820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4871912Z outputs = self.mobilebert( 2025-09-07T07:12:45.4872203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4872276Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4872569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4872639Z layer_outputs = layer_module( 2025-09-07T07:12:45.4872921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4873037Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4873312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4873429Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4873433Z 2025-09-07T07:12:45.4873531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4873730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4873794Z return mod(**inputs) 2025-09-07T07:12:45.4874078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4874147Z outputs = self.mobilebert( 2025-09-07T07:12:45.4874424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4874500Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4874775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4874851Z layer_outputs = layer_module( 2025-09-07T07:12:45.4875126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4875290Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4875570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4875664Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4875667Z 2025-09-07T07:12:45.4875809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4876003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4876073Z return mod(**inputs) 2025-09-07T07:12:45.4876354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4876422Z outputs = self.mobilebert( 2025-09-07T07:12:45.4876709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4876780Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4877067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4877136Z layer_outputs = layer_module( 2025-09-07T07:12:45.4877413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4877581Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4877859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4877988Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4878302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4878437Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4878441Z 2025-09-07T07:12:45.4878542Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4878736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4878808Z return mod(**inputs) 2025-09-07T07:12:45.4879088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4879168Z outputs = self.mobilebert( 2025-09-07T07:12:45.4879443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4879514Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4879797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4879868Z layer_outputs = layer_module( 2025-09-07T07:12:45.4880154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4880311Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4880597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4880723Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4881002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4881098Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4881103Z 2025-09-07T07:12:45.4881206Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4881408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4881474Z return mod(**inputs) 2025-09-07T07:12:45.4881757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4881834Z outputs = self.mobilebert( 2025-09-07T07:12:45.4882109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4882221Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4882546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4882620Z layer_outputs = layer_module( 2025-09-07T07:12:45.4882903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4883056Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4883333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4883450Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4883728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4883847Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4884123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4884212Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4884216Z 2025-09-07T07:12:45.4884313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4884530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4884609Z return mod(**inputs) 2025-09-07T07:12:45.4884889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4884958Z outputs = self.mobilebert( 2025-09-07T07:12:45.4885235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4885317Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4885592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4885669Z layer_outputs = layer_module( 2025-09-07T07:12:45.4885947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4886116Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4886404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4886518Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4886817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4886901Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4886904Z 2025-09-07T07:12:45.4887012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4887204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4887269Z return mod(**inputs) 2025-09-07T07:12:45.4887557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4887626Z outputs = self.mobilebert( 2025-09-07T07:12:45.4887909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4887985Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4888293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4888404Z layer_outputs = layer_module( 2025-09-07T07:12:45.4888703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4888882Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4889180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4889305Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4889614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4889709Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4890034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4890130Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4890137Z 2025-09-07T07:12:45.4890250Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4890460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4890536Z return mod(**inputs) 2025-09-07T07:12:45.4890845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4890939Z outputs = self.mobilebert( 2025-09-07T07:12:45.4891272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4891353Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4891672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4891748Z layer_outputs = layer_module( 2025-09-07T07:12:45.4892053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4892152Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4892451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4892533Z self_outputs = self.self( 2025-09-07T07:12:45.4892832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4892911Z self.query(query_tensor) 2025-09-07T07:12:45.4892916Z 2025-09-07T07:12:45.4893019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4893218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4893292Z return mod(**inputs) 2025-09-07T07:12:45.4893597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4893683Z outputs = self.mobilebert( 2025-09-07T07:12:45.4893983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4894058Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4894366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4894441Z layer_outputs = layer_module( 2025-09-07T07:12:45.4894746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4894838Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4895138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4895254Z self_outputs = self.self( 2025-09-07T07:12:45.4895553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4895631Z self.key(key_tensor) 2025-09-07T07:12:45.4895635Z 2025-09-07T07:12:45.4895744Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4895964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4896034Z return mod(**inputs) 2025-09-07T07:12:45.4896337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4896420Z outputs = self.mobilebert( 2025-09-07T07:12:45.4896718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4896802Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4897103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4897179Z layer_outputs = layer_module( 2025-09-07T07:12:45.4897484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4897574Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4897896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4897986Z self_outputs = self.self( 2025-09-07T07:12:45.4898297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4898372Z self.value(value_tensor) 2025-09-07T07:12:45.4898376Z 2025-09-07T07:12:45.4898466Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4898561Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4898668Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4898887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4898958Z return mod(**inputs) 2025-09-07T07:12:45.4899262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4899345Z outputs = self.mobilebert( 2025-09-07T07:12:45.4899647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4899731Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4900031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4900107Z layer_outputs = layer_module( 2025-09-07T07:12:45.4900420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4900510Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4900819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4900952Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4901260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4901353Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4901357Z 2025-09-07T07:12:45.4901465Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4901683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4901752Z return mod(**inputs) 2025-09-07T07:12:45.4902093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4902167Z outputs = self.mobilebert( 2025-09-07T07:12:45.4902466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4902549Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4902851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4902934Z layer_outputs = layer_module( 2025-09-07T07:12:45.4903232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4903414Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4903714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4903835Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4904140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4904228Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4904232Z 2025-09-07T07:12:45.4904373Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4904590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4904676Z return mod(**inputs) 2025-09-07T07:12:45.4905001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4905078Z outputs = self.mobilebert( 2025-09-07T07:12:45.4905399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4905478Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4905853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4905936Z layer_outputs = layer_module( 2025-09-07T07:12:45.4906236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4906336Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4906642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4906786Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4907095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4907239Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4907559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4907661Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4907665Z 2025-09-07T07:12:45.4907782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4907999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4908081Z return mod(**inputs) 2025-09-07T07:12:45.4908396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4908475Z outputs = self.mobilebert( 2025-09-07T07:12:45.4908796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4908913Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4909239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4909314Z layer_outputs = layer_module( 2025-09-07T07:12:45.4909614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4909727Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4910029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4910158Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4910460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4910559Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4910565Z 2025-09-07T07:12:45.4910673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4910884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4910963Z return mod(**inputs) 2025-09-07T07:12:45.4911267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4911367Z outputs = self.mobilebert( 2025-09-07T07:12:45.4911680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4911760Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4912066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4912141Z layer_outputs = layer_module( 2025-09-07T07:12:45.4912458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4912560Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4912869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4912988Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4913289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4913419Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4913423Z 2025-09-07T07:12:45.4913530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4913747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4913816Z return mod(**inputs) 2025-09-07T07:12:45.4914122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4914203Z outputs = self.mobilebert( 2025-09-07T07:12:45.4914505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4914591Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4914891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4914976Z layer_outputs = layer_module( 2025-09-07T07:12:45.4915274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4915379Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4915697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4915870Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4916202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4916296Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4916300Z 2025-09-07T07:12:45.4916412Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4916636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4916710Z return mod(**inputs) 2025-09-07T07:12:45.4917032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4917108Z outputs = self.mobilebert( 2025-09-07T07:12:45.4917418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4917505Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4917811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4917898Z layer_outputs = layer_module( 2025-09-07T07:12:45.4918226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4918338Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4918664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4918807Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4919135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4919275Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4919744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4919855Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4919859Z 2025-09-07T07:12:45.4919982Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4920209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4920283Z return mod(**inputs) 2025-09-07T07:12:45.4920605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4920684Z outputs = self.mobilebert( 2025-09-07T07:12:45.4920997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4921081Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4921391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4921479Z layer_outputs = layer_module( 2025-09-07T07:12:45.4921788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4921900Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4922209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4922342Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4922651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4922744Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4922825Z 2025-09-07T07:12:45.4922948Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4923166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4923244Z return mod(**inputs) 2025-09-07T07:12:45.4923556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4923637Z outputs = self.mobilebert( 2025-09-07T07:12:45.4923958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4924039Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4924364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4924442Z layer_outputs = layer_module( 2025-09-07T07:12:45.4924763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4924867Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4925178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4925312Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4925647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4925799Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4925803Z 2025-09-07T07:12:45.4925916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4926120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4926186Z return mod(**inputs) 2025-09-07T07:12:45.4926469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4926547Z outputs = self.mobilebert( 2025-09-07T07:12:45.4926824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4926900Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4927185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4927255Z layer_outputs = layer_module( 2025-09-07T07:12:45.4927533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4927624Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4927907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4928034Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4928308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4928406Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4928409Z 2025-09-07T07:12:45.4928507Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4928706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4928771Z return mod(**inputs) 2025-09-07T07:12:45.4929051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4929120Z outputs = self.mobilebert( 2025-09-07T07:12:45.4929387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4929507Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4929777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4929854Z layer_outputs = layer_module( 2025-09-07T07:12:45.4930125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4930216Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4930493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4930615Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4930894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4931016Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4931296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4931386Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4931389Z 2025-09-07T07:12:45.4931487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4931702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4931767Z return mod(**inputs) 2025-09-07T07:12:45.4932060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4932129Z outputs = self.mobilebert( 2025-09-07T07:12:45.4932398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4932481Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4932760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4932838Z layer_outputs = layer_module( 2025-09-07T07:12:45.4933131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4933228Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4933497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4933608Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4933893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4933977Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4933982Z 2025-09-07T07:12:45.4934089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4934285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4934361Z return mod(**inputs) 2025-09-07T07:12:45.4934646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4934716Z outputs = self.mobilebert( 2025-09-07T07:12:45.4934997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4935069Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4935356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4935425Z layer_outputs = layer_module( 2025-09-07T07:12:45.4935704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4935833Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4936104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4936220Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4936491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4936598Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4936608Z 2025-09-07T07:12:45.4936710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4936905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4936977Z return mod(**inputs) 2025-09-07T07:12:45.4937260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4937346Z outputs = self.mobilebert( 2025-09-07T07:12:45.4937617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4937689Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4937983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4938069Z layer_outputs = layer_module( 2025-09-07T07:12:45.4938348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4938438Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4938715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4938848Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4939125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4939218Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4939221Z 2025-09-07T07:12:45.4939324Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4939530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4939598Z return mod(**inputs) 2025-09-07T07:12:45.4939888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4939967Z outputs = self.mobilebert( 2025-09-07T07:12:45.4940251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4940334Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4940611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4940680Z layer_outputs = layer_module( 2025-09-07T07:12:45.4940966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4941058Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4941343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4941464Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4941746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4941897Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4942185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4942284Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4942287Z 2025-09-07T07:12:45.4942391Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4942602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4942668Z return mod(**inputs) 2025-09-07T07:12:45.4942961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4943040Z outputs = self.mobilebert( 2025-09-07T07:12:45.4943332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4943414Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4943705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4943786Z layer_outputs = layer_module( 2025-09-07T07:12:45.4944076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4944215Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4944528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4944616Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4944619Z 2025-09-07T07:12:45.4944729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4944928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4944997Z return mod(**inputs) 2025-09-07T07:12:45.4945293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4945364Z outputs = self.mobilebert( 2025-09-07T07:12:45.4945655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4945788Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4946119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4946199Z layer_outputs = layer_module( 2025-09-07T07:12:45.4946522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.4946671Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.4946974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4947104Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4947108Z 2025-09-07T07:12:45.4947224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4947428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4947496Z return mod(**inputs) 2025-09-07T07:12:45.4947777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4947855Z outputs = self.mobilebert( 2025-09-07T07:12:45.4948133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4948213Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4948492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4948601Z layer_outputs = layer_module( 2025-09-07T07:12:45.4948884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4949044Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4949327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.4949424Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.4949428Z 2025-09-07T07:12:45.4949534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4949729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4949794Z return mod(**inputs) 2025-09-07T07:12:45.4950084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4950153Z outputs = self.mobilebert( 2025-09-07T07:12:45.4950438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4950509Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4950805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4950886Z layer_outputs = layer_module( 2025-09-07T07:12:45.4951174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4951343Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4951624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.4951755Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.4952032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4952123Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4952127Z 2025-09-07T07:12:45.4952238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4952434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4952507Z return mod(**inputs) 2025-09-07T07:12:45.4952795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4952865Z outputs = self.mobilebert( 2025-09-07T07:12:45.4953151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4953226Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4953509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4953578Z layer_outputs = layer_module( 2025-09-07T07:12:45.4953857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4954021Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4954299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4954430Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4954708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.4954836Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4954840Z 2025-09-07T07:12:45.4954942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4955135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4955207Z return mod(**inputs) 2025-09-07T07:12:45.4955485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4955561Z outputs = self.mobilebert( 2025-09-07T07:12:45.4955839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4955918Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4956192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4956263Z layer_outputs = layer_module( 2025-09-07T07:12:45.4956549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.4956700Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.4956990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.4957112Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.4957412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.4957539Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4957813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4957915Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4957918Z 2025-09-07T07:12:45.4958018Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4958217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4958281Z return mod(**inputs) 2025-09-07T07:12:45.4958557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4958632Z outputs = self.mobilebert( 2025-09-07T07:12:45.4958906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4958984Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4959254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4959327Z layer_outputs = layer_module( 2025-09-07T07:12:45.4959606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4959766Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4960057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4960163Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4960434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4960514Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4960517Z 2025-09-07T07:12:45.4960613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4960809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4960932Z return mod(**inputs) 2025-09-07T07:12:45.4961211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4961279Z outputs = self.mobilebert( 2025-09-07T07:12:45.4961548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4961626Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4961894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4961970Z layer_outputs = layer_module( 2025-09-07T07:12:45.4962238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4962398Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4962670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.4962775Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.4963050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.4963149Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.4963443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4963538Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4963542Z 2025-09-07T07:12:45.4963651Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4963846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4963915Z return mod(**inputs) 2025-09-07T07:12:45.4964203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4964274Z outputs = self.mobilebert( 2025-09-07T07:12:45.4964555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4964629Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4964911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4964993Z layer_outputs = layer_module( 2025-09-07T07:12:45.4965275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4965367Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4965662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4965735Z self_outputs = self.self( 2025-09-07T07:12:45.4966021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.4966093Z self.query(query_tensor) 2025-09-07T07:12:45.4966097Z 2025-09-07T07:12:45.4966208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4966407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4966481Z return mod(**inputs) 2025-09-07T07:12:45.4966771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4966841Z outputs = self.mobilebert( 2025-09-07T07:12:45.4967122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4967224Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4967506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4967576Z layer_outputs = layer_module( 2025-09-07T07:12:45.4967861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4967950Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4968218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4968292Z self_outputs = self.self( 2025-09-07T07:12:45.4968561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.4968631Z self.key(key_tensor) 2025-09-07T07:12:45.4968637Z 2025-09-07T07:12:45.4968736Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4968927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4968998Z return mod(**inputs) 2025-09-07T07:12:45.4969279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4969371Z outputs = self.mobilebert( 2025-09-07T07:12:45.4969660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4969734Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4970021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4970091Z layer_outputs = layer_module( 2025-09-07T07:12:45.4970383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4970465Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4970742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.4970818Z self_outputs = self.self( 2025-09-07T07:12:45.4971097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.4971175Z self.value(value_tensor) 2025-09-07T07:12:45.4971180Z 2025-09-07T07:12:45.4971263Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4971348Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.4971449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4971646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4971721Z return mod(**inputs) 2025-09-07T07:12:45.4972000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4972075Z outputs = self.mobilebert( 2025-09-07T07:12:45.4972349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4972425Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4972710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4972778Z layer_outputs = layer_module( 2025-09-07T07:12:45.4973059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4973141Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4973424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4973579Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4973855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.4973947Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4973950Z 2025-09-07T07:12:45.4974051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4974253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4974317Z return mod(**inputs) 2025-09-07T07:12:45.4974593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4974668Z outputs = self.mobilebert( 2025-09-07T07:12:45.4974942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4975025Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4975305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4975378Z layer_outputs = layer_module( 2025-09-07T07:12:45.4975684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.4975872Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.4976174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.4976288Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.4976577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.4976664Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.4976668Z 2025-09-07T07:12:45.4976770Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4976981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4977046Z return mod(**inputs) 2025-09-07T07:12:45.4977344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4977418Z outputs = self.mobilebert( 2025-09-07T07:12:45.4977713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4977788Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4978075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4978160Z layer_outputs = layer_module( 2025-09-07T07:12:45.4978450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.4978544Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.4978834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.4978960Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.4979257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.4979388Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4979680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4979803Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4979806Z 2025-09-07T07:12:45.4979918Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4980120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4980200Z return mod(**inputs) 2025-09-07T07:12:45.4980487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4980555Z outputs = self.mobilebert( 2025-09-07T07:12:45.4980842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4980916Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4981204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4981289Z layer_outputs = layer_module( 2025-09-07T07:12:45.4981574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4981678Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4981962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4982100Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4982405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4982492Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4982496Z 2025-09-07T07:12:45.4982607Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4982808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4982885Z return mod(**inputs) 2025-09-07T07:12:45.4983173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4983243Z outputs = self.mobilebert( 2025-09-07T07:12:45.4983536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4983609Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4983900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4983974Z layer_outputs = layer_module( 2025-09-07T07:12:45.4984266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4984362Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4984647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4984773Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4985074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4985201Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4985205Z 2025-09-07T07:12:45.4985315Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4985531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4985606Z return mod(**inputs) 2025-09-07T07:12:45.4985959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4986046Z outputs = self.mobilebert( 2025-09-07T07:12:45.4986345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4986470Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4986773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4986850Z layer_outputs = layer_module( 2025-09-07T07:12:45.4987159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4987261Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4987570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4987710Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4988022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4988123Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4988127Z 2025-09-07T07:12:45.4988232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4988444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4988510Z return mod(**inputs) 2025-09-07T07:12:45.4988819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4988891Z outputs = self.mobilebert( 2025-09-07T07:12:45.4989187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4989269Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4989555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4989638Z layer_outputs = layer_module( 2025-09-07T07:12:45.4989921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4990016Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4990309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4990439Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4990731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.4990854Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.4991152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.4991248Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.4991251Z 2025-09-07T07:12:45.4991353Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4991560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4991627Z return mod(**inputs) 2025-09-07T07:12:45.4991923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4991993Z outputs = self.mobilebert( 2025-09-07T07:12:45.4992278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4992358Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4992640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4992719Z layer_outputs = layer_module( 2025-09-07T07:12:45.4993033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4993135Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4993422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4993536Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4993826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.4993910Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.4993914Z 2025-09-07T07:12:45.4994022Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4994222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4994292Z return mod(**inputs) 2025-09-07T07:12:45.4994582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4994653Z outputs = self.mobilebert( 2025-09-07T07:12:45.4994946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4995021Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4995330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4995417Z layer_outputs = layer_module( 2025-09-07T07:12:45.4995701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4995806Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4996111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.4996240Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.4996541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.4996659Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.4996670Z 2025-09-07T07:12:45.4996780Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4996994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4997070Z return mod(**inputs) 2025-09-07T07:12:45.4997370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.4997446Z outputs = self.mobilebert( 2025-09-07T07:12:45.4997736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.4997816Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.4998122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.4998197Z layer_outputs = layer_module( 2025-09-07T07:12:45.4998503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.4998602Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.4998904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.4999046Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.4999344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.4999480Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.4999484Z 2025-09-07T07:12:45.4999600Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.4999805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.4999871Z return mod(**inputs) 2025-09-07T07:12:45.5000160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5000242Z outputs = self.mobilebert( 2025-09-07T07:12:45.5000541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5000625Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5000931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5001006Z layer_outputs = layer_module( 2025-09-07T07:12:45.5001298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5001391Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5001682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5001831Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5002138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5002264Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5002548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5002654Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5002657Z 2025-09-07T07:12:45.5002761Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5002968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5003034Z return mod(**inputs) 2025-09-07T07:12:45.5003322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5003401Z outputs = self.mobilebert( 2025-09-07T07:12:45.5003687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5003767Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5004049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5004128Z layer_outputs = layer_module( 2025-09-07T07:12:45.5004414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5004508Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5004798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5004915Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5005209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5005295Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5005298Z 2025-09-07T07:12:45.5005400Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5005613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5005717Z return mod(**inputs) 2025-09-07T07:12:45.5006030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5006105Z outputs = self.mobilebert( 2025-09-07T07:12:45.5006412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5006490Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5006794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5006879Z layer_outputs = layer_module( 2025-09-07T07:12:45.5007181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5007286Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5007587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5007709Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5008017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5008138Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5008141Z 2025-09-07T07:12:45.5008276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5008504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5008584Z return mod(**inputs) 2025-09-07T07:12:45.5008894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5008969Z outputs = self.mobilebert( 2025-09-07T07:12:45.5009275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5009355Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5009663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5009739Z layer_outputs = layer_module( 2025-09-07T07:12:45.5010050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5010158Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5010460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5010601Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5010906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5011008Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5011012Z 2025-09-07T07:12:45.5011121Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5011334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5011414Z return mod(**inputs) 2025-09-07T07:12:45.5011719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5011803Z outputs = self.mobilebert( 2025-09-07T07:12:45.5012102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5012182Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5012489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5012600Z layer_outputs = layer_module( 2025-09-07T07:12:45.5012910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5013010Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5013319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5013457Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5013761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5013898Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5014198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5014309Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5014313Z 2025-09-07T07:12:45.5014422Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5014640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5014720Z return mod(**inputs) 2025-09-07T07:12:45.5015044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5015129Z outputs = self.mobilebert( 2025-09-07T07:12:45.5015444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5015531Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5015834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5015911Z layer_outputs = layer_module( 2025-09-07T07:12:45.5016222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5016352Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5016661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5016753Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5016756Z 2025-09-07T07:12:45.5016868Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5017090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5017158Z return mod(**inputs) 2025-09-07T07:12:45.5017473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5017549Z outputs = self.mobilebert( 2025-09-07T07:12:45.5017860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5017939Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5018241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5018323Z layer_outputs = layer_module( 2025-09-07T07:12:45.5018631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5018767Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5019069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5019185Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5019197Z 2025-09-07T07:12:45.5019339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5019740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5019830Z return mod(**inputs) 2025-09-07T07:12:45.5020134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5020219Z outputs = self.mobilebert( 2025-09-07T07:12:45.5020524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5020603Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5020911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5020987Z layer_outputs = layer_module( 2025-09-07T07:12:45.5021292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5021468Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5021769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5021881Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5021885Z 2025-09-07T07:12:45.5022031Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5022273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5022345Z return mod(**inputs) 2025-09-07T07:12:45.5022659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5022732Z outputs = self.mobilebert( 2025-09-07T07:12:45.5023031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5023120Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5023420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5023504Z layer_outputs = layer_module( 2025-09-07T07:12:45.5023802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5023972Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5024276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5024408Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5024712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5024813Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5024818Z 2025-09-07T07:12:45.5024935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5025145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5025216Z return mod(**inputs) 2025-09-07T07:12:45.5025527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5025601Z outputs = self.mobilebert( 2025-09-07T07:12:45.5025967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5026051Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5026358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5026510Z layer_outputs = layer_module( 2025-09-07T07:12:45.5026856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5027022Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5027300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5027431Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5027705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5027790Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5027794Z 2025-09-07T07:12:45.5027902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5028098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5028176Z return mod(**inputs) 2025-09-07T07:12:45.5028455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5028524Z outputs = self.mobilebert( 2025-09-07T07:12:45.5028822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5028895Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5029208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5029281Z layer_outputs = layer_module( 2025-09-07T07:12:45.5029579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5029740Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5030027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5030156Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5030434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5030561Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5030840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5030941Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5030944Z 2025-09-07T07:12:45.5031046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5031242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5031317Z return mod(**inputs) 2025-09-07T07:12:45.5031598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5031677Z outputs = self.mobilebert( 2025-09-07T07:12:45.5031956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5032030Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5032317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5032388Z layer_outputs = layer_module( 2025-09-07T07:12:45.5032669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5032833Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5033351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5033463Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5033740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5033835Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5033839Z 2025-09-07T07:12:45.5033943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5034144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5034208Z return mod(**inputs) 2025-09-07T07:12:45.5034490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5034571Z outputs = self.mobilebert( 2025-09-07T07:12:45.5034847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5034928Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5035206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5035301Z layer_outputs = layer_module( 2025-09-07T07:12:45.5035595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5035756Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5036046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5036154Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5036442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5036528Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5036808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5036910Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5036914Z 2025-09-07T07:12:45.5037014Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5037229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5037294Z return mod(**inputs) 2025-09-07T07:12:45.5037572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5037641Z outputs = self.mobilebert( 2025-09-07T07:12:45.5037919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5038000Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5038272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5038350Z layer_outputs = layer_module( 2025-09-07T07:12:45.5038626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5038713Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5038995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5039065Z self_outputs = self.self( 2025-09-07T07:12:45.5039346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5039445Z self.query(query_tensor) 2025-09-07T07:12:45.5039448Z 2025-09-07T07:12:45.5039558Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5039758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5039820Z return mod(**inputs) 2025-09-07T07:12:45.5040106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5040175Z outputs = self.mobilebert( 2025-09-07T07:12:45.5040453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5040521Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5040787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5040865Z layer_outputs = layer_module( 2025-09-07T07:12:45.5041134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5041221Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5041489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5041575Z self_outputs = self.self( 2025-09-07T07:12:45.5041874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5041939Z self.key(key_tensor) 2025-09-07T07:12:45.5041942Z 2025-09-07T07:12:45.5042046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5042233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5042306Z return mod(**inputs) 2025-09-07T07:12:45.5042580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5042649Z outputs = self.mobilebert( 2025-09-07T07:12:45.5042932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5043004Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5043296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5043365Z layer_outputs = layer_module( 2025-09-07T07:12:45.5043636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5043722Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5043993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5044069Z self_outputs = self.self( 2025-09-07T07:12:45.5044340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5044415Z self.value(value_tensor) 2025-09-07T07:12:45.5044418Z 2025-09-07T07:12:45.5044498Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5044577Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5044687Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5044879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5044950Z return mod(**inputs) 2025-09-07T07:12:45.5045226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5045293Z outputs = self.mobilebert( 2025-09-07T07:12:45.5045601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5045672Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5045956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5046027Z layer_outputs = layer_module( 2025-09-07T07:12:45.5046305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5046397Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5046673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5046804Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5047091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5047188Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5047191Z 2025-09-07T07:12:45.5047294Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5047504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5047579Z return mod(**inputs) 2025-09-07T07:12:45.5047874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5047966Z outputs = self.mobilebert( 2025-09-07T07:12:45.5048251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5048320Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5048607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5048678Z layer_outputs = layer_module( 2025-09-07T07:12:45.5048954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5049110Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5049398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5049509Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5049786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5049875Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5049878Z 2025-09-07T07:12:45.5049978Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5050183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5050246Z return mod(**inputs) 2025-09-07T07:12:45.5050525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5050602Z outputs = self.mobilebert( 2025-09-07T07:12:45.5050880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5050959Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5051242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5051316Z layer_outputs = layer_module( 2025-09-07T07:12:45.5051591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5051710Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5051988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5052107Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5052385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5052506Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5052777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5052872Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5052875Z 2025-09-07T07:12:45.5052971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5053169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5053234Z return mod(**inputs) 2025-09-07T07:12:45.5053520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5053590Z outputs = self.mobilebert( 2025-09-07T07:12:45.5053867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5053962Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5054259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5054334Z layer_outputs = layer_module( 2025-09-07T07:12:45.5054602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5054695Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5054982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5055095Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5055383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5055466Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5055471Z 2025-09-07T07:12:45.5055578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5055771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5055836Z return mod(**inputs) 2025-09-07T07:12:45.5056121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5056191Z outputs = self.mobilebert( 2025-09-07T07:12:45.5056475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5056547Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5056825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5056900Z layer_outputs = layer_module( 2025-09-07T07:12:45.5057178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5057281Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5057557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5057673Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5057950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5058091Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5058096Z 2025-09-07T07:12:45.5058203Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5058395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5058465Z return mod(**inputs) 2025-09-07T07:12:45.5058747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5058817Z outputs = self.mobilebert( 2025-09-07T07:12:45.5059102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5059173Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5059455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5059526Z layer_outputs = layer_module( 2025-09-07T07:12:45.5059810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5059904Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5060194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5060328Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5060647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5060740Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5060744Z 2025-09-07T07:12:45.5060844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5061049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5061113Z return mod(**inputs) 2025-09-07T07:12:45.5061392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5061471Z outputs = self.mobilebert( 2025-09-07T07:12:45.5061753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5061831Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5062109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5062179Z layer_outputs = layer_module( 2025-09-07T07:12:45.5062462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5062558Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5062840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5062964Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5063238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5063370Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5063657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5063759Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5063762Z 2025-09-07T07:12:45.5063865Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5064075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5064170Z return mod(**inputs) 2025-09-07T07:12:45.5064455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5064537Z outputs = self.mobilebert( 2025-09-07T07:12:45.5064828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5064911Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5065246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5065321Z layer_outputs = layer_module( 2025-09-07T07:12:45.5065630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5065795Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5066135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5066260Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5066587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5066680Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5066704Z 2025-09-07T07:12:45.5066819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5067067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5067140Z return mod(**inputs) 2025-09-07T07:12:45.5067471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5067547Z outputs = self.mobilebert( 2025-09-07T07:12:45.5067854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5067940Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5068243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5068331Z layer_outputs = layer_module( 2025-09-07T07:12:45.5068635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5068746Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5069047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5069167Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5069476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5069598Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5069602Z 2025-09-07T07:12:45.5069716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5069925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5070001Z return mod(**inputs) 2025-09-07T07:12:45.5070306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5070381Z outputs = self.mobilebert( 2025-09-07T07:12:45.5070699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5070771Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5071060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5071163Z layer_outputs = layer_module( 2025-09-07T07:12:45.5071465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5071573Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5071874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5072018Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5072386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5072485Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5072489Z 2025-09-07T07:12:45.5072597Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5072816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5072892Z return mod(**inputs) 2025-09-07T07:12:45.5073194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5073274Z outputs = self.mobilebert( 2025-09-07T07:12:45.5073591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5073671Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5074004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5074081Z layer_outputs = layer_module( 2025-09-07T07:12:45.5074389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5074492Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5074799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5074944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5075270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5075414Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5075738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5075842Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5075846Z 2025-09-07T07:12:45.5075957Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5076171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5076254Z return mod(**inputs) 2025-09-07T07:12:45.5076561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5076643Z outputs = self.mobilebert( 2025-09-07T07:12:45.5076947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5077035Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5077342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5077418Z layer_outputs = layer_module( 2025-09-07T07:12:45.5077733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5077834Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5078167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5078287Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5078588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5078686Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5078691Z 2025-09-07T07:12:45.5078801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5079020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5079091Z return mod(**inputs) 2025-09-07T07:12:45.5079401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5079475Z outputs = self.mobilebert( 2025-09-07T07:12:45.5079774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5079859Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5080156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5080239Z layer_outputs = layer_module( 2025-09-07T07:12:45.5080550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5080666Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5080977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5081095Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5081404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5081526Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5081529Z 2025-09-07T07:12:45.5081645Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5081860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5081928Z return mod(**inputs) 2025-09-07T07:12:45.5082241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5082317Z outputs = self.mobilebert( 2025-09-07T07:12:45.5082622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5082699Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5082997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5083078Z layer_outputs = layer_module( 2025-09-07T07:12:45.5083353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5083454Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5083727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5083857Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5084131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5084216Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5084219Z 2025-09-07T07:12:45.5084328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5084521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5084637Z return mod(**inputs) 2025-09-07T07:12:45.5084917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5084986Z outputs = self.mobilebert( 2025-09-07T07:12:45.5085272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5085346Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5085640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5085711Z layer_outputs = layer_module( 2025-09-07T07:12:45.5086005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5086114Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5086426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5086568Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5086867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5087024Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5087342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5087443Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5087447Z 2025-09-07T07:12:45.5087564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5087779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5087858Z return mod(**inputs) 2025-09-07T07:12:45.5088164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5088245Z outputs = self.mobilebert( 2025-09-07T07:12:45.5088549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5088628Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5088942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5089018Z layer_outputs = layer_module( 2025-09-07T07:12:45.5089323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5089450Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5089761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5089852Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5089855Z 2025-09-07T07:12:45.5089957Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5090162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5090230Z return mod(**inputs) 2025-09-07T07:12:45.5090520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5090590Z outputs = self.mobilebert( 2025-09-07T07:12:45.5090880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5090962Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5091249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5091362Z layer_outputs = layer_module( 2025-09-07T07:12:45.5091657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5091774Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5092062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5092173Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5092176Z 2025-09-07T07:12:45.5092283Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5092476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5092548Z return mod(**inputs) 2025-09-07T07:12:45.5092833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5092903Z outputs = self.mobilebert( 2025-09-07T07:12:45.5093192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5093264Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5093567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5093637Z layer_outputs = layer_module( 2025-09-07T07:12:45.5093930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5094098Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5094373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5094477Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5094481Z 2025-09-07T07:12:45.5094582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5094785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5094849Z return mod(**inputs) 2025-09-07T07:12:45.5095129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5095210Z outputs = self.mobilebert( 2025-09-07T07:12:45.5095496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5095577Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5095873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5095953Z layer_outputs = layer_module( 2025-09-07T07:12:45.5096261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5096432Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5096739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5096873Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5097183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5097279Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5097283Z 2025-09-07T07:12:45.5097390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5097651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5097717Z return mod(**inputs) 2025-09-07T07:12:45.5098016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5098087Z outputs = self.mobilebert( 2025-09-07T07:12:45.5098371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5098452Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5098738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5098816Z layer_outputs = layer_module( 2025-09-07T07:12:45.5099099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5099268Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5099553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5099679Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5099994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5100085Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5100089Z 2025-09-07T07:12:45.5100218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5100431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5100501Z return mod(**inputs) 2025-09-07T07:12:45.5100813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5100886Z outputs = self.mobilebert( 2025-09-07T07:12:45.5101172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5101244Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5101533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5101606Z layer_outputs = layer_module( 2025-09-07T07:12:45.5101892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5102067Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5102364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5102506Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5102805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5102934Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5103240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5103342Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5103345Z 2025-09-07T07:12:45.5103463Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5103673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5103750Z return mod(**inputs) 2025-09-07T07:12:45.5104054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5104160Z outputs = self.mobilebert( 2025-09-07T07:12:45.5104472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5104551Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5104857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5104933Z layer_outputs = layer_module( 2025-09-07T07:12:45.5105237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5105419Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5105783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5105922Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5106231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5106329Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5106334Z 2025-09-07T07:12:45.5106448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5106689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5106772Z return mod(**inputs) 2025-09-07T07:12:45.5107105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5107190Z outputs = self.mobilebert( 2025-09-07T07:12:45.5107502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5107584Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5107897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5107972Z layer_outputs = layer_module( 2025-09-07T07:12:45.5108281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5108455Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5108767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5108885Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5109185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5109285Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5109587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5109694Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5109698Z 2025-09-07T07:12:45.5109805Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5110023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5110095Z return mod(**inputs) 2025-09-07T07:12:45.5110397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5110482Z outputs = self.mobilebert( 2025-09-07T07:12:45.5110780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5110867Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5111167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5111279Z layer_outputs = layer_module( 2025-09-07T07:12:45.5111589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5111676Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5111974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5112048Z self_outputs = self.self( 2025-09-07T07:12:45.5112337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5112418Z self.query(query_tensor) 2025-09-07T07:12:45.5112421Z 2025-09-07T07:12:45.5112526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5112736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5112805Z return mod(**inputs) 2025-09-07T07:12:45.5113102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5113175Z outputs = self.mobilebert( 2025-09-07T07:12:45.5113491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5113580Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5113894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5113979Z layer_outputs = layer_module( 2025-09-07T07:12:45.5114279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5114370Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5114684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5114760Z self_outputs = self.self( 2025-09-07T07:12:45.5115069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5115141Z self.key(key_tensor) 2025-09-07T07:12:45.5115146Z 2025-09-07T07:12:45.5115264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5115482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5115551Z return mod(**inputs) 2025-09-07T07:12:45.5115864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5115939Z outputs = self.mobilebert( 2025-09-07T07:12:45.5116250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5116326Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5116629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5116710Z layer_outputs = layer_module( 2025-09-07T07:12:45.5117016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5117115Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5117422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5117503Z self_outputs = self.self( 2025-09-07T07:12:45.5117807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5117915Z self.value(value_tensor) 2025-09-07T07:12:45.5117919Z 2025-09-07T07:12:45.5118015Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5118101Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5118217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5118430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5118501Z return mod(**inputs) 2025-09-07T07:12:45.5118813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5118886Z outputs = self.mobilebert( 2025-09-07T07:12:45.5119194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5119272Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5119712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5119810Z layer_outputs = layer_module( 2025-09-07T07:12:45.5120114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5120213Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5120555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5120725Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5121030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5121122Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5121126Z 2025-09-07T07:12:45.5121243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5121458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5121533Z return mod(**inputs) 2025-09-07T07:12:45.5121844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5121917Z outputs = self.mobilebert( 2025-09-07T07:12:45.5122237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5122314Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5122621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5122694Z layer_outputs = layer_module( 2025-09-07T07:12:45.5123005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5123184Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5123498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5123630Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5123932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5124026Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5124030Z 2025-09-07T07:12:45.5124141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5124352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5124433Z return mod(**inputs) 2025-09-07T07:12:45.5124703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5124830Z outputs = self.mobilebert( 2025-09-07T07:12:45.5125099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5125175Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5125444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5125511Z layer_outputs = layer_module( 2025-09-07T07:12:45.5125794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5125877Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5126160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5126282Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5126571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5126707Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5126998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5127113Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5127116Z 2025-09-07T07:12:45.5127230Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5127435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5127503Z return mod(**inputs) 2025-09-07T07:12:45.5127781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5127862Z outputs = self.mobilebert( 2025-09-07T07:12:45.5128136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5128216Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5128495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5128566Z layer_outputs = layer_module( 2025-09-07T07:12:45.5128851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5128946Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5129230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5129343Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5129628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5129713Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5129716Z 2025-09-07T07:12:45.5129817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5130021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5130087Z return mod(**inputs) 2025-09-07T07:12:45.5130373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5130443Z outputs = self.mobilebert( 2025-09-07T07:12:45.5130717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5130796Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5131070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5131181Z layer_outputs = layer_module( 2025-09-07T07:12:45.5131461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5131562Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5131840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5131953Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5132240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5132349Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5132352Z 2025-09-07T07:12:45.5132458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5132656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5132720Z return mod(**inputs) 2025-09-07T07:12:45.5133010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5133081Z outputs = self.mobilebert( 2025-09-07T07:12:45.5133380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5133454Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5133753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5133825Z layer_outputs = layer_module( 2025-09-07T07:12:45.5134100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5134205Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5134480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5134615Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5134903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5134994Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5135005Z 2025-09-07T07:12:45.5135117Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5135329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5135406Z return mod(**inputs) 2025-09-07T07:12:45.5135703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5135786Z outputs = self.mobilebert( 2025-09-07T07:12:45.5136069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5136143Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5136434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5136507Z layer_outputs = layer_module( 2025-09-07T07:12:45.5136800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5136891Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5137160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5137290Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5137597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5137723Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5137997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5138096Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5138099Z 2025-09-07T07:12:45.5138199Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5138391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5138464Z return mod(**inputs) 2025-09-07T07:12:45.5138739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5138816Z outputs = self.mobilebert( 2025-09-07T07:12:45.5139089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5139160Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5139441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5139525Z layer_outputs = layer_module( 2025-09-07T07:12:45.5139823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5139918Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5140200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5140311Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5140590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5140679Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5140682Z 2025-09-07T07:12:45.5140782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5140983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5141048Z return mod(**inputs) 2025-09-07T07:12:45.5141324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5141401Z outputs = self.mobilebert( 2025-09-07T07:12:45.5141681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5141761Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5142043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5142125Z layer_outputs = layer_module( 2025-09-07T07:12:45.5142406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5142500Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5142791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5142905Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5143194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5143306Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5143310Z 2025-09-07T07:12:45.5143413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5143650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5143717Z return mod(**inputs) 2025-09-07T07:12:45.5144011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5144082Z outputs = self.mobilebert( 2025-09-07T07:12:45.5144374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5144447Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5144732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5144810Z layer_outputs = layer_module( 2025-09-07T07:12:45.5145097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5145208Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5145511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5145646Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5146180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5146279Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5146283Z 2025-09-07T07:12:45.5146416Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5146630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5146710Z return mod(**inputs) 2025-09-07T07:12:45.5147018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5147094Z outputs = self.mobilebert( 2025-09-07T07:12:45.5147388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5147463Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5147753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5147828Z layer_outputs = layer_module( 2025-09-07T07:12:45.5148111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5148217Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5148508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5148640Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5148916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5149042Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5149316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5149408Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5149411Z 2025-09-07T07:12:45.5149521Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5149716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5149788Z return mod(**inputs) 2025-09-07T07:12:45.5150067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5150182Z outputs = self.mobilebert( 2025-09-07T07:12:45.5150463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5150535Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5150816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5150887Z layer_outputs = layer_module( 2025-09-07T07:12:45.5151172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5151268Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5151544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5151663Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5151940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5152031Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5152035Z 2025-09-07T07:12:45.5152135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5152329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5152414Z return mod(**inputs) 2025-09-07T07:12:45.5152709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5152789Z outputs = self.mobilebert( 2025-09-07T07:12:45.5153062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5153140Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5153415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5153488Z layer_outputs = layer_module( 2025-09-07T07:12:45.5153767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5153859Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5154141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5154253Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5154527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5154642Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5154645Z 2025-09-07T07:12:45.5154746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5154949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5155013Z return mod(**inputs) 2025-09-07T07:12:45.5155296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5155365Z outputs = self.mobilebert( 2025-09-07T07:12:45.5155650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5155730Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5156015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5156104Z layer_outputs = layer_module( 2025-09-07T07:12:45.5156378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5156499Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5156783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5156903Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5157185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5157268Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5157272Z 2025-09-07T07:12:45.5157378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5157572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5157636Z return mod(**inputs) 2025-09-07T07:12:45.5157925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5157997Z outputs = self.mobilebert( 2025-09-07T07:12:45.5158280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5158351Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5158624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5158715Z layer_outputs = layer_module( 2025-09-07T07:12:45.5159009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5159110Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5159387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5159516Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5159795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5159915Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5160200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5160290Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5160294Z 2025-09-07T07:12:45.5160403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5160598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5160662Z return mod(**inputs) 2025-09-07T07:12:45.5160949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5161021Z outputs = self.mobilebert( 2025-09-07T07:12:45.5161302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5161374Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5161658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5161728Z layer_outputs = layer_module( 2025-09-07T07:12:45.5162001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5162129Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5162404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5162491Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5162525Z 2025-09-07T07:12:45.5162627Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5162829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5162900Z return mod(**inputs) 2025-09-07T07:12:45.5163170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5163246Z outputs = self.mobilebert( 2025-09-07T07:12:45.5163525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5163604Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5163878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5163946Z layer_outputs = layer_module( 2025-09-07T07:12:45.5164236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5164355Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5164638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5164747Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5164751Z 2025-09-07T07:12:45.5164900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5165110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5165177Z return mod(**inputs) 2025-09-07T07:12:45.5165478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5165549Z outputs = self.mobilebert( 2025-09-07T07:12:45.5165852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5165933Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5166237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5166321Z layer_outputs = layer_module( 2025-09-07T07:12:45.5166626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5166805Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5167115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5167214Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5167225Z 2025-09-07T07:12:45.5167330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5167535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5167608Z return mod(**inputs) 2025-09-07T07:12:45.5167895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5167974Z outputs = self.mobilebert( 2025-09-07T07:12:45.5168260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5168336Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5168634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5168707Z layer_outputs = layer_module( 2025-09-07T07:12:45.5169007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5169196Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5169473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5169603Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5169880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5169978Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5169981Z 2025-09-07T07:12:45.5170083Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5170283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5170347Z return mod(**inputs) 2025-09-07T07:12:45.5170627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5170707Z outputs = self.mobilebert( 2025-09-07T07:12:45.5170982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5171058Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5171352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5171423Z layer_outputs = layer_module( 2025-09-07T07:12:45.5171727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5171884Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5172170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5172294Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5172587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5172671Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5172676Z 2025-09-07T07:12:45.5172776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5172982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5173045Z return mod(**inputs) 2025-09-07T07:12:45.5173339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5173408Z outputs = self.mobilebert( 2025-09-07T07:12:45.5173699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5173775Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5174066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5174143Z layer_outputs = layer_module( 2025-09-07T07:12:45.5174435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5174604Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5174904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5175036Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5175356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5175494Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5175817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5175910Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5175913Z 2025-09-07T07:12:45.5176025Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5176225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5176292Z return mod(**inputs) 2025-09-07T07:12:45.5176588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5176660Z outputs = self.mobilebert( 2025-09-07T07:12:45.5176951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5177028Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5177311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5177392Z layer_outputs = layer_module( 2025-09-07T07:12:45.5177673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5177861Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5178183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5178313Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5178610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5178699Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5178706Z 2025-09-07T07:12:45.5178826Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5179036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5179114Z return mod(**inputs) 2025-09-07T07:12:45.5179416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5179488Z outputs = self.mobilebert( 2025-09-07T07:12:45.5179787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5179865Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5180176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5180251Z layer_outputs = layer_module( 2025-09-07T07:12:45.5180558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5180740Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5181045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5181171Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5181526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5181629Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5181927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5182025Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5182036Z 2025-09-07T07:12:45.5182173Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5182385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5182463Z return mod(**inputs) 2025-09-07T07:12:45.5182767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5182850Z outputs = self.mobilebert( 2025-09-07T07:12:45.5183154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5183234Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5183543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5183620Z layer_outputs = layer_module( 2025-09-07T07:12:45.5183927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5184023Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5184324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5184408Z self_outputs = self.self( 2025-09-07T07:12:45.5184724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5184808Z self.query(query_tensor) 2025-09-07T07:12:45.5184812Z 2025-09-07T07:12:45.5184939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5185160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5185231Z return mod(**inputs) 2025-09-07T07:12:45.5185531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5185616Z outputs = self.mobilebert( 2025-09-07T07:12:45.5185985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5186075Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5186376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5186455Z layer_outputs = layer_module( 2025-09-07T07:12:45.5186763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5186856Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5187164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5187239Z self_outputs = self.self( 2025-09-07T07:12:45.5187547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5187619Z self.key(key_tensor) 2025-09-07T07:12:45.5187623Z 2025-09-07T07:12:45.5187733Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5187949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5188019Z return mod(**inputs) 2025-09-07T07:12:45.5188329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5188404Z outputs = self.mobilebert( 2025-09-07T07:12:45.5188703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5188788Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5189087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5189203Z layer_outputs = layer_module( 2025-09-07T07:12:45.5189505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5189595Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5189907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5189979Z self_outputs = self.self( 2025-09-07T07:12:45.5190291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5190365Z self.value(value_tensor) 2025-09-07T07:12:45.5190368Z 2025-09-07T07:12:45.5190464Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5190549Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5190661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5190882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5190952Z return mod(**inputs) 2025-09-07T07:12:45.5191261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5191335Z outputs = self.mobilebert( 2025-09-07T07:12:45.5191655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5191755Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5192056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5192140Z layer_outputs = layer_module( 2025-09-07T07:12:45.5192439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5192531Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5192843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5192968Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5193263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5193348Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5193353Z 2025-09-07T07:12:45.5193464Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5193662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5193727Z return mod(**inputs) 2025-09-07T07:12:45.5194018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5194094Z outputs = self.mobilebert( 2025-09-07T07:12:45.5194384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5194457Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5194737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5194819Z layer_outputs = layer_module( 2025-09-07T07:12:45.5195102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5195277Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5195577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5195737Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5196039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5196126Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5196130Z 2025-09-07T07:12:45.5196247Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5196461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5196537Z return mod(**inputs) 2025-09-07T07:12:45.5196841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5196923Z outputs = self.mobilebert( 2025-09-07T07:12:45.5197234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5197310Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5197600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5197669Z layer_outputs = layer_module( 2025-09-07T07:12:45.5197973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5198087Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5198407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5198546Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5198848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5198990Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5199297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5199403Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5199407Z 2025-09-07T07:12:45.5199515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5199727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5199804Z return mod(**inputs) 2025-09-07T07:12:45.5200111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5200195Z outputs = self.mobilebert( 2025-09-07T07:12:45.5200496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5200577Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5200892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5200967Z layer_outputs = layer_module( 2025-09-07T07:12:45.5201274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5201375Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5201682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5201806Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5202107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5202205Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5202208Z 2025-09-07T07:12:45.5202354Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5202573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5202643Z return mod(**inputs) 2025-09-07T07:12:45.5202948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5203033Z outputs = self.mobilebert( 2025-09-07T07:12:45.5203346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5203432Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5203732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5203814Z layer_outputs = layer_module( 2025-09-07T07:12:45.5204115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5204219Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5204526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5204645Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5204967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5205089Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5205108Z 2025-09-07T07:12:45.5205217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5205435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5205504Z return mod(**inputs) 2025-09-07T07:12:45.5205814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5205893Z outputs = self.mobilebert( 2025-09-07T07:12:45.5206200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5206278Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5206583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5206665Z layer_outputs = layer_module( 2025-09-07T07:12:45.5206968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5207075Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5207375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5207511Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5207820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5207910Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5207913Z 2025-09-07T07:12:45.5208025Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5208236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5208314Z return mod(**inputs) 2025-09-07T07:12:45.5208617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5208693Z outputs = self.mobilebert( 2025-09-07T07:12:45.5208996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5209105Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5209413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5209488Z layer_outputs = layer_module( 2025-09-07T07:12:45.5209789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5209898Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5210198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5210341Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5210640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5210776Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5211080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5211178Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5211182Z 2025-09-07T07:12:45.5211300Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5211527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5211609Z return mod(**inputs) 2025-09-07T07:12:45.5211930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5212007Z outputs = self.mobilebert( 2025-09-07T07:12:45.5212315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5212392Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5212705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5212780Z layer_outputs = layer_module( 2025-09-07T07:12:45.5213093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5213191Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5213497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5213628Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5213932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5214029Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5214032Z 2025-09-07T07:12:45.5214143Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5214355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5214434Z return mod(**inputs) 2025-09-07T07:12:45.5214739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5214823Z outputs = self.mobilebert( 2025-09-07T07:12:45.5215125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5215212Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5215511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5215587Z layer_outputs = layer_module( 2025-09-07T07:12:45.5215897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5216031Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5216339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5216460Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5216762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5216889Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5216894Z 2025-09-07T07:12:45.5217003Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5217220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5217290Z return mod(**inputs) 2025-09-07T07:12:45.5217601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5217680Z outputs = self.mobilebert( 2025-09-07T07:12:45.5217983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5218070Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5218390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5218474Z layer_outputs = layer_module( 2025-09-07T07:12:45.5218789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5218891Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5219200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5219336Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5219809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5219910Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5219914Z 2025-09-07T07:12:45.5220035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5220260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5220334Z return mod(**inputs) 2025-09-07T07:12:45.5220658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5220737Z outputs = self.mobilebert( 2025-09-07T07:12:45.5221067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5221146Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5221447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5221533Z layer_outputs = layer_module( 2025-09-07T07:12:45.5221837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5221947Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5222252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5222392Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5222694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5222824Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5223210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5223311Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5223315Z 2025-09-07T07:12:45.5223430Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5223654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5223725Z return mod(**inputs) 2025-09-07T07:12:45.5224047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5224123Z outputs = self.mobilebert( 2025-09-07T07:12:45.5224439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5224518Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5224837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5224917Z layer_outputs = layer_module( 2025-09-07T07:12:45.5225226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5225336Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5225667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5225884Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5226211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5226303Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5226307Z 2025-09-07T07:12:45.5226430Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5226652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5226733Z return mod(**inputs) 2025-09-07T07:12:45.5227056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5227142Z outputs = self.mobilebert( 2025-09-07T07:12:45.5227452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5227537Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5227855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5227933Z layer_outputs = layer_module( 2025-09-07T07:12:45.5228254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5228364Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5228675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5228808Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5229122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5229253Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5229258Z 2025-09-07T07:12:45.5229371Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5229599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5229673Z return mod(**inputs) 2025-09-07T07:12:45.5229984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5230107Z outputs = self.mobilebert( 2025-09-07T07:12:45.5230416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5230503Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5230814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5230892Z layer_outputs = layer_module( 2025-09-07T07:12:45.5231221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5231324Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5231645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5231784Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5232102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5232195Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5232199Z 2025-09-07T07:12:45.5232309Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5232557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5232623Z return mod(**inputs) 2025-09-07T07:12:45.5232925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5232995Z outputs = self.mobilebert( 2025-09-07T07:12:45.5233288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5233368Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5233696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5233774Z layer_outputs = layer_module( 2025-09-07T07:12:45.5234058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5234160Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5234447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5234571Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5234866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5234988Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5235296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5235390Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5235394Z 2025-09-07T07:12:45.5235496Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5235714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5235782Z return mod(**inputs) 2025-09-07T07:12:45.5236088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5236161Z outputs = self.mobilebert( 2025-09-07T07:12:45.5236466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5236539Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5236846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5236925Z layer_outputs = layer_module( 2025-09-07T07:12:45.5237206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5237333Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5237613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5237698Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5237709Z 2025-09-07T07:12:45.5237811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5238045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5238122Z return mod(**inputs) 2025-09-07T07:12:45.5238397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5238471Z outputs = self.mobilebert( 2025-09-07T07:12:45.5238740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5238809Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5239102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5239186Z layer_outputs = layer_module( 2025-09-07T07:12:45.5239468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5239584Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5239860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5239976Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5239980Z 2025-09-07T07:12:45.5240076Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5240275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5240338Z return mod(**inputs) 2025-09-07T07:12:45.5240622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5240691Z outputs = self.mobilebert( 2025-09-07T07:12:45.5240962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5241037Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5241305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5241383Z layer_outputs = layer_module( 2025-09-07T07:12:45.5241657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5241816Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5242103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5242199Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5242203Z 2025-09-07T07:12:45.5242312Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5242507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5242579Z return mod(**inputs) 2025-09-07T07:12:45.5242865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5242972Z outputs = self.mobilebert( 2025-09-07T07:12:45.5243257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5243329Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5243616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5243686Z layer_outputs = layer_module( 2025-09-07T07:12:45.5243965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5244143Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5244413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5244544Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5244817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5244913Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5244916Z 2025-09-07T07:12:45.5245017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5245222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5245295Z return mod(**inputs) 2025-09-07T07:12:45.5245587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5245665Z outputs = self.mobilebert( 2025-09-07T07:12:45.5245943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5246020Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5246312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5246385Z layer_outputs = layer_module( 2025-09-07T07:12:45.5246676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5246838Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5247127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5247259Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5247528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5247622Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5247625Z 2025-09-07T07:12:45.5247724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5247920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5247983Z return mod(**inputs) 2025-09-07T07:12:45.5248255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5248338Z outputs = self.mobilebert( 2025-09-07T07:12:45.5248615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5248695Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5248969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5249044Z layer_outputs = layer_module( 2025-09-07T07:12:45.5249362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5249516Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5249803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5249924Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5250207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5250326Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5250604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5250694Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5250701Z 2025-09-07T07:12:45.5250801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5251011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5251072Z return mod(**inputs) 2025-09-07T07:12:45.5251346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5251426Z outputs = self.mobilebert( 2025-09-07T07:12:45.5251712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5251793Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5252062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5252140Z layer_outputs = layer_module( 2025-09-07T07:12:45.5252417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5252586Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5252863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5252975Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5253262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5253343Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5253347Z 2025-09-07T07:12:45.5253454Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5253646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5253713Z return mod(**inputs) 2025-09-07T07:12:45.5254002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5254071Z outputs = self.mobilebert( 2025-09-07T07:12:45.5254353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5254425Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5254707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5254778Z layer_outputs = layer_module( 2025-09-07T07:12:45.5255052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5255217Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5255525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5255639Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5255911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5255995Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5256277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5256367Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5256370Z 2025-09-07T07:12:45.5256478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5256672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5256742Z return mod(**inputs) 2025-09-07T07:12:45.5257028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5257099Z outputs = self.mobilebert( 2025-09-07T07:12:45.5257385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5257458Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5257762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5257834Z layer_outputs = layer_module( 2025-09-07T07:12:45.5258132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5258227Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5258509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5258591Z self_outputs = self.self( 2025-09-07T07:12:45.5258880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5258958Z self.query(query_tensor) 2025-09-07T07:12:45.5258962Z 2025-09-07T07:12:45.5259065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5259266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5259340Z return mod(**inputs) 2025-09-07T07:12:45.5259629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5259705Z outputs = self.mobilebert( 2025-09-07T07:12:45.5259986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5260062Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5260356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5260429Z layer_outputs = layer_module( 2025-09-07T07:12:45.5260717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5260805Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5261089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5261168Z self_outputs = self.self( 2025-09-07T07:12:45.5261449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5261528Z self.key(key_tensor) 2025-09-07T07:12:45.5261531Z 2025-09-07T07:12:45.5261635Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5261875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5261942Z return mod(**inputs) 2025-09-07T07:12:45.5262230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5262307Z outputs = self.mobilebert( 2025-09-07T07:12:45.5262596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5262679Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5262964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5263036Z layer_outputs = layer_module( 2025-09-07T07:12:45.5263326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5263415Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5263704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5263775Z self_outputs = self.self( 2025-09-07T07:12:45.5264079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5264152Z self.value(value_tensor) 2025-09-07T07:12:45.5264156Z 2025-09-07T07:12:45.5264254Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5264345Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5264447Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5264650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5264718Z return mod(**inputs) 2025-09-07T07:12:45.5265026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5265108Z outputs = self.mobilebert( 2025-09-07T07:12:45.5265406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5265491Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5265865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5265945Z layer_outputs = layer_module( 2025-09-07T07:12:45.5266260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5266353Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5266674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5266814Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5267135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5267230Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5267233Z 2025-09-07T07:12:45.5267346Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5267550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5267617Z return mod(**inputs) 2025-09-07T07:12:45.5267922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5268000Z outputs = self.mobilebert( 2025-09-07T07:12:45.5268317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5268475Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5268786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5268870Z layer_outputs = layer_module( 2025-09-07T07:12:45.5269178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5269367Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5269680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5269805Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5270124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5270216Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5270220Z 2025-09-07T07:12:45.5270337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5270556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5270627Z return mod(**inputs) 2025-09-07T07:12:45.5270963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5271042Z outputs = self.mobilebert( 2025-09-07T07:12:45.5271397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5271480Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5271800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5271880Z layer_outputs = layer_module( 2025-09-07T07:12:45.5272190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5272289Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5272595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5272739Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5273048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5273187Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5273502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5273603Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5273609Z 2025-09-07T07:12:45.5273729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5273946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5274025Z return mod(**inputs) 2025-09-07T07:12:45.5274335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5274413Z outputs = self.mobilebert( 2025-09-07T07:12:45.5274709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5274778Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5275052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5275120Z layer_outputs = layer_module( 2025-09-07T07:12:45.5275423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5275525Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5275801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5275920Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5276195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5276288Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5276291Z 2025-09-07T07:12:45.5276389Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5276583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5276656Z return mod(**inputs) 2025-09-07T07:12:45.5276936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5277012Z outputs = self.mobilebert( 2025-09-07T07:12:45.5277287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5277358Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5277659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5277731Z layer_outputs = layer_module( 2025-09-07T07:12:45.5278031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5278126Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5278414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5278524Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5278792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5278909Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5278912Z 2025-09-07T07:12:45.5279009Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5279207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5279273Z return mod(**inputs) 2025-09-07T07:12:45.5279550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5279628Z outputs = self.mobilebert( 2025-09-07T07:12:45.5279903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5279986Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5280263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5280338Z layer_outputs = layer_module( 2025-09-07T07:12:45.5280609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5280700Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5280984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5281104Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5281377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5281488Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5281491Z 2025-09-07T07:12:45.5281595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5281787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5281849Z return mod(**inputs) 2025-09-07T07:12:45.5282130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5282198Z outputs = self.mobilebert( 2025-09-07T07:12:45.5282471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5282541Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5282808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5282884Z layer_outputs = layer_module( 2025-09-07T07:12:45.5283152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5283250Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5283517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5283653Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5283949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5284070Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5284362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5284453Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5284459Z 2025-09-07T07:12:45.5284567Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5284764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5284828Z return mod(**inputs) 2025-09-07T07:12:45.5285124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5285195Z outputs = self.mobilebert( 2025-09-07T07:12:45.5285482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5285554Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5285835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5285914Z layer_outputs = layer_module( 2025-09-07T07:12:45.5286204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5286309Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5286597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5286715Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5287007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5287092Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5287096Z 2025-09-07T07:12:45.5287205Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5287405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5287477Z return mod(**inputs) 2025-09-07T07:12:45.5288525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5288597Z outputs = self.mobilebert( 2025-09-07T07:12:45.5288896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5288970Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5289265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5289335Z layer_outputs = layer_module( 2025-09-07T07:12:45.5289628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5289722Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5290005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5290127Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5290418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5290537Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5290540Z 2025-09-07T07:12:45.5290642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5290859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5290940Z return mod(**inputs) 2025-09-07T07:12:45.5291219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5291297Z outputs = self.mobilebert( 2025-09-07T07:12:45.5291580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5291668Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5291955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5292030Z layer_outputs = layer_module( 2025-09-07T07:12:45.5292323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5292424Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5292717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5292849Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5293138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5293239Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5293242Z 2025-09-07T07:12:45.5293350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5293561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5293631Z return mod(**inputs) 2025-09-07T07:12:45.5293929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5294005Z outputs = self.mobilebert( 2025-09-07T07:12:45.5294292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5294379Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5294666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5294748Z layer_outputs = layer_module( 2025-09-07T07:12:45.5295120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5295221Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5295529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5295664Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5295972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5296102Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5296413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5296506Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5296513Z 2025-09-07T07:12:45.5296615Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5296821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5296887Z return mod(**inputs) 2025-09-07T07:12:45.5297179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5297303Z outputs = self.mobilebert( 2025-09-07T07:12:45.5297613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5297694Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5297980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5298060Z layer_outputs = layer_module( 2025-09-07T07:12:45.5298344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5298447Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5298730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5298842Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5299135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5299222Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5299225Z 2025-09-07T07:12:45.5299335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5299536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5299609Z return mod(**inputs) 2025-09-07T07:12:45.5299899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5299974Z outputs = self.mobilebert( 2025-09-07T07:12:45.5300263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5300336Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5300630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5300701Z layer_outputs = layer_module( 2025-09-07T07:12:45.5300988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5301091Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5301379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5301532Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5301815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5301928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5301938Z 2025-09-07T07:12:45.5302042Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5302245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5302321Z return mod(**inputs) 2025-09-07T07:12:45.5302609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5302689Z outputs = self.mobilebert( 2025-09-07T07:12:45.5302975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5303050Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5303337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5303407Z layer_outputs = layer_module( 2025-09-07T07:12:45.5303693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5303802Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5304106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5304240Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5304520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5304615Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5304619Z 2025-09-07T07:12:45.5304720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5304925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5304992Z return mod(**inputs) 2025-09-07T07:12:45.5305291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5305373Z outputs = self.mobilebert( 2025-09-07T07:12:45.5305671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5305836Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5306141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5306217Z layer_outputs = layer_module( 2025-09-07T07:12:45.5306530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5306630Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5306940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5307075Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5307384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5307514Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5307816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5307923Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5307962Z 2025-09-07T07:12:45.5308072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5308294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5308366Z return mod(**inputs) 2025-09-07T07:12:45.5308677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5308753Z outputs = self.mobilebert( 2025-09-07T07:12:45.5309057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5309142Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5309444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5309526Z layer_outputs = layer_module( 2025-09-07T07:12:45.5309828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5309961Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5310273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5310362Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5310366Z 2025-09-07T07:12:45.5310501Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5310733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5310813Z return mod(**inputs) 2025-09-07T07:12:45.5311115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5311193Z outputs = self.mobilebert( 2025-09-07T07:12:45.5311504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5311585Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5311890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5311966Z layer_outputs = layer_module( 2025-09-07T07:12:45.5312274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5312412Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5312716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5312845Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5312849Z 2025-09-07T07:12:45.5312957Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5313178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5313247Z return mod(**inputs) 2025-09-07T07:12:45.5313562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5313644Z outputs = self.mobilebert( 2025-09-07T07:12:45.5313944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5314028Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5314336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5314413Z layer_outputs = layer_module( 2025-09-07T07:12:45.5314734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5314940Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5315249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5315354Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5315358Z 2025-09-07T07:12:45.5315473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5315689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5315759Z return mod(**inputs) 2025-09-07T07:12:45.5316079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5316154Z outputs = self.mobilebert( 2025-09-07T07:12:45.5316472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5316553Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5316866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5316951Z layer_outputs = layer_module( 2025-09-07T07:12:45.5317261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5317459Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5317786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5317925Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5318234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5318335Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5318339Z 2025-09-07T07:12:45.5318455Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5318665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5318742Z return mod(**inputs) 2025-09-07T07:12:45.5319054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5319129Z outputs = self.mobilebert( 2025-09-07T07:12:45.5319458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5319536Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5319986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5320064Z layer_outputs = layer_module( 2025-09-07T07:12:45.5320391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5320559Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5320879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5321020Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5321326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5321426Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5321431Z 2025-09-07T07:12:45.5321539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5321750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5321896Z return mod(**inputs) 2025-09-07T07:12:45.5322212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5322294Z outputs = self.mobilebert( 2025-09-07T07:12:45.5322589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5322669Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5322946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5323017Z layer_outputs = layer_module( 2025-09-07T07:12:45.5323300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5323455Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5335459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5335683Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5336022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5336267Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5336592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5336705Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5336712Z 2025-09-07T07:12:45.5336829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5337048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5337127Z return mod(**inputs) 2025-09-07T07:12:45.5337424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5337506Z outputs = self.mobilebert( 2025-09-07T07:12:45.5337790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5337880Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5338170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5338253Z layer_outputs = layer_module( 2025-09-07T07:12:45.5338534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5338700Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5338997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5339113Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5339404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5339489Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5339495Z 2025-09-07T07:12:45.5339612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5339818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5339885Z return mod(**inputs) 2025-09-07T07:12:45.5340177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5340251Z outputs = self.mobilebert( 2025-09-07T07:12:45.5340573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5340657Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5340935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5341007Z layer_outputs = layer_module( 2025-09-07T07:12:45.5341295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5341456Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5341743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5341853Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5342145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5342237Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5342522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5342626Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5342630Z 2025-09-07T07:12:45.5342750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5342981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5343053Z return mod(**inputs) 2025-09-07T07:12:45.5343344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5343427Z outputs = self.mobilebert( 2025-09-07T07:12:45.5343713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5343802Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5344088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5344167Z layer_outputs = layer_module( 2025-09-07T07:12:45.5344453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5344543Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5344838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5344912Z self_outputs = self.self( 2025-09-07T07:12:45.5345202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5345278Z self.query(query_tensor) 2025-09-07T07:12:45.5345282Z 2025-09-07T07:12:45.5345388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5345602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5345670Z return mod(**inputs) 2025-09-07T07:12:45.5346071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5346157Z outputs = self.mobilebert( 2025-09-07T07:12:45.5346478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5346561Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5346870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5346967Z layer_outputs = layer_module( 2025-09-07T07:12:45.5347281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5347374Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5347659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5347727Z self_outputs = self.self( 2025-09-07T07:12:45.5348006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5348074Z self.key(key_tensor) 2025-09-07T07:12:45.5348078Z 2025-09-07T07:12:45.5348178Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5348379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5348444Z return mod(**inputs) 2025-09-07T07:12:45.5348724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5348796Z outputs = self.mobilebert( 2025-09-07T07:12:45.5349062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5349139Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5349424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5349505Z layer_outputs = layer_module( 2025-09-07T07:12:45.5349794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5349885Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5350159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5350231Z self_outputs = self.self( 2025-09-07T07:12:45.5350525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5350590Z self.value(value_tensor) 2025-09-07T07:12:45.5350594Z 2025-09-07T07:12:45.5350681Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5350757Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5350856Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5351055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5351118Z return mod(**inputs) 2025-09-07T07:12:45.5351396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5351463Z outputs = self.mobilebert( 2025-09-07T07:12:45.5351730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5351812Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5352087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5352162Z layer_outputs = layer_module( 2025-09-07T07:12:45.5352447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5352532Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5352799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5352921Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5353205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5353319Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5353323Z 2025-09-07T07:12:45.5353431Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5353633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5353697Z return mod(**inputs) 2025-09-07T07:12:45.5353973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5354041Z outputs = self.mobilebert( 2025-09-07T07:12:45.5354315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5354386Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5354657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5354730Z layer_outputs = layer_module( 2025-09-07T07:12:45.5355003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5355170Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5355459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5355578Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5355869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5355953Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5355964Z 2025-09-07T07:12:45.5356068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5356267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5356346Z return mod(**inputs) 2025-09-07T07:12:45.5356638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5356716Z outputs = self.mobilebert( 2025-09-07T07:12:45.5356994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5357069Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5357361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5357432Z layer_outputs = layer_module( 2025-09-07T07:12:45.5357724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5357809Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5358095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5358228Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5358510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5358650Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5358934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5359038Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5359042Z 2025-09-07T07:12:45.5359145Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5359347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5359474Z return mod(**inputs) 2025-09-07T07:12:45.5359763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5359845Z outputs = self.mobilebert( 2025-09-07T07:12:45.5360130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5360205Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5360498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5360573Z layer_outputs = layer_module( 2025-09-07T07:12:45.5360866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5360967Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5361261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5361380Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5361667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5361765Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5361769Z 2025-09-07T07:12:45.5361890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5362117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5362185Z return mod(**inputs) 2025-09-07T07:12:45.5362477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5362558Z outputs = self.mobilebert( 2025-09-07T07:12:45.5362844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5362928Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5363217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5363296Z layer_outputs = layer_module( 2025-09-07T07:12:45.5363589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5363686Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5363981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5364096Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5364391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5364512Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5364516Z 2025-09-07T07:12:45.5364626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5364831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5364898Z return mod(**inputs) 2025-09-07T07:12:45.5365197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5365268Z outputs = self.mobilebert( 2025-09-07T07:12:45.5365565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5365638Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5365923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5366035Z layer_outputs = layer_module( 2025-09-07T07:12:45.5366320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5366423Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5366707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5366840Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5367131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5367217Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5367221Z 2025-09-07T07:12:45.5367334Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5367536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5367613Z return mod(**inputs) 2025-09-07T07:12:45.5367899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5367971Z outputs = self.mobilebert( 2025-09-07T07:12:45.5368258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5368349Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5368656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5368728Z layer_outputs = layer_module( 2025-09-07T07:12:45.5369010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5369115Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5369402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5369537Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5369824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5369957Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5370253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5370347Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5370350Z 2025-09-07T07:12:45.5370462Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5370661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5370738Z return mod(**inputs) 2025-09-07T07:12:45.5371028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5371100Z outputs = self.mobilebert( 2025-09-07T07:12:45.5371400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5371471Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5371757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5371829Z layer_outputs = layer_module( 2025-09-07T07:12:45.5372110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5372204Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5372480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5372628Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5372902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5372993Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5372997Z 2025-09-07T07:12:45.5373100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5373302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5373369Z return mod(**inputs) 2025-09-07T07:12:45.5373647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5373726Z outputs = self.mobilebert( 2025-09-07T07:12:45.5374001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5374083Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5374357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5374426Z layer_outputs = layer_module( 2025-09-07T07:12:45.5374726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5374819Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5375115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5375227Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5375501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5375623Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5375627Z 2025-09-07T07:12:45.5375734Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5375957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5376026Z return mod(**inputs) 2025-09-07T07:12:45.5376339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5376418Z outputs = self.mobilebert( 2025-09-07T07:12:45.5376729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5376817Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5377128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5377213Z layer_outputs = layer_module( 2025-09-07T07:12:45.5377521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5377616Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5377905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5378032Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5378323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5378409Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5378412Z 2025-09-07T07:12:45.5378521Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5378723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5378820Z return mod(**inputs) 2025-09-07T07:12:45.5379115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5379185Z outputs = self.mobilebert( 2025-09-07T07:12:45.5379466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5379538Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5379816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5379895Z layer_outputs = layer_module( 2025-09-07T07:12:45.5380170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5380272Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5380550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5380682Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5380957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5381098Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5381406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5381498Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5381502Z 2025-09-07T07:12:45.5381610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5381805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5381879Z return mod(**inputs) 2025-09-07T07:12:45.5382156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5382227Z outputs = self.mobilebert( 2025-09-07T07:12:45.5382512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5382584Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5382873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5382945Z layer_outputs = layer_module( 2025-09-07T07:12:45.5383230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5383331Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5383612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5383733Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5384023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5384108Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5384120Z 2025-09-07T07:12:45.5384224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5384430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5384506Z return mod(**inputs) 2025-09-07T07:12:45.5384798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5384875Z outputs = self.mobilebert( 2025-09-07T07:12:45.5385157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5385297Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5385605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5385681Z layer_outputs = layer_module( 2025-09-07T07:12:45.5386076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5386181Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5386483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5386612Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5386909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5387042Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5387046Z 2025-09-07T07:12:45.5387156Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5387376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5387446Z return mod(**inputs) 2025-09-07T07:12:45.5387769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5387856Z outputs = self.mobilebert( 2025-09-07T07:12:45.5388171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5388259Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5388555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5388635Z layer_outputs = layer_module( 2025-09-07T07:12:45.5388940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5389041Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5389348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5389484Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5389792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5389884Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5389887Z 2025-09-07T07:12:45.5389997Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5390216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5390291Z return mod(**inputs) 2025-09-07T07:12:45.5390600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5390677Z outputs = self.mobilebert( 2025-09-07T07:12:45.5390976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5391061Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5391361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5391446Z layer_outputs = layer_module( 2025-09-07T07:12:45.5391742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5391850Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5392177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5392312Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5392623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5392754Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5393064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5393162Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5393166Z 2025-09-07T07:12:45.5393283Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5393494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5393564Z return mod(**inputs) 2025-09-07T07:12:45.5393877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5393952Z outputs = self.mobilebert( 2025-09-07T07:12:45.5394261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5394339Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5394650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5394753Z layer_outputs = layer_module( 2025-09-07T07:12:45.5395047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5395187Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5395489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5395583Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5395594Z 2025-09-07T07:12:45.5395702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5395912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5395990Z return mod(**inputs) 2025-09-07T07:12:45.5396297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5396382Z outputs = self.mobilebert( 2025-09-07T07:12:45.5396682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5396760Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5397072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5397150Z layer_outputs = layer_module( 2025-09-07T07:12:45.5397458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5397586Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5397888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5398016Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5398020Z 2025-09-07T07:12:45.5398132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5398352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5398421Z return mod(**inputs) 2025-09-07T07:12:45.5398732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5398849Z outputs = self.mobilebert( 2025-09-07T07:12:45.5399134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5399215Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5399507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5399587Z layer_outputs = layer_module( 2025-09-07T07:12:45.5399875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5400040Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5400331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5400430Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5400433Z 2025-09-07T07:12:45.5400544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5400745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5400818Z return mod(**inputs) 2025-09-07T07:12:45.5401125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5401197Z outputs = self.mobilebert( 2025-09-07T07:12:45.5401517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5401590Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5401879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5401954Z layer_outputs = layer_module( 2025-09-07T07:12:45.5402239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5402408Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5402690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5402824Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5403109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5403210Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5403213Z 2025-09-07T07:12:45.5403316Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5403514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5403592Z return mod(**inputs) 2025-09-07T07:12:45.5403878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5403956Z outputs = self.mobilebert( 2025-09-07T07:12:45.5404240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5404321Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5404605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5404676Z layer_outputs = layer_module( 2025-09-07T07:12:45.5404965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5405125Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5405444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5405576Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5405877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5405977Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5405982Z 2025-09-07T07:12:45.5406089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5406312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5406383Z return mod(**inputs) 2025-09-07T07:12:45.5406696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5406787Z outputs = self.mobilebert( 2025-09-07T07:12:45.5407071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5407153Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5407436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5407514Z layer_outputs = layer_module( 2025-09-07T07:12:45.5407817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5407992Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5408283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5408407Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5408717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5408845Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5409154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5409255Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5409258Z 2025-09-07T07:12:45.5409367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5409589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5409659Z return mod(**inputs) 2025-09-07T07:12:45.5409973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5410045Z outputs = self.mobilebert( 2025-09-07T07:12:45.5410332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5410413Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5410696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5410774Z layer_outputs = layer_module( 2025-09-07T07:12:45.5411068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5411251Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5411553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5411673Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5412007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5412097Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5412101Z 2025-09-07T07:12:45.5412215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5412425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5412498Z return mod(**inputs) 2025-09-07T07:12:45.5412810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5412887Z outputs = self.mobilebert( 2025-09-07T07:12:45.5413189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5413263Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5413552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5413626Z layer_outputs = layer_module( 2025-09-07T07:12:45.5413907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5414077Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5414378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5414513Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5414797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5414886Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5415175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5415272Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5415275Z 2025-09-07T07:12:45.5415387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5415586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5415659Z return mod(**inputs) 2025-09-07T07:12:45.5415945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5416018Z outputs = self.mobilebert( 2025-09-07T07:12:45.5416328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5416399Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5416689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5416763Z layer_outputs = layer_module( 2025-09-07T07:12:45.5417044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5417138Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5417422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5417498Z self_outputs = self.self( 2025-09-07T07:12:45.5417781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5417860Z self.query(query_tensor) 2025-09-07T07:12:45.5417863Z 2025-09-07T07:12:45.5417967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5418166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5418272Z return mod(**inputs) 2025-09-07T07:12:45.5418563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5418641Z outputs = self.mobilebert( 2025-09-07T07:12:45.5418924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5418998Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5419296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5419367Z layer_outputs = layer_module( 2025-09-07T07:12:45.5419829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5419922Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5420220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5420292Z self_outputs = self.self( 2025-09-07T07:12:45.5420579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5420656Z self.key(key_tensor) 2025-09-07T07:12:45.5420660Z 2025-09-07T07:12:45.5420802Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5421016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5421104Z return mod(**inputs) 2025-09-07T07:12:45.5421396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5421477Z outputs = self.mobilebert( 2025-09-07T07:12:45.5421771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5421861Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5422157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5422237Z layer_outputs = layer_module( 2025-09-07T07:12:45.5422558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5422655Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5422979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5423059Z self_outputs = self.self( 2025-09-07T07:12:45.5423384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5423467Z self.value(value_tensor) 2025-09-07T07:12:45.5423471Z 2025-09-07T07:12:45.5423565Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5423667Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5423786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5424014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5424089Z return mod(**inputs) 2025-09-07T07:12:45.5424403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5424491Z outputs = self.mobilebert( 2025-09-07T07:12:45.5424799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5424887Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5425191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5425310Z layer_outputs = layer_module( 2025-09-07T07:12:45.5425618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5425757Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5426083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5426222Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5426536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5426640Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5426644Z 2025-09-07T07:12:45.5426752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5426973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5427045Z return mod(**inputs) 2025-09-07T07:12:45.5427352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5427423Z outputs = self.mobilebert( 2025-09-07T07:12:45.5427718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5427797Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5428088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5428166Z layer_outputs = layer_module( 2025-09-07T07:12:45.5428452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5428622Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5428909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5429020Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5429311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5429394Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5429398Z 2025-09-07T07:12:45.5429509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5429705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5429769Z return mod(**inputs) 2025-09-07T07:12:45.5430056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5430129Z outputs = self.mobilebert( 2025-09-07T07:12:45.5430412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5430480Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5430760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5430830Z layer_outputs = layer_module( 2025-09-07T07:12:45.5431107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5431195Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5431469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5431597Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5431910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5432035Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5432317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5432409Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5432414Z 2025-09-07T07:12:45.5432520Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5432714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5432785Z return mod(**inputs) 2025-09-07T07:12:45.5433062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5433134Z outputs = self.mobilebert( 2025-09-07T07:12:45.5433420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5433491Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5433774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5433844Z layer_outputs = layer_module( 2025-09-07T07:12:45.5434135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5434258Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5434533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5434650Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5434926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5435020Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5435023Z 2025-09-07T07:12:45.5435124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5435319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5435390Z return mod(**inputs) 2025-09-07T07:12:45.5435671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5435748Z outputs = self.mobilebert( 2025-09-07T07:12:45.5436030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5436101Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5436397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5436471Z layer_outputs = layer_module( 2025-09-07T07:12:45.5436762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5436858Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5437152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5437265Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5437549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5437683Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5437687Z 2025-09-07T07:12:45.5437785Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5437989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5438084Z return mod(**inputs) 2025-09-07T07:12:45.5438364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5438441Z outputs = self.mobilebert( 2025-09-07T07:12:45.5438719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5438797Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5439076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5439155Z layer_outputs = layer_module( 2025-09-07T07:12:45.5439431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5439525Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5439816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5439943Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5440234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5440335Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5440338Z 2025-09-07T07:12:45.5440448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5440663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5440730Z return mod(**inputs) 2025-09-07T07:12:45.5441023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5441098Z outputs = self.mobilebert( 2025-09-07T07:12:45.5441387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5441459Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5441741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5441819Z layer_outputs = layer_module( 2025-09-07T07:12:45.5442102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5442204Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5442488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5442615Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5442906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5443030Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5443322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5443414Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5443419Z 2025-09-07T07:12:45.5443530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5443732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5443798Z return mod(**inputs) 2025-09-07T07:12:45.5444091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5444164Z outputs = self.mobilebert( 2025-09-07T07:12:45.5444485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5444557Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5444839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5444917Z layer_outputs = layer_module( 2025-09-07T07:12:45.5445201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5445304Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5445587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5445709Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5445990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5446077Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5446081Z 2025-09-07T07:12:45.5446193Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5446395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5446467Z return mod(**inputs) 2025-09-07T07:12:45.5446765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5446853Z outputs = self.mobilebert( 2025-09-07T07:12:45.5447144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5447216Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5447503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5447577Z layer_outputs = layer_module( 2025-09-07T07:12:45.5447863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5447957Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5448239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5448358Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5448640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5448761Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5448765Z 2025-09-07T07:12:45.5448866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5449072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5449141Z return mod(**inputs) 2025-09-07T07:12:45.5449428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5449508Z outputs = self.mobilebert( 2025-09-07T07:12:45.5449798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5449876Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5450159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5450230Z layer_outputs = layer_module( 2025-09-07T07:12:45.5450517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5450612Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5450938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5451065Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5451352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5451435Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5451438Z 2025-09-07T07:12:45.5451536Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5451735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5451797Z return mod(**inputs) 2025-09-07T07:12:45.5452074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5452145Z outputs = self.mobilebert( 2025-09-07T07:12:45.5452411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5452488Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5452756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5452832Z layer_outputs = layer_module( 2025-09-07T07:12:45.5453115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5453221Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5453499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5453618Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5453896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5454014Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5454292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5454379Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5454384Z 2025-09-07T07:12:45.5454480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5454681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5454744Z return mod(**inputs) 2025-09-07T07:12:45.5455022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5455091Z outputs = self.mobilebert( 2025-09-07T07:12:45.5455369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5455438Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5455714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5455792Z layer_outputs = layer_module( 2025-09-07T07:12:45.5456079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5456181Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5456465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5456581Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5456876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5457006Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5457010Z 2025-09-07T07:12:45.5457116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5457310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5457381Z return mod(**inputs) 2025-09-07T07:12:45.5457650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5457718Z outputs = self.mobilebert( 2025-09-07T07:12:45.5457995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5458064Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5458341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5458411Z layer_outputs = layer_module( 2025-09-07T07:12:45.5458678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5458776Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5459062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5459176Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5459460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5459575Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5459578Z 2025-09-07T07:12:45.5459677Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5459867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5459941Z return mod(**inputs) 2025-09-07T07:12:45.5460210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5460284Z outputs = self.mobilebert( 2025-09-07T07:12:45.5460554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5460623Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5460898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5460965Z layer_outputs = layer_module( 2025-09-07T07:12:45.5461237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5461327Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5461601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5461733Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5462009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5462102Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5462105Z 2025-09-07T07:12:45.5462205Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5462408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5462472Z return mod(**inputs) 2025-09-07T07:12:45.5462751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5462827Z outputs = self.mobilebert( 2025-09-07T07:12:45.5463140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5463217Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5463494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5463563Z layer_outputs = layer_module( 2025-09-07T07:12:45.5463851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5463946Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5464235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5464358Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5464647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5464770Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5465053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5465153Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5465175Z 2025-09-07T07:12:45.5465285Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5465530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5465600Z return mod(**inputs) 2025-09-07T07:12:45.5465969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5466068Z outputs = self.mobilebert( 2025-09-07T07:12:45.5466374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5466462Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5466766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5466843Z layer_outputs = layer_module( 2025-09-07T07:12:45.5467162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5467301Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5467584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5467667Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5467671Z 2025-09-07T07:12:45.5467772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5467996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5468066Z return mod(**inputs) 2025-09-07T07:12:45.5468375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5468450Z outputs = self.mobilebert( 2025-09-07T07:12:45.5468755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5468831Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5469129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5469210Z layer_outputs = layer_module( 2025-09-07T07:12:45.5469507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5469674Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5469977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5470096Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5470106Z 2025-09-07T07:12:45.5470214Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5470431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5470508Z return mod(**inputs) 2025-09-07T07:12:45.5470813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5470894Z outputs = self.mobilebert( 2025-09-07T07:12:45.5471198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5471279Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5471589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5471662Z layer_outputs = layer_module( 2025-09-07T07:12:45.5471969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5472159Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5472476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5472588Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5472592Z 2025-09-07T07:12:45.5472701Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5472920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5472994Z return mod(**inputs) 2025-09-07T07:12:45.5473301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5473376Z outputs = self.mobilebert( 2025-09-07T07:12:45.5473675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5473770Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5474038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5474114Z layer_outputs = layer_module( 2025-09-07T07:12:45.5474378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5474531Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5474808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5474929Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5475220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5475310Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5475315Z 2025-09-07T07:12:45.5475421Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5475619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5475683Z return mod(**inputs) 2025-09-07T07:12:45.5475972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5476070Z outputs = self.mobilebert( 2025-09-07T07:12:45.5476354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5476424Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5476700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5476778Z layer_outputs = layer_module( 2025-09-07T07:12:45.5477059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5477219Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5477493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5477617Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5477891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5477973Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5477977Z 2025-09-07T07:12:45.5478081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5478269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5478354Z return mod(**inputs) 2025-09-07T07:12:45.5478651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5478722Z outputs = self.mobilebert( 2025-09-07T07:12:45.5479004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5479074Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5479353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5479422Z layer_outputs = layer_module( 2025-09-07T07:12:45.5479696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5479847Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5480123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5480249Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5480515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5480637Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5480907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5481002Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5481005Z 2025-09-07T07:12:45.5481103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5481292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5481365Z return mod(**inputs) 2025-09-07T07:12:45.5481638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5481714Z outputs = self.mobilebert( 2025-09-07T07:12:45.5481983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5482054Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5482331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5482426Z layer_outputs = layer_module( 2025-09-07T07:12:45.5482702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5482858Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5483142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5483251Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5483523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5483610Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5483613Z 2025-09-07T07:12:45.5483713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5483914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5483975Z return mod(**inputs) 2025-09-07T07:12:45.5484247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5484321Z outputs = self.mobilebert( 2025-09-07T07:12:45.5484609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5484730Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5485002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5485076Z layer_outputs = layer_module( 2025-09-07T07:12:45.5485354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5485518Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5485808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5485916Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5486204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5486292Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5486572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5486669Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5486672Z 2025-09-07T07:12:45.5486773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5486982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5487046Z return mod(**inputs) 2025-09-07T07:12:45.5487345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5487414Z outputs = self.mobilebert( 2025-09-07T07:12:45.5487686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5487764Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5488035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5488108Z layer_outputs = layer_module( 2025-09-07T07:12:45.5488378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5488490Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5488768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5488835Z self_outputs = self.self( 2025-09-07T07:12:45.5489123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5489192Z self.query(query_tensor) 2025-09-07T07:12:45.5489195Z 2025-09-07T07:12:45.5489302Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5489555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5489617Z return mod(**inputs) 2025-09-07T07:12:45.5489895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5489965Z outputs = self.mobilebert( 2025-09-07T07:12:45.5490245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5490314Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5490587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5490661Z layer_outputs = layer_module( 2025-09-07T07:12:45.5490951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5491056Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5491325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5491398Z self_outputs = self.self( 2025-09-07T07:12:45.5491667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5491734Z self.key(key_tensor) 2025-09-07T07:12:45.5491737Z 2025-09-07T07:12:45.5491843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5492032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5492103Z return mod(**inputs) 2025-09-07T07:12:45.5492375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5492441Z outputs = self.mobilebert( 2025-09-07T07:12:45.5492715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5492783Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5493057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5493129Z layer_outputs = layer_module( 2025-09-07T07:12:45.5493394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5493481Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5493744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5493820Z self_outputs = self.self( 2025-09-07T07:12:45.5494089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5494162Z self.value(value_tensor) 2025-09-07T07:12:45.5494165Z 2025-09-07T07:12:45.5494244Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5494322Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5494428Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5494647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5494718Z return mod(**inputs) 2025-09-07T07:12:45.5494997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5495066Z outputs = self.mobilebert( 2025-09-07T07:12:45.5495354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5495426Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5495721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5495793Z layer_outputs = layer_module( 2025-09-07T07:12:45.5496083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5496176Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5496451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5496579Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5496872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5496966Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5496970Z 2025-09-07T07:12:45.5497085Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5497281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5497365Z return mod(**inputs) 2025-09-07T07:12:45.5497637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5497716Z outputs = self.mobilebert( 2025-09-07T07:12:45.5497987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5498060Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5498342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5498413Z layer_outputs = layer_module( 2025-09-07T07:12:45.5498697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5498857Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5499142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5499251Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5499531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5499619Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5499623Z 2025-09-07T07:12:45.5499723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5499924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5499988Z return mod(**inputs) 2025-09-07T07:12:45.5500270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5500348Z outputs = self.mobilebert( 2025-09-07T07:12:45.5500625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5500703Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5501007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5501082Z layer_outputs = layer_module( 2025-09-07T07:12:45.5501358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5501440Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5501725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5501849Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5502133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5502257Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5502541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5502630Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5502634Z 2025-09-07T07:12:45.5502734Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5502937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5503021Z return mod(**inputs) 2025-09-07T07:12:45.5503328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5503400Z outputs = self.mobilebert( 2025-09-07T07:12:45.5503679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5503760Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5504040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5504121Z layer_outputs = layer_module( 2025-09-07T07:12:45.5504402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5504499Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5504799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5504916Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5505205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5505290Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5505293Z 2025-09-07T07:12:45.5505402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5505604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5505669Z return mod(**inputs) 2025-09-07T07:12:45.5506039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5506116Z outputs = self.mobilebert( 2025-09-07T07:12:45.5506432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5506512Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5506832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5506916Z layer_outputs = layer_module( 2025-09-07T07:12:45.5507216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5507365Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5507669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5507802Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5508107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5508229Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5508234Z 2025-09-07T07:12:45.5508356Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5508569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5508648Z return mod(**inputs) 2025-09-07T07:12:45.5508954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5509033Z outputs = self.mobilebert( 2025-09-07T07:12:45.5509342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5509420Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5509728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5509822Z layer_outputs = layer_module( 2025-09-07T07:12:45.5510147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5510250Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5510548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5510693Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5510995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5511095Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5511101Z 2025-09-07T07:12:45.5511211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5511431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5511502Z return mod(**inputs) 2025-09-07T07:12:45.5511806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5511891Z outputs = self.mobilebert( 2025-09-07T07:12:45.5512186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5512271Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5512572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5512648Z layer_outputs = layer_module( 2025-09-07T07:12:45.5512953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5513050Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5513358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5513496Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5513793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5513932Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5514315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5514420Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5514424Z 2025-09-07T07:12:45.5514534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5514754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5514824Z return mod(**inputs) 2025-09-07T07:12:45.5515138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5515224Z outputs = self.mobilebert( 2025-09-07T07:12:45.5515532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5515618Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5515933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5516014Z layer_outputs = layer_module( 2025-09-07T07:12:45.5516330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5516433Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5516772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5516898Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5517232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5517327Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5517331Z 2025-09-07T07:12:45.5517447Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5517687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5517768Z return mod(**inputs) 2025-09-07T07:12:45.5518094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5518179Z outputs = self.mobilebert( 2025-09-07T07:12:45.5518497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5518592Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5518916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5519007Z layer_outputs = layer_module( 2025-09-07T07:12:45.5519323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5519439Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5519914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5520043Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5520364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5520492Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5520496Z 2025-09-07T07:12:45.5520617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5520836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5520916Z return mod(**inputs) 2025-09-07T07:12:45.5521235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5521375Z outputs = self.mobilebert( 2025-09-07T07:12:45.5521682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5521757Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5522065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5522141Z layer_outputs = layer_module( 2025-09-07T07:12:45.5522443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5522553Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5522856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5522999Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5523300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5523397Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5523401Z 2025-09-07T07:12:45.5523511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5523747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5523827Z return mod(**inputs) 2025-09-07T07:12:45.5524165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5524250Z outputs = self.mobilebert( 2025-09-07T07:12:45.5524553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5524630Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5524941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5525016Z layer_outputs = layer_module( 2025-09-07T07:12:45.5525330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5525431Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5525744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5525886Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5526186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5526326Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5526629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5526741Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5526744Z 2025-09-07T07:12:45.5526854Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5527066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5527153Z return mod(**inputs) 2025-09-07T07:12:45.5527434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5527514Z outputs = self.mobilebert( 2025-09-07T07:12:45.5527793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5527873Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5528148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5528259Z layer_outputs = layer_module( 2025-09-07T07:12:45.5528551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5528643Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5528930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5529042Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5529320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5529411Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5529415Z 2025-09-07T07:12:45.5529516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5529725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5529789Z return mod(**inputs) 2025-09-07T07:12:45.5530077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5530149Z outputs = self.mobilebert( 2025-09-07T07:12:45.5530439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5530520Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5530808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5530885Z layer_outputs = layer_module( 2025-09-07T07:12:45.5531161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5531256Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5531548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5531659Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5531951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5532063Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5532066Z 2025-09-07T07:12:45.5532175Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5532371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5532435Z return mod(**inputs) 2025-09-07T07:12:45.5532722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5532795Z outputs = self.mobilebert( 2025-09-07T07:12:45.5533079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5533150Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5533422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5533500Z layer_outputs = layer_module( 2025-09-07T07:12:45.5533777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5533876Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5534151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5534282Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5534591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5534673Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5534677Z 2025-09-07T07:12:45.5534785Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5535001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5535079Z return mod(**inputs) 2025-09-07T07:12:45.5535383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5535459Z outputs = self.mobilebert( 2025-09-07T07:12:45.5535775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5535854Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5536174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5536252Z layer_outputs = layer_module( 2025-09-07T07:12:45.5536569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5536679Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5537013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5537163Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5537445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5537574Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5537857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5537952Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5537956Z 2025-09-07T07:12:45.5538065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5538264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5538345Z return mod(**inputs) 2025-09-07T07:12:45.5538625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5538702Z outputs = self.mobilebert( 2025-09-07T07:12:45.5538977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5539049Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5539327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5539399Z layer_outputs = layer_module( 2025-09-07T07:12:45.5542210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5542335Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5542629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5542726Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5542731Z 2025-09-07T07:12:45.5542839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5543049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5543116Z return mod(**inputs) 2025-09-07T07:12:45.5543405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5543509Z outputs = self.mobilebert( 2025-09-07T07:12:45.5543803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5543923Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5544232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5544308Z layer_outputs = layer_module( 2025-09-07T07:12:45.5544608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5544743Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5545048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5545175Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5545179Z 2025-09-07T07:12:45.5545288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5545509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5545579Z return mod(**inputs) 2025-09-07T07:12:45.5546003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5546098Z outputs = self.mobilebert( 2025-09-07T07:12:45.5546442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5546533Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5546859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5546937Z layer_outputs = layer_module( 2025-09-07T07:12:45.5547297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5547466Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5547760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5547860Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5547865Z 2025-09-07T07:12:45.5547980Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5548182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5548247Z return mod(**inputs) 2025-09-07T07:12:45.5548550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5548624Z outputs = self.mobilebert( 2025-09-07T07:12:45.5548919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5549054Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5549336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5549415Z layer_outputs = layer_module( 2025-09-07T07:12:45.5549700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5549867Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5550152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5550285Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5550589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5550684Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5550688Z 2025-09-07T07:12:45.5550798Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5551005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5551080Z return mod(**inputs) 2025-09-07T07:12:45.5551368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5551448Z outputs = self.mobilebert( 2025-09-07T07:12:45.5551730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5551804Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5552097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5552171Z layer_outputs = layer_module( 2025-09-07T07:12:45.5552459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5552635Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5553704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5553844Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5554131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5554226Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5554232Z 2025-09-07T07:12:45.5554335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5554544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5554612Z return mod(**inputs) 2025-09-07T07:12:45.5554902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5554986Z outputs = self.mobilebert( 2025-09-07T07:12:45.5555279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5555359Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5555652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5555722Z layer_outputs = layer_module( 2025-09-07T07:12:45.5556025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5556187Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5556525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5556653Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5556947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5557082Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5557359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5557458Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5557461Z 2025-09-07T07:12:45.5557579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5557781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5557847Z return mod(**inputs) 2025-09-07T07:12:45.5558127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5558205Z outputs = self.mobilebert( 2025-09-07T07:12:45.5558482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5558563Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5558838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5558914Z layer_outputs = layer_module( 2025-09-07T07:12:45.5559193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5559357Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5559643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5559754Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5560056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5560153Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5560157Z 2025-09-07T07:12:45.5560267Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5560463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5560529Z return mod(**inputs) 2025-09-07T07:12:45.5560815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5560886Z outputs = self.mobilebert( 2025-09-07T07:12:45.5561167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5561239Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5561518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5561597Z layer_outputs = layer_module( 2025-09-07T07:12:45.5561873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5562042Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5562318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5562429Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5562733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5562817Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5563104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5563193Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5563197Z 2025-09-07T07:12:45.5563307Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5563504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5563567Z return mod(**inputs) 2025-09-07T07:12:45.5563855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5563944Z outputs = self.mobilebert( 2025-09-07T07:12:45.5564230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5564304Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5564580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5564659Z layer_outputs = layer_module( 2025-09-07T07:12:45.5564934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5565024Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5565298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5565376Z self_outputs = self.self( 2025-09-07T07:12:45.5565651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5565722Z self.query(query_tensor) 2025-09-07T07:12:45.5565725Z 2025-09-07T07:12:45.5565834Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5566031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5566117Z return mod(**inputs) 2025-09-07T07:12:45.5566425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5566498Z outputs = self.mobilebert( 2025-09-07T07:12:45.5566788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5566861Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5567160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5567228Z layer_outputs = layer_module( 2025-09-07T07:12:45.5567515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5567598Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5567872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5567953Z self_outputs = self.self( 2025-09-07T07:12:45.5568228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5568301Z self.key(key_tensor) 2025-09-07T07:12:45.5568304Z 2025-09-07T07:12:45.5568404Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5568599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5568669Z return mod(**inputs) 2025-09-07T07:12:45.5568963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5569042Z outputs = self.mobilebert( 2025-09-07T07:12:45.5569326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5569405Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5569674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5569742Z layer_outputs = layer_module( 2025-09-07T07:12:45.5570020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5570116Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5570393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5570462Z self_outputs = self.self( 2025-09-07T07:12:45.5570731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5570809Z self.value(value_tensor) 2025-09-07T07:12:45.5570813Z 2025-09-07T07:12:45.5570892Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5570976Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5571074Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5571264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5571335Z return mod(**inputs) 2025-09-07T07:12:45.5571606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5571681Z outputs = self.mobilebert( 2025-09-07T07:12:45.5571947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5572024Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5572306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5572374Z layer_outputs = layer_module( 2025-09-07T07:12:45.5572662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5572744Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5573019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5573139Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5573407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5573498Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5573501Z 2025-09-07T07:12:45.5573598Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5573794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5573858Z return mod(**inputs) 2025-09-07T07:12:45.5574138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5574209Z outputs = self.mobilebert( 2025-09-07T07:12:45.5574477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5574559Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5574828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5574923Z layer_outputs = layer_module( 2025-09-07T07:12:45.5575189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5575343Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5575622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5575729Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5576011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5576094Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5576114Z 2025-09-07T07:12:45.5576227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5576426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5576494Z return mod(**inputs) 2025-09-07T07:12:45.5576796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5576868Z outputs = self.mobilebert( 2025-09-07T07:12:45.5577166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5577240Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5577530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5577609Z layer_outputs = layer_module( 2025-09-07T07:12:45.5577910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5578001Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5578284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5578411Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5578709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5578855Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5579141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5579233Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5579237Z 2025-09-07T07:12:45.5579349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5579549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5579617Z return mod(**inputs) 2025-09-07T07:12:45.5579914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5579984Z outputs = self.mobilebert( 2025-09-07T07:12:45.5580273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5580345Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5580628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5580698Z layer_outputs = layer_module( 2025-09-07T07:12:45.5580980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5581084Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5581369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5581507Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5581792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5581877Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5581889Z 2025-09-07T07:12:45.5581992Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5582192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5582266Z return mod(**inputs) 2025-09-07T07:12:45.5582557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5582654Z outputs = self.mobilebert( 2025-09-07T07:12:45.5582940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5583014Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5583310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5583384Z layer_outputs = layer_module( 2025-09-07T07:12:45.5583684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5583782Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5584075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5584198Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5584494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5584618Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5584622Z 2025-09-07T07:12:45.5584723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5584950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5585020Z return mod(**inputs) 2025-09-07T07:12:45.5585339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5585426Z outputs = self.mobilebert( 2025-09-07T07:12:45.5585793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5585892Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5586196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5586274Z layer_outputs = layer_module( 2025-09-07T07:12:45.5586591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5586696Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5587012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5587142Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5587439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5587526Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5587531Z 2025-09-07T07:12:45.5587637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5587847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5587940Z return mod(**inputs) 2025-09-07T07:12:45.5588235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5588310Z outputs = self.mobilebert( 2025-09-07T07:12:45.5588597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5588679Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5588965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5589042Z layer_outputs = layer_module( 2025-09-07T07:12:45.5589325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5589447Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5589738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5589865Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5590160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5590286Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5590579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5590671Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5590674Z 2025-09-07T07:12:45.5590779Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5590988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5591057Z return mod(**inputs) 2025-09-07T07:12:45.5591414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5591486Z outputs = self.mobilebert( 2025-09-07T07:12:45.5591824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5591912Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5592195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5592273Z layer_outputs = layer_module( 2025-09-07T07:12:45.5592555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5592658Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5592943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5593059Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5593357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5593443Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5593447Z 2025-09-07T07:12:45.5593559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5593756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5593829Z return mod(**inputs) 2025-09-07T07:12:45.5594113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5594185Z outputs = self.mobilebert( 2025-09-07T07:12:45.5594486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5594582Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5594891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5594969Z layer_outputs = layer_module( 2025-09-07T07:12:45.5595271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5595379Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5595675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5595820Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5596119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5596246Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5596250Z 2025-09-07T07:12:45.5596358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5596571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5596649Z return mod(**inputs) 2025-09-07T07:12:45.5596954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5597035Z outputs = self.mobilebert( 2025-09-07T07:12:45.5597331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5597410Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5597716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5597792Z layer_outputs = layer_module( 2025-09-07T07:12:45.5598095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5598223Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5598525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5598653Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5598936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5599031Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5599036Z 2025-09-07T07:12:45.5599142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5599354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5599425Z return mod(**inputs) 2025-09-07T07:12:45.5599712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5599792Z outputs = self.mobilebert( 2025-09-07T07:12:45.5600098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5600175Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5600462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5600542Z layer_outputs = layer_module( 2025-09-07T07:12:45.5600828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5600933Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5601237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5601370Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5601657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5601784Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5602076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5602171Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5602174Z 2025-09-07T07:12:45.5602285Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5602503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5602572Z return mod(**inputs) 2025-09-07T07:12:45.5602866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5602937Z outputs = self.mobilebert( 2025-09-07T07:12:45.5603227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5603303Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5603591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5603665Z layer_outputs = layer_module( 2025-09-07T07:12:45.5603948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5604051Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5604337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5604460Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5604769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5604862Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5604872Z 2025-09-07T07:12:45.5604996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5605216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5605294Z return mod(**inputs) 2025-09-07T07:12:45.5605601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5605686Z outputs = self.mobilebert( 2025-09-07T07:12:45.5605986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5606064Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5606374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5606448Z layer_outputs = layer_module( 2025-09-07T07:12:45.5606761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5606855Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5607139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5607263Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5607565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5607713Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5607717Z 2025-09-07T07:12:45.5607825Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5608043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5608113Z return mod(**inputs) 2025-09-07T07:12:45.5608413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5608494Z outputs = self.mobilebert( 2025-09-07T07:12:45.5608792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5608891Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5609189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5609267Z layer_outputs = layer_module( 2025-09-07T07:12:45.5609574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5609675Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5609983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5610117Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5610422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5610511Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5610516Z 2025-09-07T07:12:45.5610623Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5610843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5610914Z return mod(**inputs) 2025-09-07T07:12:45.5611222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5611314Z outputs = self.mobilebert( 2025-09-07T07:12:45.5611629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5611715Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5612012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5612095Z layer_outputs = layer_module( 2025-09-07T07:12:45.5612397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5612503Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5612803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5612936Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5613243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5613374Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5613682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5613778Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5613781Z 2025-09-07T07:12:45.5613897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5614113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5614203Z return mod(**inputs) 2025-09-07T07:12:45.5614517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5614594Z outputs = self.mobilebert( 2025-09-07T07:12:45.5614906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5614985Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5615290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5615375Z layer_outputs = layer_module( 2025-09-07T07:12:45.5615679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5615834Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5616135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5616228Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5616238Z 2025-09-07T07:12:45.5616347Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5616558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5616636Z return mod(**inputs) 2025-09-07T07:12:45.5616939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5617020Z outputs = self.mobilebert( 2025-09-07T07:12:45.5617320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5617398Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5617700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5617776Z layer_outputs = layer_module( 2025-09-07T07:12:45.5618078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5618223Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5618542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5618670Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5618674Z 2025-09-07T07:12:45.5618783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5619006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5619077Z return mod(**inputs) 2025-09-07T07:12:45.5619401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5619481Z outputs = self.mobilebert( 2025-09-07T07:12:45.5619921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5620015Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5620322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5620409Z layer_outputs = layer_module( 2025-09-07T07:12:45.5620722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5620898Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5621215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5621375Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5621380Z 2025-09-07T07:12:45.5621504Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5621727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5621807Z return mod(**inputs) 2025-09-07T07:12:45.5622123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5622203Z outputs = self.mobilebert( 2025-09-07T07:12:45.5622520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5622599Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5622950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5623032Z layer_outputs = layer_module( 2025-09-07T07:12:45.5623342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5623528Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5623838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5623982Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5624288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5624396Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5624403Z 2025-09-07T07:12:45.5624515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5624731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5624813Z return mod(**inputs) 2025-09-07T07:12:45.5625126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5625235Z outputs = self.mobilebert( 2025-09-07T07:12:45.5625565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5625648Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5626036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5626119Z layer_outputs = layer_module( 2025-09-07T07:12:45.5626440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5626617Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5626945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5627082Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5627392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5627496Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5627500Z 2025-09-07T07:12:45.5627612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5627835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5627905Z return mod(**inputs) 2025-09-07T07:12:45.5628238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5628343Z outputs = self.mobilebert( 2025-09-07T07:12:45.5628661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5628750Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5629071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5629159Z layer_outputs = layer_module( 2025-09-07T07:12:45.5629476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5629650Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5629974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5630134Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5630529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5630661Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5630994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5631096Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5631100Z 2025-09-07T07:12:45.5631212Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5631441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5631514Z return mod(**inputs) 2025-09-07T07:12:45.5631846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5631925Z outputs = self.mobilebert( 2025-09-07T07:12:45.5632246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5632336Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5632669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5632771Z layer_outputs = layer_module( 2025-09-07T07:12:45.5633094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5633279Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5633652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5633776Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5634106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5634199Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5634204Z 2025-09-07T07:12:45.5634325Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5634540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5634612Z return mod(**inputs) 2025-09-07T07:12:45.5634929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5635000Z outputs = self.mobilebert( 2025-09-07T07:12:45.5635291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5635366Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5635671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5635766Z layer_outputs = layer_module( 2025-09-07T07:12:45.5636076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5636263Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5636556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5636676Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5636962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5637067Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5637353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5637448Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5637451Z 2025-09-07T07:12:45.5637563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5637764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5637836Z return mod(**inputs) 2025-09-07T07:12:45.5638126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5638198Z outputs = self.mobilebert( 2025-09-07T07:12:45.5638488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5638562Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5638852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5638927Z layer_outputs = layer_module( 2025-09-07T07:12:45.5639211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5639320Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5639618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5639699Z self_outputs = self.self( 2025-09-07T07:12:45.5639982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5640061Z self.query(query_tensor) 2025-09-07T07:12:45.5640066Z 2025-09-07T07:12:45.5640169Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5640368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5640443Z return mod(**inputs) 2025-09-07T07:12:45.5640732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5640811Z outputs = self.mobilebert( 2025-09-07T07:12:45.5641094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5641167Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5641453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5641525Z layer_outputs = layer_module( 2025-09-07T07:12:45.5641827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5641918Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5642242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5642316Z self_outputs = self.self( 2025-09-07T07:12:45.5642614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5642694Z self.key(key_tensor) 2025-09-07T07:12:45.5642698Z 2025-09-07T07:12:45.5642806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5643031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5643095Z return mod(**inputs) 2025-09-07T07:12:45.5643378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5643476Z outputs = self.mobilebert( 2025-09-07T07:12:45.5643762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5643845Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5644129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5644200Z layer_outputs = layer_module( 2025-09-07T07:12:45.5644491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5644576Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5644865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5644939Z self_outputs = self.self( 2025-09-07T07:12:45.5645249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5645327Z self.value(value_tensor) 2025-09-07T07:12:45.5645330Z 2025-09-07T07:12:45.5645419Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5645513Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5645623Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5645855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5645925Z return mod(**inputs) 2025-09-07T07:12:45.5646242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5646327Z outputs = self.mobilebert( 2025-09-07T07:12:45.5646627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5646714Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5647014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5647088Z layer_outputs = layer_module( 2025-09-07T07:12:45.5647379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5647465Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5647758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5647883Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5648173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5648258Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5648262Z 2025-09-07T07:12:45.5648366Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5648572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5648658Z return mod(**inputs) 2025-09-07T07:12:45.5648947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5649021Z outputs = self.mobilebert( 2025-09-07T07:12:45.5649306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5649391Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5649690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5649773Z layer_outputs = layer_module( 2025-09-07T07:12:45.5650093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5650267Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5650552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5650666Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5650959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5651041Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5651044Z 2025-09-07T07:12:45.5651154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5651354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5651428Z return mod(**inputs) 2025-09-07T07:12:45.5651716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5651788Z outputs = self.mobilebert( 2025-09-07T07:12:45.5652076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5652149Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5652452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5652540Z layer_outputs = layer_module( 2025-09-07T07:12:45.5652823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5652917Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5653196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5653327Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5653611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5653745Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5654026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5654122Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5654126Z 2025-09-07T07:12:45.5654235Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5654432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5654507Z return mod(**inputs) 2025-09-07T07:12:45.5654790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5654871Z outputs = self.mobilebert( 2025-09-07T07:12:45.5655213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5655286Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5655591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5655667Z layer_outputs = layer_module( 2025-09-07T07:12:45.5655974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5656076Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5656371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5656522Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5656821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5656920Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5656924Z 2025-09-07T07:12:45.5657034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5657246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5657325Z return mod(**inputs) 2025-09-07T07:12:45.5657630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5657711Z outputs = self.mobilebert( 2025-09-07T07:12:45.5658012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5658098Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5658403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5658480Z layer_outputs = layer_module( 2025-09-07T07:12:45.5658793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5658911Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5659231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5659353Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5659654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5659783Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5659789Z 2025-09-07T07:12:45.5659897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5660121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5660190Z return mod(**inputs) 2025-09-07T07:12:45.5660501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5660576Z outputs = self.mobilebert( 2025-09-07T07:12:45.5660875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5660959Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5661259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5661342Z layer_outputs = layer_module( 2025-09-07T07:12:45.5661642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5661760Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5662065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5662205Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5662514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5662605Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5662608Z 2025-09-07T07:12:45.5662724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5662932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5663001Z return mod(**inputs) 2025-09-07T07:12:45.5663327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5663405Z outputs = self.mobilebert( 2025-09-07T07:12:45.5663710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5663787Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5664085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5664170Z layer_outputs = layer_module( 2025-09-07T07:12:45.5664467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5664577Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5664879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5665027Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5665333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5665467Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5665872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5666001Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5666005Z 2025-09-07T07:12:45.5666130Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5666353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5666428Z return mod(**inputs) 2025-09-07T07:12:45.5666752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5666842Z outputs = self.mobilebert( 2025-09-07T07:12:45.5667155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5667234Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5667546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5667623Z layer_outputs = layer_module( 2025-09-07T07:12:45.5667925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5668034Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5668342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5668476Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5668754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5668857Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5668860Z 2025-09-07T07:12:45.5668968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5669162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5669235Z return mod(**inputs) 2025-09-07T07:12:45.5669533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5669619Z outputs = self.mobilebert( 2025-09-07T07:12:45.5669920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5670017Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5670330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5670409Z layer_outputs = layer_module( 2025-09-07T07:12:45.5670726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5670823Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5671106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5671229Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5671516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5671632Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5671637Z 2025-09-07T07:12:45.5671737Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5671941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5672007Z return mod(**inputs) 2025-09-07T07:12:45.5672291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5672384Z outputs = self.mobilebert( 2025-09-07T07:12:45.5672676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5672758Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5673040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5673109Z layer_outputs = layer_module( 2025-09-07T07:12:45.5673394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5673489Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5673773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5673900Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5674189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5674278Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5674281Z 2025-09-07T07:12:45.5674384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5674589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5674656Z return mod(**inputs) 2025-09-07T07:12:45.5674956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5675049Z outputs = self.mobilebert( 2025-09-07T07:12:45.5675352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5675437Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5675740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5675825Z layer_outputs = layer_module( 2025-09-07T07:12:45.5676126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5676234Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5676532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5676681Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5676985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5677112Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5677418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5677516Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5677519Z 2025-09-07T07:12:45.5677626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5677855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5677924Z return mod(**inputs) 2025-09-07T07:12:45.5678240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5678311Z outputs = self.mobilebert( 2025-09-07T07:12:45.5678597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5678671Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5678968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5679050Z layer_outputs = layer_module( 2025-09-07T07:12:45.5679364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5679467Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5679757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5679873Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5680178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5680264Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5680268Z 2025-09-07T07:12:45.5680380Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5680586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5680658Z return mod(**inputs) 2025-09-07T07:12:45.5680957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5681028Z outputs = self.mobilebert( 2025-09-07T07:12:45.5681329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5681404Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5681703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5681796Z layer_outputs = layer_module( 2025-09-07T07:12:45.5682080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5682186Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5682475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5682595Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5682877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5682999Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5683020Z 2025-09-07T07:12:45.5683132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5683344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5683425Z return mod(**inputs) 2025-09-07T07:12:45.5683726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5683810Z outputs = self.mobilebert( 2025-09-07T07:12:45.5684109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5684184Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5684494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5684565Z layer_outputs = layer_module( 2025-09-07T07:12:45.5684862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5684957Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5685250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5685378Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5685693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5685808Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5685812Z 2025-09-07T07:12:45.5685922Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5686139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5686209Z return mod(**inputs) 2025-09-07T07:12:45.5686519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5686604Z outputs = self.mobilebert( 2025-09-07T07:12:45.5686901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5686985Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5687287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5687371Z layer_outputs = layer_module( 2025-09-07T07:12:45.5687671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5687770Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5688074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5688207Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5688534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5688664Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5688967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5689076Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5689080Z 2025-09-07T07:12:45.5689190Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5689410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5689480Z return mod(**inputs) 2025-09-07T07:12:45.5689797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5689909Z outputs = self.mobilebert( 2025-09-07T07:12:45.5690216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5690299Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5690618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5690703Z layer_outputs = layer_module( 2025-09-07T07:12:45.5691025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5691170Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5691482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5691573Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5691577Z 2025-09-07T07:12:45.5691693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5691907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5691984Z return mod(**inputs) 2025-09-07T07:12:45.5692306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5692382Z outputs = self.mobilebert( 2025-09-07T07:12:45.5692711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5692788Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5693097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5693174Z layer_outputs = layer_module( 2025-09-07T07:12:45.5693484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5693617Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5693931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5694058Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5694062Z 2025-09-07T07:12:45.5694172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5694404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5694476Z return mod(**inputs) 2025-09-07T07:12:45.5694795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5694879Z outputs = self.mobilebert( 2025-09-07T07:12:45.5695187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5695287Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5695596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5695673Z layer_outputs = layer_module( 2025-09-07T07:12:45.5696004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5696179Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5696490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5696592Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5696644Z 2025-09-07T07:12:45.5696763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5696977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5697049Z return mod(**inputs) 2025-09-07T07:12:45.5697364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5697441Z outputs = self.mobilebert( 2025-09-07T07:12:45.5697758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5697835Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5698148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5698233Z layer_outputs = layer_module( 2025-09-07T07:12:45.5698548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5698726Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5699039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5699199Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5699524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5699624Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5699628Z 2025-09-07T07:12:45.5699746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5699964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5700045Z return mod(**inputs) 2025-09-07T07:12:45.5700373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5700452Z outputs = self.mobilebert( 2025-09-07T07:12:45.5700769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5700847Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5701169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5701248Z layer_outputs = layer_module( 2025-09-07T07:12:45.5701562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5701734Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5702051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5702197Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5702518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5702620Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5702623Z 2025-09-07T07:12:45.5702738Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5702964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5703035Z return mod(**inputs) 2025-09-07T07:12:45.5703346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5703429Z outputs = self.mobilebert( 2025-09-07T07:12:45.5703746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5703849Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5704164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5704242Z layer_outputs = layer_module( 2025-09-07T07:12:45.5704568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5704744Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5705065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5705200Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5705531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5705667Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5706058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5706174Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5706178Z 2025-09-07T07:12:45.5706319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5706563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5706638Z return mod(**inputs) 2025-09-07T07:12:45.5706953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5707041Z outputs = self.mobilebert( 2025-09-07T07:12:45.5707365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5707455Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5707771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5707847Z layer_outputs = layer_module( 2025-09-07T07:12:45.5708182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5708363Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5708686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5708808Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5709141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5709234Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5709238Z 2025-09-07T07:12:45.5709366Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5709589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5709659Z return mod(**inputs) 2025-09-07T07:12:45.5709984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5710061Z outputs = self.mobilebert( 2025-09-07T07:12:45.5710377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5710455Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5710762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5710866Z layer_outputs = layer_module( 2025-09-07T07:12:45.5711176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5711361Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5711677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5711797Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5712115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5712211Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5712526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5712628Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5712632Z 2025-09-07T07:12:45.5712749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5712967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5713041Z return mod(**inputs) 2025-09-07T07:12:45.5713373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5713452Z outputs = self.mobilebert( 2025-09-07T07:12:45.5713782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5713864Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5714188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5714274Z layer_outputs = layer_module( 2025-09-07T07:12:45.5714591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5714698Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5715008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5715095Z self_outputs = self.self( 2025-09-07T07:12:45.5715406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5715484Z self.query(query_tensor) 2025-09-07T07:12:45.5715488Z 2025-09-07T07:12:45.5715610Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5715825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5715905Z return mod(**inputs) 2025-09-07T07:12:45.5716219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5716315Z outputs = self.mobilebert( 2025-09-07T07:12:45.5716631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5716711Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5717033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5717113Z layer_outputs = layer_module( 2025-09-07T07:12:45.5717423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5717526Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5717835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5717936Z self_outputs = self.self( 2025-09-07T07:12:45.5718249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5718331Z self.key(key_tensor) 2025-09-07T07:12:45.5718335Z 2025-09-07T07:12:45.5718449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5718671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5718751Z return mod(**inputs) 2025-09-07T07:12:45.5719065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5719150Z outputs = self.mobilebert( 2025-09-07T07:12:45.5719460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5719703Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5720036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5720118Z layer_outputs = layer_module( 2025-09-07T07:12:45.5720440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5720581Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5720937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5721015Z self_outputs = self.self( 2025-09-07T07:12:45.5721315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5721398Z self.value(value_tensor) 2025-09-07T07:12:45.5721402Z 2025-09-07T07:12:45.5721497Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5721595Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5721707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5721921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5722003Z return mod(**inputs) 2025-09-07T07:12:45.5722307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5722391Z outputs = self.mobilebert( 2025-09-07T07:12:45.5722694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5722771Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5723080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5723158Z layer_outputs = layer_module( 2025-09-07T07:12:45.5723467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5723582Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5723892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5724027Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5724330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5724431Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5724435Z 2025-09-07T07:12:45.5724545Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5724764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5724857Z return mod(**inputs) 2025-09-07T07:12:45.5725166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5725249Z outputs = self.mobilebert( 2025-09-07T07:12:45.5725555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5725641Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5725949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5726032Z layer_outputs = layer_module( 2025-09-07T07:12:45.5726338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5726513Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5726830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5726951Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5727263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5727351Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5727369Z 2025-09-07T07:12:45.5727487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5727712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5727783Z return mod(**inputs) 2025-09-07T07:12:45.5728098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5728176Z outputs = self.mobilebert( 2025-09-07T07:12:45.5728482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5728558Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5728858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5728940Z layer_outputs = layer_module( 2025-09-07T07:12:45.5729242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5729348Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5729623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5729743Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5730029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5730157Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5730462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5730554Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5730557Z 2025-09-07T07:12:45.5730667Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5730865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5730932Z return mod(**inputs) 2025-09-07T07:12:45.5731234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5731306Z outputs = self.mobilebert( 2025-09-07T07:12:45.5731592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5731690Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5731992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5732072Z layer_outputs = layer_module( 2025-09-07T07:12:45.5732359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5732462Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5732749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5732868Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5733154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5733239Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5733243Z 2025-09-07T07:12:45.5733353Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5733557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5733629Z return mod(**inputs) 2025-09-07T07:12:45.5733951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5734022Z outputs = self.mobilebert( 2025-09-07T07:12:45.5734325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5734399Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5734689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5734761Z layer_outputs = layer_module( 2025-09-07T07:12:45.5735046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5735143Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5735419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5735540Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5735825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5735947Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5735950Z 2025-09-07T07:12:45.5736054Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5736251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5736325Z return mod(**inputs) 2025-09-07T07:12:45.5736611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5736710Z outputs = self.mobilebert( 2025-09-07T07:12:45.5737012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5737106Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5737402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5737472Z layer_outputs = layer_module( 2025-09-07T07:12:45.5737763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5737855Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5738151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5738276Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5738551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5738642Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5738646Z 2025-09-07T07:12:45.5738746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5738947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5739013Z return mod(**inputs) 2025-09-07T07:12:45.5739306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5739378Z outputs = self.mobilebert( 2025-09-07T07:12:45.5739659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5739741Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5740025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5740103Z layer_outputs = layer_module( 2025-09-07T07:12:45.5740400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5740511Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5740806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5740933Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5741225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5741348Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5741647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5741740Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5741743Z 2025-09-07T07:12:45.5741848Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5742056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5742124Z return mod(**inputs) 2025-09-07T07:12:45.5742420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5742489Z outputs = self.mobilebert( 2025-09-07T07:12:45.5742772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5742853Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5743141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5743241Z layer_outputs = layer_module( 2025-09-07T07:12:45.5743523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5743625Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5743913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5744027Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5744319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5744420Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5744423Z 2025-09-07T07:12:45.5744532Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5744735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5744800Z return mod(**inputs) 2025-09-07T07:12:45.5745093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5745163Z outputs = self.mobilebert( 2025-09-07T07:12:45.5745457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5745532Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5745900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5745984Z layer_outputs = layer_module( 2025-09-07T07:12:45.5746306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5746422Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5746730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5746878Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5747206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5747332Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5747344Z 2025-09-07T07:12:45.5747458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5747672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5747749Z return mod(**inputs) 2025-09-07T07:12:45.5748033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5748117Z outputs = self.mobilebert( 2025-09-07T07:12:45.5748402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5748479Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5748770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5748843Z layer_outputs = layer_module( 2025-09-07T07:12:45.5749133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5749228Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5749514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5749647Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5749946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5750041Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5750044Z 2025-09-07T07:12:45.5750148Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5750357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5750424Z return mod(**inputs) 2025-09-07T07:12:45.5750708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5750788Z outputs = self.mobilebert( 2025-09-07T07:12:45.5751070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5751170Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5751455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5751527Z layer_outputs = layer_module( 2025-09-07T07:12:45.5751823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5751920Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5752210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5752335Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5752626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5752751Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5753040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5753142Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5753145Z 2025-09-07T07:12:45.5753262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5753473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5753554Z return mod(**inputs) 2025-09-07T07:12:45.5753852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5753924Z outputs = self.mobilebert( 2025-09-07T07:12:45.5754209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5754294Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5754578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5754658Z layer_outputs = layer_module( 2025-09-07T07:12:45.5754945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5755039Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5755331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5755447Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5755738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5755824Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5755828Z 2025-09-07T07:12:45.5755940Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5756168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5756233Z return mod(**inputs) 2025-09-07T07:12:45.5756531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5756603Z outputs = self.mobilebert( 2025-09-07T07:12:45.5756897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5756971Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5757254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5757355Z layer_outputs = layer_module( 2025-09-07T07:12:45.5757640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5757742Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5758028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5758143Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5758435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5758548Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5758551Z 2025-09-07T07:12:45.5758661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5758861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5758935Z return mod(**inputs) 2025-09-07T07:12:45.5759224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5759296Z outputs = self.mobilebert( 2025-09-07T07:12:45.5759655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5759748Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5760055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5760126Z layer_outputs = layer_module( 2025-09-07T07:12:45.5760410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5760513Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5760797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5760932Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5761218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5761310Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5761313Z 2025-09-07T07:12:45.5761417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5761619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5761694Z return mod(**inputs) 2025-09-07T07:12:45.5761984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5762062Z outputs = self.mobilebert( 2025-09-07T07:12:45.5762342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5762416Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5762731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5762801Z layer_outputs = layer_module( 2025-09-07T07:12:45.5763087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5763191Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5763472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5763594Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5763869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5764013Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5764289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5764386Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5764389Z 2025-09-07T07:12:45.5764490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5764691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5764757Z return mod(**inputs) 2025-09-07T07:12:45.5765034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5765111Z outputs = self.mobilebert( 2025-09-07T07:12:45.5765388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5765467Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5765749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5765823Z layer_outputs = layer_module( 2025-09-07T07:12:45.5766131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5766257Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5766560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5766646Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5766649Z 2025-09-07T07:12:45.5766760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5766976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5767042Z return mod(**inputs) 2025-09-07T07:12:45.5767336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5767408Z outputs = self.mobilebert( 2025-09-07T07:12:45.5767698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5767770Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5768050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5768128Z layer_outputs = layer_module( 2025-09-07T07:12:45.5768406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5768532Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5768815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5768941Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5768953Z 2025-09-07T07:12:45.5769052Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5769249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5769321Z return mod(**inputs) 2025-09-07T07:12:45.5769606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5769683Z outputs = self.mobilebert( 2025-09-07T07:12:45.5769961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5770031Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5770329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5770399Z layer_outputs = layer_module( 2025-09-07T07:12:45.5770678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5770838Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5771121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5771216Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5771219Z 2025-09-07T07:12:45.5771319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5771518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5771586Z return mod(**inputs) 2025-09-07T07:12:45.5771872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5771943Z outputs = self.mobilebert( 2025-09-07T07:12:45.5772214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5772291Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5772584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5772676Z layer_outputs = layer_module( 2025-09-07T07:12:45.5772952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5773109Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5773395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5773518Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5773801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5773893Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5773897Z 2025-09-07T07:12:45.5774006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5774200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5774265Z return mod(**inputs) 2025-09-07T07:12:45.5774552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5774621Z outputs = self.mobilebert( 2025-09-07T07:12:45.5774905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5774976Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5775278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5775348Z layer_outputs = layer_module( 2025-09-07T07:12:45.5775632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5775801Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5776084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5776216Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5776502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5776604Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5776617Z 2025-09-07T07:12:45.5776720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5776934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5777005Z return mod(**inputs) 2025-09-07T07:12:45.5777292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5777370Z outputs = self.mobilebert( 2025-09-07T07:12:45.5777653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5777724Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5778011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5778080Z layer_outputs = layer_module( 2025-09-07T07:12:45.5778370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5778527Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5778825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5778986Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5779258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5779382Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5779658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5779757Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5779761Z 2025-09-07T07:12:45.5779862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5780056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5780127Z return mod(**inputs) 2025-09-07T07:12:45.5780406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5780485Z outputs = self.mobilebert( 2025-09-07T07:12:45.5780757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5780827Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5781105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5781176Z layer_outputs = layer_module( 2025-09-07T07:12:45.5781455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5781636Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5781925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5782035Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5782314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5782404Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5782408Z 2025-09-07T07:12:45.5782509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5782732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5782797Z return mod(**inputs) 2025-09-07T07:12:45.5783079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5783159Z outputs = self.mobilebert( 2025-09-07T07:12:45.5783444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5783526Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5783812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5783893Z layer_outputs = layer_module( 2025-09-07T07:12:45.5784178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5784344Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5784637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5784750Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5785071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5785165Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5785488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5785587Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5785591Z 2025-09-07T07:12:45.5785880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5786112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5786186Z return mod(**inputs) 2025-09-07T07:12:45.5786501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5786580Z outputs = self.mobilebert( 2025-09-07T07:12:45.5786887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5786975Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5787279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5787364Z layer_outputs = layer_module( 2025-09-07T07:12:45.5787664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5787759Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5788071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5788169Z self_outputs = self.self( 2025-09-07T07:12:45.5788464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5788534Z self.query(query_tensor) 2025-09-07T07:12:45.5788538Z 2025-09-07T07:12:45.5788647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5788840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5788904Z return mod(**inputs) 2025-09-07T07:12:45.5789198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5789268Z outputs = self.mobilebert( 2025-09-07T07:12:45.5789551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5789638Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5789919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5789999Z layer_outputs = layer_module( 2025-09-07T07:12:45.5790291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5790389Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5790671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5790749Z self_outputs = self.self( 2025-09-07T07:12:45.5791033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5791100Z self.key(key_tensor) 2025-09-07T07:12:45.5791104Z 2025-09-07T07:12:45.5791215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5791413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5791486Z return mod(**inputs) 2025-09-07T07:12:45.5791848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5791918Z outputs = self.mobilebert( 2025-09-07T07:12:45.5792218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5792291Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5792573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5792643Z layer_outputs = layer_module( 2025-09-07T07:12:45.5792931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5793017Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5793296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5793372Z self_outputs = self.self( 2025-09-07T07:12:45.5793654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5793731Z self.value(value_tensor) 2025-09-07T07:12:45.5793734Z 2025-09-07T07:12:45.5793817Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5793898Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5794009Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5794204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5794280Z return mod(**inputs) 2025-09-07T07:12:45.5794565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5794655Z outputs = self.mobilebert( 2025-09-07T07:12:45.5794945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5795018Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5795306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5795377Z layer_outputs = layer_module( 2025-09-07T07:12:45.5795667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5795751Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5796049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5796185Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5796468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5796561Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5796564Z 2025-09-07T07:12:45.5796666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5796876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5796950Z return mod(**inputs) 2025-09-07T07:12:45.5797237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5797316Z outputs = self.mobilebert( 2025-09-07T07:12:45.5797600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5797681Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5797963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5798035Z layer_outputs = layer_module( 2025-09-07T07:12:45.5798338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5798517Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5798807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5798920Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5799201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5799291Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5799296Z 2025-09-07T07:12:45.5799399Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5799605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5799673Z return mod(**inputs) 2025-09-07T07:12:45.5799968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5800041Z outputs = self.mobilebert( 2025-09-07T07:12:45.5800319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5800409Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5800680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5800759Z layer_outputs = layer_module( 2025-09-07T07:12:45.5801077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5801159Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5801441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5801564Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5801846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5801971Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5802273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5802383Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5802386Z 2025-09-07T07:12:45.5802489Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5802702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5802768Z return mod(**inputs) 2025-09-07T07:12:45.5803073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5803147Z outputs = self.mobilebert( 2025-09-07T07:12:45.5803441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5803522Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5803814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5803893Z layer_outputs = layer_module( 2025-09-07T07:12:45.5804184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5804290Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5804613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5804726Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5805022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5805109Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5805113Z 2025-09-07T07:12:45.5805220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5805415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5805482Z return mod(**inputs) 2025-09-07T07:12:45.5805772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5805845Z outputs = self.mobilebert( 2025-09-07T07:12:45.5806133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5806208Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5806499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5806571Z layer_outputs = layer_module( 2025-09-07T07:12:45.5806852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5806956Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5807244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5807382Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5807663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5807777Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5807789Z 2025-09-07T07:12:45.5807893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5808097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5808169Z return mod(**inputs) 2025-09-07T07:12:45.5808453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5808530Z outputs = self.mobilebert( 2025-09-07T07:12:45.5808828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5808903Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5809193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5809263Z layer_outputs = layer_module( 2025-09-07T07:12:45.5809554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5809650Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5809933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5810066Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5810347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5810441Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5810446Z 2025-09-07T07:12:45.5810547Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5810752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5810819Z return mod(**inputs) 2025-09-07T07:12:45.5812070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5812174Z outputs = self.mobilebert( 2025-09-07T07:12:45.5812472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5812552Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5812847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5812922Z layer_outputs = layer_module( 2025-09-07T07:12:45.5813230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5813331Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5813638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5813775Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5814082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5814216Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5814517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5814628Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5814632Z 2025-09-07T07:12:45.5814759Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5814978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5815047Z return mod(**inputs) 2025-09-07T07:12:45.5815350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5815434Z outputs = self.mobilebert( 2025-09-07T07:12:45.5815734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5815822Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5816123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5816225Z layer_outputs = layer_module( 2025-09-07T07:12:45.5816525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5816636Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5816931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5817046Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5817348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5817440Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5817443Z 2025-09-07T07:12:45.5817553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5817775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5817846Z return mod(**inputs) 2025-09-07T07:12:45.5818163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5818242Z outputs = self.mobilebert( 2025-09-07T07:12:45.5818547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5818640Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5818954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5819038Z layer_outputs = layer_module( 2025-09-07T07:12:45.5819334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5819442Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5819956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5820086Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5820399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5820523Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5820527Z 2025-09-07T07:12:45.5820648Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5820866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5820947Z return mod(**inputs) 2025-09-07T07:12:45.5821250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5821325Z outputs = self.mobilebert( 2025-09-07T07:12:45.5821642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5821768Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5822073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5822150Z layer_outputs = layer_module( 2025-09-07T07:12:45.5822456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5822570Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5822876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5823022Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5823328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5823461Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5823467Z 2025-09-07T07:12:45.5823580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5823799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5823879Z return mod(**inputs) 2025-09-07T07:12:45.5824208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5824298Z outputs = self.mobilebert( 2025-09-07T07:12:45.5824609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5824691Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5825016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5825096Z layer_outputs = layer_module( 2025-09-07T07:12:45.5825420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5825524Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5825925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5826070Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5826415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5826558Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5826866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5826975Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5826979Z 2025-09-07T07:12:45.5827094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5827314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5827394Z return mod(**inputs) 2025-09-07T07:12:45.5827717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5827805Z outputs = self.mobilebert( 2025-09-07T07:12:45.5828118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5828207Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5828517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5828594Z layer_outputs = layer_module( 2025-09-07T07:12:45.5828912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5829033Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5829350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5829473Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5829780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5829879Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5829883Z 2025-09-07T07:12:45.5829996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5830223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5830321Z return mod(**inputs) 2025-09-07T07:12:45.5830640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5830719Z outputs = self.mobilebert( 2025-09-07T07:12:45.5831027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5831117Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5831426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5831511Z layer_outputs = layer_module( 2025-09-07T07:12:45.5831821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5831924Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5832247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5832370Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5832689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5832829Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5832833Z 2025-09-07T07:12:45.5832952Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5833188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5833261Z return mod(**inputs) 2025-09-07T07:12:45.5833589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5833667Z outputs = self.mobilebert( 2025-09-07T07:12:45.5833996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5834078Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5834376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5834456Z layer_outputs = layer_module( 2025-09-07T07:12:45.5834742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5834845Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5835126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5835259Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5835553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5835648Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5835669Z 2025-09-07T07:12:45.5835787Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5836000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5836077Z return mod(**inputs) 2025-09-07T07:12:45.5836384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5836461Z outputs = self.mobilebert( 2025-09-07T07:12:45.5836769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5836845Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5837154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5837249Z layer_outputs = layer_module( 2025-09-07T07:12:45.5837554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5837657Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5837958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5838103Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5838388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5838518Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5838799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5838893Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5838903Z 2025-09-07T07:12:45.5839008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5839210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5839283Z return mod(**inputs) 2025-09-07T07:12:45.5839584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5839663Z outputs = self.mobilebert( 2025-09-07T07:12:45.5839962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5840036Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5840328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5840401Z layer_outputs = layer_module( 2025-09-07T07:12:45.5840693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5840820Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5841110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5841203Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5841207Z 2025-09-07T07:12:45.5841311Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5841520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5841587Z return mod(**inputs) 2025-09-07T07:12:45.5841882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5841956Z outputs = self.mobilebert( 2025-09-07T07:12:45.5842240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5842339Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5842631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5842711Z layer_outputs = layer_module( 2025-09-07T07:12:45.5843001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5843121Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5843417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5843532Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5843550Z 2025-09-07T07:12:45.5843661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5843859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5843935Z return mod(**inputs) 2025-09-07T07:12:45.5844224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5844295Z outputs = self.mobilebert( 2025-09-07T07:12:45.5844588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5844659Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5844945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5845015Z layer_outputs = layer_module( 2025-09-07T07:12:45.5845299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5845467Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5845751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5845853Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5845872Z 2025-09-07T07:12:45.5845977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5846196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5846262Z return mod(**inputs) 2025-09-07T07:12:45.5846561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5846644Z outputs = self.mobilebert( 2025-09-07T07:12:45.5846945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5847031Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5847331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5847404Z layer_outputs = layer_module( 2025-09-07T07:12:45.5847725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5847887Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5848176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5848301Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5848610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5848708Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5848732Z 2025-09-07T07:12:45.5848842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5849064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5849137Z return mod(**inputs) 2025-09-07T07:12:45.5849457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5849534Z outputs = self.mobilebert( 2025-09-07T07:12:45.5849841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5849927Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5850234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5850341Z layer_outputs = layer_module( 2025-09-07T07:12:45.5850659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5850839Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5851155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5851278Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5851561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5851646Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5851649Z 2025-09-07T07:12:45.5851756Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5851950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5852024Z return mod(**inputs) 2025-09-07T07:12:45.5852301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5852374Z outputs = self.mobilebert( 2025-09-07T07:12:45.5852680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5852770Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5853062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5853132Z layer_outputs = layer_module( 2025-09-07T07:12:45.5853414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5853581Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5853861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5853992Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5854277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5854409Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5854695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5854788Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5854791Z 2025-09-07T07:12:45.5854901Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5855101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5855174Z return mod(**inputs) 2025-09-07T07:12:45.5855485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5855556Z outputs = self.mobilebert( 2025-09-07T07:12:45.5855847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5855921Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5856212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5856283Z layer_outputs = layer_module( 2025-09-07T07:12:45.5856573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5856751Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5857033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5857155Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5857441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5857532Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5857536Z 2025-09-07T07:12:45.5857641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5857841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5857913Z return mod(**inputs) 2025-09-07T07:12:45.5858199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5858279Z outputs = self.mobilebert( 2025-09-07T07:12:45.5858569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5858649Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5858939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5859009Z layer_outputs = layer_module( 2025-09-07T07:12:45.5859309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5859468Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5859750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5859859Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5860135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5860229Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5860512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5860613Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5860616Z 2025-09-07T07:12:45.5860720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5860935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5861005Z return mod(**inputs) 2025-09-07T07:12:45.5861310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5861395Z outputs = self.mobilebert( 2025-09-07T07:12:45.5861695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5861800Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5862102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5862178Z layer_outputs = layer_module( 2025-09-07T07:12:45.5862490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5862585Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5862892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5862969Z self_outputs = self.self( 2025-09-07T07:12:45.5863276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5863372Z self.query(query_tensor) 2025-09-07T07:12:45.5863377Z 2025-09-07T07:12:45.5863487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5863706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5863775Z return mod(**inputs) 2025-09-07T07:12:45.5864088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5864165Z outputs = self.mobilebert( 2025-09-07T07:12:45.5864464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5864550Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5864850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5864935Z layer_outputs = layer_module( 2025-09-07T07:12:45.5865234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5865334Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5865652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5865796Z self_outputs = self.self( 2025-09-07T07:12:45.5866136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5866210Z self.key(key_tensor) 2025-09-07T07:12:45.5866214Z 2025-09-07T07:12:45.5866337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5866551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5866625Z return mod(**inputs) 2025-09-07T07:12:45.5866957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5867037Z outputs = self.mobilebert( 2025-09-07T07:12:45.5867362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5867446Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5867760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5867844Z layer_outputs = layer_module( 2025-09-07T07:12:45.5868145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5868244Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5868550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5868659Z self_outputs = self.self( 2025-09-07T07:12:45.5868956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5869031Z self.value(value_tensor) 2025-09-07T07:12:45.5869034Z 2025-09-07T07:12:45.5869134Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5869219Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5869338Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5869549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5869619Z return mod(**inputs) 2025-09-07T07:12:45.5869929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5870021Z outputs = self.mobilebert( 2025-09-07T07:12:45.5870337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5870415Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5870720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5870804Z layer_outputs = layer_module( 2025-09-07T07:12:45.5871103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5871203Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5871502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5871642Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5871942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5872034Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5872038Z 2025-09-07T07:12:45.5872156Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5872373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5872458Z return mod(**inputs) 2025-09-07T07:12:45.5872750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5872822Z outputs = self.mobilebert( 2025-09-07T07:12:45.5873105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5873177Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5873470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5873540Z layer_outputs = layer_module( 2025-09-07T07:12:45.5873828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5873987Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5874269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5874389Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5874665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5874753Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5874756Z 2025-09-07T07:12:45.5874857Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5875063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5875149Z return mod(**inputs) 2025-09-07T07:12:45.5875443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5875523Z outputs = self.mobilebert( 2025-09-07T07:12:45.5875820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5875900Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5876191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5876262Z layer_outputs = layer_module( 2025-09-07T07:12:45.5876558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5876663Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5876953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5877078Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5877362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5877506Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5877781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5877878Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5877881Z 2025-09-07T07:12:45.5877981Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5878180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5878243Z return mod(**inputs) 2025-09-07T07:12:45.5878523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5878603Z outputs = self.mobilebert( 2025-09-07T07:12:45.5878901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5878981Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5879291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5879361Z layer_outputs = layer_module( 2025-09-07T07:12:45.5879641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5879736Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5880017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5880129Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5880411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5880494Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5880497Z 2025-09-07T07:12:45.5880597Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5880799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5880864Z return mod(**inputs) 2025-09-07T07:12:45.5881145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5881216Z outputs = self.mobilebert( 2025-09-07T07:12:45.5881488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5881581Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5881857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5881938Z layer_outputs = layer_module( 2025-09-07T07:12:45.5882229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5882328Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5882600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5882707Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5883004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5883116Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5883119Z 2025-09-07T07:12:45.5883227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5883431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5883501Z return mod(**inputs) 2025-09-07T07:12:45.5883774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5883843Z outputs = self.mobilebert( 2025-09-07T07:12:45.5884118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5884187Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5884468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5884536Z layer_outputs = layer_module( 2025-09-07T07:12:45.5884806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5884905Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5885194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5885349Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5885629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5885713Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5885725Z 2025-09-07T07:12:45.5885825Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5886020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5886096Z return mod(**inputs) 2025-09-07T07:12:45.5886378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5886453Z outputs = self.mobilebert( 2025-09-07T07:12:45.5886734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5886807Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5887097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5887168Z layer_outputs = layer_module( 2025-09-07T07:12:45.5887457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5887552Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5887843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5887995Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5888278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5888411Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5888695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5888796Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5888799Z 2025-09-07T07:12:45.5888902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5889119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5889195Z return mod(**inputs) 2025-09-07T07:12:45.5889482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5889560Z outputs = self.mobilebert( 2025-09-07T07:12:45.5889852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5889927Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5890217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5890289Z layer_outputs = layer_module( 2025-09-07T07:12:45.5890579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5890677Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5890967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5891082Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5891368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5891474Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5891478Z 2025-09-07T07:12:45.5891597Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5891805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5891872Z return mod(**inputs) 2025-09-07T07:12:45.5892167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5892240Z outputs = self.mobilebert( 2025-09-07T07:12:45.5892527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5892610Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5892893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5892970Z layer_outputs = layer_module( 2025-09-07T07:12:45.5893256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5893350Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5893644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5893754Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5894046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5894174Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5894177Z 2025-09-07T07:12:45.5894285Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5894488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5894556Z return mod(**inputs) 2025-09-07T07:12:45.5894862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5894932Z outputs = self.mobilebert( 2025-09-07T07:12:45.5895232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5895304Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5895600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5895680Z layer_outputs = layer_module( 2025-09-07T07:12:45.5895963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5896065Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5896349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5896475Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5896767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5896852Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5896855Z 2025-09-07T07:12:45.5896966Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5897165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5897241Z return mod(**inputs) 2025-09-07T07:12:45.5897525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5897596Z outputs = self.mobilebert( 2025-09-07T07:12:45.5897902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5897991Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5898281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5898352Z layer_outputs = layer_module( 2025-09-07T07:12:45.5898631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5898735Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5899016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5899149Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5899433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5899565Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5899847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5899939Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5899943Z 2025-09-07T07:12:45.5900053Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5900252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5900328Z return mod(**inputs) 2025-09-07T07:12:45.5900629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5900708Z outputs = self.mobilebert( 2025-09-07T07:12:45.5900992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5901065Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5901356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5901427Z layer_outputs = layer_module( 2025-09-07T07:12:45.5901712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5901826Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5902107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5902230Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5902514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5902606Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5902609Z 2025-09-07T07:12:45.5902713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5902920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5902985Z return mod(**inputs) 2025-09-07T07:12:45.5903266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5903347Z outputs = self.mobilebert( 2025-09-07T07:12:45.5903627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5903710Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5903994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5904089Z layer_outputs = layer_module( 2025-09-07T07:12:45.5904427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5904521Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5904813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5904925Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5905221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5905333Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5905338Z 2025-09-07T07:12:45.5905439Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5905648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5905786Z return mod(**inputs) 2025-09-07T07:12:45.5906111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5906189Z outputs = self.mobilebert( 2025-09-07T07:12:45.5906488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5906575Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5906873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5906962Z layer_outputs = layer_module( 2025-09-07T07:12:45.5907279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5907375Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5907668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5907797Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5908106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5908195Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5908199Z 2025-09-07T07:12:45.5908317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5908549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5908622Z return mod(**inputs) 2025-09-07T07:12:45.5908934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5909009Z outputs = self.mobilebert( 2025-09-07T07:12:45.5909319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5909396Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5909695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5909779Z layer_outputs = layer_module( 2025-09-07T07:12:45.5910078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5910186Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5910486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5910627Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5910944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5911073Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5911394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5911492Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5911496Z 2025-09-07T07:12:45.5911609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5911821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5911898Z return mod(**inputs) 2025-09-07T07:12:45.5912198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5912274Z outputs = self.mobilebert( 2025-09-07T07:12:45.5912587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5912662Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5912970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5913045Z layer_outputs = layer_module( 2025-09-07T07:12:45.5913345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5913483Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5913782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5913898Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5913903Z 2025-09-07T07:12:45.5914010Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5914228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5914297Z return mod(**inputs) 2025-09-07T07:12:45.5914604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5914686Z outputs = self.mobilebert( 2025-09-07T07:12:45.5914989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5915091Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5915398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5915474Z layer_outputs = layer_module( 2025-09-07T07:12:45.5915785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5915914Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5916228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5916346Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5916350Z 2025-09-07T07:12:45.5916467Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5916682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5916755Z return mod(**inputs) 2025-09-07T07:12:45.5917076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5917154Z outputs = self.mobilebert( 2025-09-07T07:12:45.5917471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5917548Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5917872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5917972Z layer_outputs = layer_module( 2025-09-07T07:12:45.5918272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5918449Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5918754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5918863Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5918869Z 2025-09-07T07:12:45.5918977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5919189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5919270Z return mod(**inputs) 2025-09-07T07:12:45.5919754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5919850Z outputs = self.mobilebert( 2025-09-07T07:12:45.5920151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5920230Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5920549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5920627Z layer_outputs = layer_module( 2025-09-07T07:12:45.5920985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5921154Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5921515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5921646Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5921955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5922059Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5922064Z 2025-09-07T07:12:45.5922202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5922421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5922494Z return mod(**inputs) 2025-09-07T07:12:45.5922809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5922895Z outputs = self.mobilebert( 2025-09-07T07:12:45.5923208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5923295Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5923610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5923694Z layer_outputs = layer_module( 2025-09-07T07:12:45.5924004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5924173Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5924471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5924589Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5924890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5924996Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5924999Z 2025-09-07T07:12:45.5925102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5925301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5925366Z return mod(**inputs) 2025-09-07T07:12:45.5925659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5925731Z outputs = self.mobilebert( 2025-09-07T07:12:45.5926024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5926096Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5926382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5926462Z layer_outputs = layer_module( 2025-09-07T07:12:45.5926750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5926924Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5927219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5927339Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5927623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5927759Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5928042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5928134Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5928138Z 2025-09-07T07:12:45.5928248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5928441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5928505Z return mod(**inputs) 2025-09-07T07:12:45.5928793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5928890Z outputs = self.mobilebert( 2025-09-07T07:12:45.5929171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5929244Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5929522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5929598Z layer_outputs = layer_module( 2025-09-07T07:12:45.5929879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5930046Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5930323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5930439Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5930714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5930798Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5930801Z 2025-09-07T07:12:45.5930910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5931119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5931189Z return mod(**inputs) 2025-09-07T07:12:45.5931489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5931567Z outputs = self.mobilebert( 2025-09-07T07:12:45.5931844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5931917Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5932206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5932276Z layer_outputs = layer_module( 2025-09-07T07:12:45.5932562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5932723Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5933001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-09-07T07:12:45.5933118Z shared_attention_input = self.attention(hidden_states) 2025-09-07T07:12:45.5933395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-09-07T07:12:45.5933488Z layer_input = self.LayerNorm(layer_input) 2025-09-07T07:12:45.5933768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5933881Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5933884Z 2025-09-07T07:12:45.5933985Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5934180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5934253Z return mod(**inputs) 2025-09-07T07:12:45.5934535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5934611Z outputs = self.mobilebert( 2025-09-07T07:12:45.5934896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5934965Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5935258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5935330Z layer_outputs = layer_module( 2025-09-07T07:12:45.5935613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5935697Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5935982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5936053Z self_outputs = self.self( 2025-09-07T07:12:45.5936325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-09-07T07:12:45.5936402Z self.query(query_tensor) 2025-09-07T07:12:45.5936405Z 2025-09-07T07:12:45.5936505Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5936721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5936790Z return mod(**inputs) 2025-09-07T07:12:45.5937073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5937162Z outputs = self.mobilebert( 2025-09-07T07:12:45.5937454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5937534Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5937823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5937894Z layer_outputs = layer_module( 2025-09-07T07:12:45.5938178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5938264Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5938560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5938629Z self_outputs = self.self( 2025-09-07T07:12:45.5938913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-09-07T07:12:45.5938977Z self.key(key_tensor) 2025-09-07T07:12:45.5938980Z 2025-09-07T07:12:45.5939077Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5939275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5939335Z return mod(**inputs) 2025-09-07T07:12:45.5939619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5939689Z outputs = self.mobilebert( 2025-09-07T07:12:45.5939967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5940060Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5940334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5940411Z layer_outputs = layer_module( 2025-09-07T07:12:45.5940689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5940780Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5941061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-09-07T07:12:45.5941131Z self_outputs = self.self( 2025-09-07T07:12:45.5941421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-09-07T07:12:45.5941517Z self.value(value_tensor) 2025-09-07T07:12:45.5941522Z 2025-09-07T07:12:45.5941614Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5941695Z cudagraph partition due to non gpu ops 2025-09-07T07:12:45.5941799Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5942009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5942075Z return mod(**inputs) 2025-09-07T07:12:45.5942374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5942445Z outputs = self.mobilebert( 2025-09-07T07:12:45.5942732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5942811Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5943099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5943180Z layer_outputs = layer_module( 2025-09-07T07:12:45.5943474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5943570Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5943890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5944036Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5944342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-09-07T07:12:45.5944432Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5944438Z 2025-09-07T07:12:45.5944552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5944761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5944832Z return mod(**inputs) 2025-09-07T07:12:45.5945143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5945217Z outputs = self.mobilebert( 2025-09-07T07:12:45.5945537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5945617Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5945997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5946076Z layer_outputs = layer_module( 2025-09-07T07:12:45.5946377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-09-07T07:12:45.5946566Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-09-07T07:12:45.5946897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-09-07T07:12:45.5947027Z bottlenecked_hidden_states = self.input(hidden_states) 2025-09-07T07:12:45.5947329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-09-07T07:12:45.5947415Z layer_input = self.dense(hidden_states) 2025-09-07T07:12:45.5947426Z 2025-09-07T07:12:45.5947531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5947728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5947804Z return mod(**inputs) 2025-09-07T07:12:45.5948089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5948190Z outputs = self.mobilebert( 2025-09-07T07:12:45.5948475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5948551Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5948846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5948919Z layer_outputs = layer_module( 2025-09-07T07:12:45.5949214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-09-07T07:12:45.5949301Z self_attention_outputs = self.attention( 2025-09-07T07:12:45.5949586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-09-07T07:12:45.5949719Z attention_output = self.output(self_outputs[0], layer_input) 2025-09-07T07:12:45.5950005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-09-07T07:12:45.5950144Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5950445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5950546Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5950550Z 2025-09-07T07:12:45.5950665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5950866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5950941Z return mod(**inputs) 2025-09-07T07:12:45.5951228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5951310Z outputs = self.mobilebert( 2025-09-07T07:12:45.5951592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5951666Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5951958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5952029Z layer_outputs = layer_module( 2025-09-07T07:12:45.5952318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5952415Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5952703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5952820Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5953105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5953223Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5953227Z 2025-09-07T07:12:45.5953338Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5953534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5953596Z return mod(**inputs) 2025-09-07T07:12:45.5953868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5953944Z outputs = self.mobilebert( 2025-09-07T07:12:45.5954212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5954289Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5954569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5954645Z layer_outputs = layer_module( 2025-09-07T07:12:45.5954910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5955000Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5955276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5955384Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5955657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5955768Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5955773Z 2025-09-07T07:12:45.5955874Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5956074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5956140Z return mod(**inputs) 2025-09-07T07:12:45.5956422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5956491Z outputs = self.mobilebert( 2025-09-07T07:12:45.5956792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5956878Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5957162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5957239Z layer_outputs = layer_module( 2025-09-07T07:12:45.5957515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5957616Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5957891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5958012Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5958294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5958378Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5958381Z 2025-09-07T07:12:45.5958485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5958674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5958744Z return mod(**inputs) 2025-09-07T07:12:45.5959014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5959083Z outputs = self.mobilebert( 2025-09-07T07:12:45.5959371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5959439Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5959712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5959779Z layer_outputs = layer_module( 2025-09-07T07:12:45.5960047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5960144Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5960409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5960553Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5960821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5960946Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5961220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5961309Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5961312Z 2025-09-07T07:12:45.5961418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5961605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5961675Z return mod(**inputs) 2025-09-07T07:12:45.5961951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5962021Z outputs = self.mobilebert( 2025-09-07T07:12:45.5962299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5962370Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5962663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5962733Z layer_outputs = layer_module( 2025-09-07T07:12:45.5963046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5963138Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5963405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5963520Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5963792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5963882Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5963886Z 2025-09-07T07:12:45.5963985Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5964176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5964249Z return mod(**inputs) 2025-09-07T07:12:45.5964527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5964601Z outputs = self.mobilebert( 2025-09-07T07:12:45.5964871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5964949Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5965221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5965306Z layer_outputs = layer_module( 2025-09-07T07:12:45.5965583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5965674Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5965975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5966086Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5966365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5966481Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5966500Z 2025-09-07T07:12:45.5966600Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5966803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5966871Z return mod(**inputs) 2025-09-07T07:12:45.5967216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5967284Z outputs = self.mobilebert( 2025-09-07T07:12:45.5967557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5967637Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5967912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5967987Z layer_outputs = layer_module( 2025-09-07T07:12:45.5968258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5968350Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5968628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5968749Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5969043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5969142Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5969145Z 2025-09-07T07:12:45.5969254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5969450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5969514Z return mod(**inputs) 2025-09-07T07:12:45.5969802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5969873Z outputs = self.mobilebert( 2025-09-07T07:12:45.5970161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5970233Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5970516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5970593Z layer_outputs = layer_module( 2025-09-07T07:12:45.5970873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5970972Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5971249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5971381Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5971661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5971817Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5972103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5972194Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5972197Z 2025-09-07T07:12:45.5972305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5972497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5972561Z return mod(**inputs) 2025-09-07T07:12:45.5972848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5972937Z outputs = self.mobilebert( 2025-09-07T07:12:45.5973218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5973289Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5973572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5973640Z layer_outputs = layer_module( 2025-09-07T07:12:45.5973917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5974016Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5974292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5974409Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5974687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5974773Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5974783Z 2025-09-07T07:12:45.5974885Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5975097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5975171Z return mod(**inputs) 2025-09-07T07:12:45.5975480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5975557Z outputs = self.mobilebert( 2025-09-07T07:12:45.5975833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5975903Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5976187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5976258Z layer_outputs = layer_module( 2025-09-07T07:12:45.5976538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5976630Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5976908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-09-07T07:12:45.5977026Z intermediate_output = self.intermediate(hidden_states) 2025-09-07T07:12:45.5977300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5977416Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5977421Z 2025-09-07T07:12:45.5977522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5977723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5977802Z return mod(**inputs) 2025-09-07T07:12:45.5978081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5978158Z outputs = self.mobilebert( 2025-09-07T07:12:45.5978434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5978515Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5978791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5978860Z layer_outputs = layer_module( 2025-09-07T07:12:45.5979145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5979252Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5979539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5979661Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5979951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-09-07T07:12:45.5980037Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5980040Z 2025-09-07T07:12:45.5980141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5980341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5980405Z return mod(**inputs) 2025-09-07T07:12:45.5980695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5980766Z outputs = self.mobilebert( 2025-09-07T07:12:45.5981043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5981123Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5981416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5981495Z layer_outputs = layer_module( 2025-09-07T07:12:45.5981787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-09-07T07:12:45.5981886Z attention_output = ffn_module(attention_output) 2025-09-07T07:12:45.5982159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-09-07T07:12:45.5982283Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-09-07T07:12:45.5982567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-09-07T07:12:45.5982690Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5982974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5983065Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5983068Z 2025-09-07T07:12:45.5983178Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5983371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5983434Z return mod(**inputs) 2025-09-07T07:12:45.5983720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5983791Z outputs = self.mobilebert( 2025-09-07T07:12:45.5984072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5984158Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5984434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5984514Z layer_outputs = layer_module( 2025-09-07T07:12:45.5984794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5984920Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5985204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-09-07T07:12:45.5985289Z hidden_states = self.dense(hidden_states) 2025-09-07T07:12:45.5985318Z 2025-09-07T07:12:45.5985424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5985623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5985770Z return mod(**inputs) 2025-09-07T07:12:45.5986064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5986148Z outputs = self.mobilebert( 2025-09-07T07:12:45.5986451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5986527Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5986834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5986909Z layer_outputs = layer_module( 2025-09-07T07:12:45.5987218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-09-07T07:12:45.5987346Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:12:45.5987657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-09-07T07:12:45.5987784Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:12:45.5987810Z 2025-09-07T07:12:45.5987923Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5988161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5988234Z return mod(**inputs) 2025-09-07T07:12:45.5988553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5988630Z outputs = self.mobilebert( 2025-09-07T07:12:45.5988940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5989029Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5989336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5989421Z layer_outputs = layer_module( 2025-09-07T07:12:45.5989731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5989902Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5990219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-09-07T07:12:45.5990321Z layer_output = self.dense(intermediate_states) 2025-09-07T07:12:45.5990325Z 2025-09-07T07:12:45.5990446Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5990661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5990753Z return mod(**inputs) 2025-09-07T07:12:45.5991060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5991135Z outputs = self.mobilebert( 2025-09-07T07:12:45.5991446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5991525Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5991836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5991911Z layer_outputs = layer_module( 2025-09-07T07:12:45.5992211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5992403Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5992702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-09-07T07:12:45.5992840Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-09-07T07:12:45.5993143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5993249Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5993253Z 2025-09-07T07:12:45.5993362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5993569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5993645Z return mod(**inputs) 2025-09-07T07:12:45.5993946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5994030Z outputs = self.mobilebert( 2025-09-07T07:12:45.5994332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5994410Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5994735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5994812Z layer_outputs = layer_module( 2025-09-07T07:12:45.5995134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5995311Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5995604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5995735Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5996024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-09-07T07:12:45.5996122Z layer_outputs = self.dense(hidden_states) 2025-09-07T07:12:45.5996125Z 2025-09-07T07:12:45.5996233Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5996442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5996512Z return mod(**inputs) 2025-09-07T07:12:45.5996810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-09-07T07:12:45.5996885Z outputs = self.mobilebert( 2025-09-07T07:12:45.5997175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-09-07T07:12:45.5997260Z encoder_outputs = self.encoder( 2025-09-07T07:12:45.5997545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-09-07T07:12:45.5997641Z layer_outputs = layer_module( 2025-09-07T07:12:45.5997935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-09-07T07:12:45.5998091Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-09-07T07:12:45.5998382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-09-07T07:12:45.5998502Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-09-07T07:12:45.5998783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-09-07T07:12:45.5998920Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-09-07T07:12:45.5999205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-09-07T07:12:45.5999300Z return input_tensor * self.weight + self.bias 2025-09-07T07:12:45.5999304Z 2025-09-07T07:12:45.5999406Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.5999613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.5999680Z return mod(**inputs) 2025-09-07T07:12:45.5999974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1256, in forward 2025-09-07T07:12:45.6000062Z logits = self.qa_outputs(sequence_output) 2025-09-07T07:12:45.6000066Z 2025-09-07T07:12:45.6000170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.6000379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.6000446Z return mod(**inputs) 2025-09-07T07:12:45.6000742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1274, in forward 2025-09-07T07:12:45.6000855Z start_loss = loss_fct(start_logits, start_positions) 2025-09-07T07:12:45.6000859Z 2025-09-07T07:12:45.6001004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:12:45.6001231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:12:45.6001301Z return mod(**inputs) 2025-09-07T07:12:45.6001615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1275, in forward 2025-09-07T07:12:45.6001718Z end_loss = loss_fct(end_logits, end_positions) 2025-09-07T07:12:45.6001722Z 2025-09-07T07:13:00.0720099Z Compilation time (from dynamo_timed): 40.779609122 2025-09-07T07:13:00.0720431Z pass 2025-09-07T07:13:00.0720764Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:00.0721686Z TIMING: _recursive_pre_grad_passes:0.0238 _recursive_joint_graph_passes:1.36629 _recursive_post_grad_passes:0.22069 async_compile.wait:0.29025 code_gen:11.27991 inductor_compile:16.09046 backend_compile:28.82049 gc:0.00126 entire_frame_compile:40.77961 total_wall_time:40.77961 2025-09-07T07:13:00.0722705Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:56755 | FakeTensor.__torch_dispatch__:15375 | ProxyTorchDispatchMode.__torch_dispatch__:21655 2025-09-07T07:13:00.0723277Z Dynamo produced 1 graphs covering 1453 ops with 0 graph breaks (0 unique) 2025-09-07T07:13:03.6167731Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:13:03.6170873Z import pynvml # type: ignore[import] 2025-09-07T07:13:06.3756047Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:13:06.3756973Z from pkg_resources import resource_filename 2025-09-07T07:13:07.0412345Z 2025-09-07T07:13:08.8110555Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:13:08.8110928Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:13:08.8131807Z cpu eval OPTForCausalLM 2025-09-07T07:13:10.4835426Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:11.2265359Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:12.1967561Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:19.8067504Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8067980Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8068820Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8069157Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8069409Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8069619Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8069849Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8070065Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8070279Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8070492Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8070697Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8070903Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8071159Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8071620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8072006Z return mod(**inputs) 2025-09-07T07:13:19.8072399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8072807Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8073560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8074051Z outputs = self.model.decoder( 2025-09-07T07:13:19.8074422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8074796Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8075212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8075643Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8076025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8076447Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8076880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8077341Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8077802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8078250Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8078446Z 2025-09-07T07:13:19.8078566Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8078984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8079345Z return mod(**inputs) 2025-09-07T07:13:19.8079712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8080166Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8080583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8081010Z outputs = self.model.decoder( 2025-09-07T07:13:19.8081401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8081791Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8082216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8082627Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8083014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8083459Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8083872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8084316Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8084751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8085173Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8085325Z 2025-09-07T07:13:19.8085443Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8085843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8086211Z return mod(**inputs) 2025-09-07T07:13:19.8086569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8086955Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8087356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8087903Z outputs = self.model.decoder( 2025-09-07T07:13:19.8088284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8088673Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8089101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8089519Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8089923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8090318Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8090725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8091158Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8091590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8092021Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8092180Z 2025-09-07T07:13:19.8092284Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8092527Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8092759Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8092989Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8093253Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8093653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8094010Z return mod(**inputs) 2025-09-07T07:13:19.8094371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8094760Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8095168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8095609Z outputs = self.model.decoder( 2025-09-07T07:13:19.8095984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8096367Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8096778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8097187Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8097559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8097952Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8098357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8098820Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8099287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8099731Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8100231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8100793Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8101004Z 2025-09-07T07:13:19.8101130Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8101556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8101938Z return mod(**inputs) 2025-09-07T07:13:19.8102322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8103035Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8103458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8103893Z outputs = self.model.decoder( 2025-09-07T07:13:19.8104282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8104708Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8105151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8105565Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8106234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8106659Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8107087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8107549Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8108007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8108460Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8108976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8109501Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8109684Z 2025-09-07T07:13:19.8109803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8110204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8110566Z return mod(**inputs) 2025-09-07T07:13:19.8110932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8111331Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8111784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8112218Z outputs = self.model.decoder( 2025-09-07T07:13:19.8112606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8113003Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8113417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8113840Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8114228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8114635Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8115109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8115549Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8116014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8116445Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8116594Z 2025-09-07T07:13:19.8116715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8117111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8117463Z return mod(**inputs) 2025-09-07T07:13:19.8117824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8118209Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8118614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8119021Z outputs = self.model.decoder( 2025-09-07T07:13:19.8119391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8120039Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8120509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8120921Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8121329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8121725Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8122136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8122557Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8122713Z 2025-09-07T07:13:19.8122837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8123226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8123578Z return mod(**inputs) 2025-09-07T07:13:19.8123929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8124311Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8124713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8125118Z outputs = self.model.decoder( 2025-09-07T07:13:19.8125491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8125874Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8126276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8126676Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8127054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8127474Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8127886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8128326Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8128499Z 2025-09-07T07:13:19.8128612Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8129001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8129355Z return mod(**inputs) 2025-09-07T07:13:19.8129707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8130093Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8130519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8130932Z outputs = self.model.decoder( 2025-09-07T07:13:19.8131316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8131691Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8132078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8132472Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8132838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8133219Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8133601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8134003Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8134154Z 2025-09-07T07:13:19.8134263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8134645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8134982Z return mod(**inputs) 2025-09-07T07:13:19.8135329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8135704Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8136105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8136494Z outputs = self.model.decoder( 2025-09-07T07:13:19.8136839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8137201Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8137591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8137976Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8138335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8138701Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8139089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8139526Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8139964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8140419Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8140600Z 2025-09-07T07:13:19.8140712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8141105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8141457Z return mod(**inputs) 2025-09-07T07:13:19.8141826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8142204Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8142615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8143022Z outputs = self.model.decoder( 2025-09-07T07:13:19.8143397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8143776Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8144260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8144681Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8145086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8145481Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8145976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8146471Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8146927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8147368Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8147519Z 2025-09-07T07:13:19.8147641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8148028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8148389Z return mod(**inputs) 2025-09-07T07:13:19.8148755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8149142Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8149551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8149966Z outputs = self.model.decoder( 2025-09-07T07:13:19.8150365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8150753Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8151188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8151598Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8151975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8152367Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8152777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8153211Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8153635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8154068Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8154226Z 2025-09-07T07:13:19.8154315Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8154550Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8154769Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8154992Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8155249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8155650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8156024Z return mod(**inputs) 2025-09-07T07:13:19.8156382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8156751Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8157160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8157550Z outputs = self.model.decoder( 2025-09-07T07:13:19.8157897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8158262Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8158649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8159036Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8159394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8159759Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8160165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8160604Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8161037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8161460Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8161948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8162477Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8162690Z 2025-09-07T07:13:19.8162807Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8163230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8163581Z return mod(**inputs) 2025-09-07T07:13:19.8163943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8164329Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8164719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8165102Z outputs = self.model.decoder( 2025-09-07T07:13:19.8165478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8165859Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8166250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8166641Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8166996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8167376Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8167768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8168183Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8168592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8169001Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8169461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8169939Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8170138Z 2025-09-07T07:13:19.8170262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8170664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8170999Z return mod(**inputs) 2025-09-07T07:13:19.8171337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8171735Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8172118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8172500Z outputs = self.model.decoder( 2025-09-07T07:13:19.8172856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8173219Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8173602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8173985Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8174336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8174724Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8175110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8175523Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8175934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8176346Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8176499Z 2025-09-07T07:13:19.8176613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8177016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8177375Z return mod(**inputs) 2025-09-07T07:13:19.8177730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8178128Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8178571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8178986Z outputs = self.model.decoder( 2025-09-07T07:13:19.8179367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8179754Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8180191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8180625Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8181004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8181395Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8181812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8182234Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8182384Z 2025-09-07T07:13:19.8182506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8182971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8183328Z return mod(**inputs) 2025-09-07T07:13:19.8183694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8184084Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8184500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8184923Z outputs = self.model.decoder( 2025-09-07T07:13:19.8185305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8185784Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8186245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8186771Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8187142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8187541Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8187952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8188405Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8188573Z 2025-09-07T07:13:19.8188693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8189095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8189455Z return mod(**inputs) 2025-09-07T07:13:19.8189866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8190282Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8190700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8191116Z outputs = self.model.decoder( 2025-09-07T07:13:19.8191498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8191880Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8192286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8192696Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8193072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8193465Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8193877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8194304Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8194451Z 2025-09-07T07:13:19.8194565Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8194954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8195340Z return mod(**inputs) 2025-09-07T07:13:19.8195703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8196103Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8196491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8196880Z outputs = self.model.decoder( 2025-09-07T07:13:19.8197237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8197633Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8198033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8198447Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8198822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8199214Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8199624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8200063Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8200495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8200946Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8201127Z 2025-09-07T07:13:19.8201248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8201633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8202013Z return mod(**inputs) 2025-09-07T07:13:19.8202371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8202741Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8203127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8203515Z outputs = self.model.decoder( 2025-09-07T07:13:19.8203894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8204286Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8204691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8205108Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8205485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8205881Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8206292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8206723Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8207145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8207559Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8207710Z 2025-09-07T07:13:19.8207825Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8208214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8208568Z return mod(**inputs) 2025-09-07T07:13:19.8208912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8209297Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8209702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8210128Z outputs = self.model.decoder( 2025-09-07T07:13:19.8210500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8210910Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8211311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8211702Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8212079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8212466Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8212875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8213309Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8213738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8214153Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8214313Z 2025-09-07T07:13:19.8214400Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8214633Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8214861Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8215078Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8215326Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8215720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8216078Z return mod(**inputs) 2025-09-07T07:13:19.8216431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8216834Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8217249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8217667Z outputs = self.model.decoder( 2025-09-07T07:13:19.8218052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8218437Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8218843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8219260Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8219836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8220299Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8220708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8221147Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8221584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8222026Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8222521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8223052Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8223263Z 2025-09-07T07:13:19.8223377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8223777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8224134Z return mod(**inputs) 2025-09-07T07:13:19.8224493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8224873Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8226362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8226832Z outputs = self.model.decoder( 2025-09-07T07:13:19.8227290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8227668Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8228090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8228510Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8228903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8229313Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8229732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8230183Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8230632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8231079Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8231580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8232092Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8232282Z 2025-09-07T07:13:19.8232396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8232800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8233234Z return mod(**inputs) 2025-09-07T07:13:19.8233592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8233973Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8234346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8234721Z outputs = self.model.decoder( 2025-09-07T07:13:19.8235067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8235416Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8235790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8236183Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8236525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8236885Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8237251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8237649Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8238054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8238470Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8238619Z 2025-09-07T07:13:19.8238732Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8239121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8239470Z return mod(**inputs) 2025-09-07T07:13:19.8239825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8240194Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8240558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8240929Z outputs = self.model.decoder( 2025-09-07T07:13:19.8241289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8241647Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8242040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8242418Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8242778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8243140Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8243518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8243895Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8244037Z 2025-09-07T07:13:19.8244142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8244504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8244832Z return mod(**inputs) 2025-09-07T07:13:19.8245160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8245510Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8245885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8246261Z outputs = self.model.decoder( 2025-09-07T07:13:19.8246602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8246949Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8247345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8247715Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8248066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8248434Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8248818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8249243Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8249405Z 2025-09-07T07:13:19.8249512Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8249879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8250300Z return mod(**inputs) 2025-09-07T07:13:19.8250619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8279754Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8280424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8280888Z outputs = self.model.decoder( 2025-09-07T07:13:19.8281280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8281690Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8282094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8282475Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8282842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8283223Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8283620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8284019Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8284168Z 2025-09-07T07:13:19.8284292Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8284758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8285099Z return mod(**inputs) 2025-09-07T07:13:19.8285473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8285839Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8286224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8286612Z outputs = self.model.decoder( 2025-09-07T07:13:19.8286975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8287332Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8287701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8288080Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8288432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8288800Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8289176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-09-07T07:13:19.8289602Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-09-07T07:13:19.8289797Z 2025-09-07T07:13:19.8289905Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8290270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8290627Z return mod(**inputs) 2025-09-07T07:13:19.8290949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8291296Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8291666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8292041Z outputs = self.model.decoder( 2025-09-07T07:13:19.8292383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8292727Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8293109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8293519Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8293879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8294254Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8294635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8295053Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8295468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8295893Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8296065Z 2025-09-07T07:13:19.8296172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8296543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8296877Z return mod(**inputs) 2025-09-07T07:13:19.8297221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8297588Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8297967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8298354Z outputs = self.model.decoder( 2025-09-07T07:13:19.8298723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8299086Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8299480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8299869Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8300227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8300602Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8301009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8301442Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8301875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8302296Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8302443Z 2025-09-07T07:13:19.8302565Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8302957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8303299Z return mod(**inputs) 2025-09-07T07:13:19.8303655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8304047Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8304454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8304861Z outputs = self.model.decoder( 2025-09-07T07:13:19.8305254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8305634Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8306152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8306563Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8306935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8307323Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8307712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8308145Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8308578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8308990Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8309145Z 2025-09-07T07:13:19.8309230Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8309457Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8309674Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8309879Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8310120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8310489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8310823Z return mod(**inputs) 2025-09-07T07:13:19.8311150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8311511Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8311893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8312280Z outputs = self.model.decoder( 2025-09-07T07:13:19.8312630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8312982Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8313395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8313798Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8314158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8314528Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8314909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8315323Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8315732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8316149Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8316607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8317111Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8317313Z 2025-09-07T07:13:19.8317423Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8317799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8318137Z return mod(**inputs) 2025-09-07T07:13:19.8318473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8318874Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8319292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8319890Z outputs = self.model.decoder( 2025-09-07T07:13:19.8320271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8320650Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8321037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8321427Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8321793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8322184Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8322594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8323093Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8323543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8323960Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8324438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8324947Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8325138Z 2025-09-07T07:13:19.8325257Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8325661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8326045Z return mod(**inputs) 2025-09-07T07:13:19.8326403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8326798Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8327211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8327627Z outputs = self.model.decoder( 2025-09-07T07:13:19.8327998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8328432Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8328866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8329280Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8329666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8330058Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8330474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8330918Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8331364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8331794Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8331959Z 2025-09-07T07:13:19.8332080Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8332479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8332842Z return mod(**inputs) 2025-09-07T07:13:19.8333201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8333587Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8334003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8334420Z outputs = self.model.decoder( 2025-09-07T07:13:19.8334802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8335211Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8335604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8335987Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8336345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8336715Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8337093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8337488Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8337636Z 2025-09-07T07:13:19.8337762Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8338135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8338474Z return mod(**inputs) 2025-09-07T07:13:19.8338803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8339165Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8339551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8339964Z outputs = self.model.decoder( 2025-09-07T07:13:19.8340329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8340715Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8341115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8341522Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8341900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8342286Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8342693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8343154Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8343322Z 2025-09-07T07:13:19.8343441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8343849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8344194Z return mod(**inputs) 2025-09-07T07:13:19.8344547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8344933Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8345341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8345815Z outputs = self.model.decoder( 2025-09-07T07:13:19.8346206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8346598Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8347015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8347431Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8347780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8348155Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8348539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8348930Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8349069Z 2025-09-07T07:13:19.8349184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8349564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8349905Z return mod(**inputs) 2025-09-07T07:13:19.8350233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8350583Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8350966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8351344Z outputs = self.model.decoder( 2025-09-07T07:13:19.8351690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8352045Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8352416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8352856Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8353216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8353589Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8353972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8354391Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8354786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8355200Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8355364Z 2025-09-07T07:13:19.8355475Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8355822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8356147Z return mod(**inputs) 2025-09-07T07:13:19.8356469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8356824Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8357213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8357581Z outputs = self.model.decoder( 2025-09-07T07:13:19.8357936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8358286Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8358655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8359021Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8359373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8359734Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8360117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8360513Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8360904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8361286Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8361426Z 2025-09-07T07:13:19.8361528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8361888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8362207Z return mod(**inputs) 2025-09-07T07:13:19.8362532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8362887Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8363268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8363660Z outputs = self.model.decoder( 2025-09-07T07:13:19.8363996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8364345Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8364722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8365093Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8365440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8365791Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8366168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8366600Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8366995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8367374Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8367522Z 2025-09-07T07:13:19.8367606Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8367824Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8368036Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8368243Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8368468Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8368828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8369163Z return mod(**inputs) 2025-09-07T07:13:19.8369478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8369814Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8370178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8370544Z outputs = self.model.decoder( 2025-09-07T07:13:19.8370890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8371234Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8371602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8371967Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8372307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8372660Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8373019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8373410Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8373797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8374185Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8374617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8375082Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8375276Z 2025-09-07T07:13:19.8375375Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8375717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8376033Z return mod(**inputs) 2025-09-07T07:13:19.8376357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8376698Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8377079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8377450Z outputs = self.model.decoder( 2025-09-07T07:13:19.8377788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8378125Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8378504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8378878Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8379234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8379626Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8380019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8380445Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8380857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8381311Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8381793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8382304Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8382491Z 2025-09-07T07:13:19.8382609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8383019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8383396Z return mod(**inputs) 2025-09-07T07:13:19.8383751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8384125Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8384539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8384953Z outputs = self.model.decoder( 2025-09-07T07:13:19.8385351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8385825Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8386252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8386672Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8387063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8387454Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8387868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8388304Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8388734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8389158Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8389307Z 2025-09-07T07:13:19.8389417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8389808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8390155Z return mod(**inputs) 2025-09-07T07:13:19.8390507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8390898Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8391291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8391715Z outputs = self.model.decoder( 2025-09-07T07:13:19.8392084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8392475Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8392872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8393278Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8393656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8394046Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8394454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8394877Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8395024Z 2025-09-07T07:13:19.8395129Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8395494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8395825Z return mod(**inputs) 2025-09-07T07:13:19.8396165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8396529Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8396920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8397312Z outputs = self.model.decoder( 2025-09-07T07:13:19.8397655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8398000Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8398378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8398760Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8399117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8399488Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8399890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8400305Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8400481Z 2025-09-07T07:13:19.8400585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8400940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8401253Z return mod(**inputs) 2025-09-07T07:13:19.8401577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8401930Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8402303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8402684Z outputs = self.model.decoder( 2025-09-07T07:13:19.8403019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8403371Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8403746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8404120Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8404461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8404819Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8405196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8405584Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8405738Z 2025-09-07T07:13:19.8405849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8406204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8406525Z return mod(**inputs) 2025-09-07T07:13:19.8406852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8407209Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8407581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8407950Z outputs = self.model.decoder( 2025-09-07T07:13:19.8408296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8408664Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8409028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8409392Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8409735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8410099Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8410475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-09-07T07:13:19.8410907Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-09-07T07:13:19.8411092Z 2025-09-07T07:13:19.8411196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8411557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8411880Z return mod(**inputs) 2025-09-07T07:13:19.8412206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8412564Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8412925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8413301Z outputs = self.model.decoder( 2025-09-07T07:13:19.8413661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8414028Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8414400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8414771Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8415107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8415462Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8415830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8416217Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8416613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8417017Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8417180Z 2025-09-07T07:13:19.8417289Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8417645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8417966Z return mod(**inputs) 2025-09-07T07:13:19.8418296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8418651Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8419023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8419431Z outputs = self.model.decoder( 2025-09-07T07:13:19.8419933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8420302Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8420688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8421072Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8421421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8421798Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8422185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8422632Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8423030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8423422Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8423565Z 2025-09-07T07:13:19.8423670Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8424037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8424368Z return mod(**inputs) 2025-09-07T07:13:19.8424693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8425057Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8425436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8425872Z outputs = self.model.decoder( 2025-09-07T07:13:19.8426232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8426617Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8426989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8427358Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8427748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8428153Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8428564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8429004Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8429434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8429855Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8430008Z 2025-09-07T07:13:19.8430101Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8430321Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8430533Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8430746Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8430978Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8431361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8431733Z return mod(**inputs) 2025-09-07T07:13:19.8432093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8432467Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8432868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8433277Z outputs = self.model.decoder( 2025-09-07T07:13:19.8433650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8434074Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8434471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8434880Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8435258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8435652Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8436072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8436512Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8436954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8437418Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8437878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8438346Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8438534Z 2025-09-07T07:13:19.8438636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8438985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8439301Z return mod(**inputs) 2025-09-07T07:13:19.8439618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8439960Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8440326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8440697Z outputs = self.model.decoder( 2025-09-07T07:13:19.8441032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8441379Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8441737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8442116Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8442471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8442822Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8443181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8443569Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8443954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8444341Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8444771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8445212Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8445377Z 2025-09-07T07:13:19.8445480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8445828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8446142Z return mod(**inputs) 2025-09-07T07:13:19.8446460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8446794Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8447153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8447522Z outputs = self.model.decoder( 2025-09-07T07:13:19.8447872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8448208Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8448582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8448958Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8449299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8449651Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8450012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8450403Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8450803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8451186Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8451316Z 2025-09-07T07:13:19.8451423Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8451767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8452086Z return mod(**inputs) 2025-09-07T07:13:19.8452405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8452747Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8453102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8453466Z outputs = self.model.decoder( 2025-09-07T07:13:19.8453798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8454140Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8454506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8454866Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8455220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8455575Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8455959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8456328Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8456473Z 2025-09-07T07:13:19.8456577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8456941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8457268Z return mod(**inputs) 2025-09-07T07:13:19.8457596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8457942Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8458319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8458695Z outputs = self.model.decoder( 2025-09-07T07:13:19.8459042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8459394Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8459779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8460160Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8460514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8460887Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8461268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8461698Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8461864Z 2025-09-07T07:13:19.8461971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8462343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8462675Z return mod(**inputs) 2025-09-07T07:13:19.8463007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8463369Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8463752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8464159Z outputs = self.model.decoder( 2025-09-07T07:13:19.8464506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8464875Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8465259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8465671Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8466132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8466523Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8466931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8467316Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8467461Z 2025-09-07T07:13:19.8467573Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8467930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8468241Z return mod(**inputs) 2025-09-07T07:13:19.8468560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8468910Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8469307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8469682Z outputs = self.model.decoder( 2025-09-07T07:13:19.8470057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8470417Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8470798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8471173Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8471515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8471879Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8472255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8472659Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8473050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8473470Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8473643Z 2025-09-07T07:13:19.8473748Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8474106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8474431Z return mod(**inputs) 2025-09-07T07:13:19.8474750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8475102Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8475492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8475866Z outputs = self.model.decoder( 2025-09-07T07:13:19.8476205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8476548Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8476917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8477288Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8477631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8477983Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8478388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8478798Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8479203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8479591Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8479732Z 2025-09-07T07:13:19.8479839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8480207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8480531Z return mod(**inputs) 2025-09-07T07:13:19.8480856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8481200Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8481573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8481946Z outputs = self.model.decoder( 2025-09-07T07:13:19.8482285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8482634Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8483010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8483389Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8483750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8484110Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8484487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8484877Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8485269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8485643Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8485779Z 2025-09-07T07:13:19.8485866Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8486067Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8486151Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8486226Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8486334Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8486535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8486600Z return mod(**inputs) 2025-09-07T07:13:19.8486822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8486894Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8487142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8487251Z outputs = self.model.decoder( 2025-09-07T07:13:19.8487464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8487539Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8487777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8487857Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8488076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8488161Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8488398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8488513Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8488758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8488858Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8489165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8489294Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8489298Z 2025-09-07T07:13:19.8489406Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8489597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8489660Z return mod(**inputs) 2025-09-07T07:13:19.8489879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8489953Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8490192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8490264Z outputs = self.model.decoder( 2025-09-07T07:13:19.8490471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8490548Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8490803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8490895Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8491111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8491188Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8491427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8491524Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8491762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8491855Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8492145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8492252Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8492256Z 2025-09-07T07:13:19.8492355Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8492556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8492621Z return mod(**inputs) 2025-09-07T07:13:19.8492838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8492912Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8493145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8493239Z outputs = self.model.decoder( 2025-09-07T07:13:19.8493448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8493527Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8493761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8493831Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8494051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8494128Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8494366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8494477Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8494717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8494798Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8494802Z 2025-09-07T07:13:19.8494900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8495100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8495166Z return mod(**inputs) 2025-09-07T07:13:19.8495381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8495454Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8495681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8495760Z outputs = self.model.decoder( 2025-09-07T07:13:19.8495965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8496044Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8496272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8496361Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8496583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8496676Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8496915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8496995Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8496999Z 2025-09-07T07:13:19.8497106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8497298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8497362Z return mod(**inputs) 2025-09-07T07:13:19.8497578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8497649Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8497891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8497961Z outputs = self.model.decoder( 2025-09-07T07:13:19.8498180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8498261Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8498546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8498628Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8498854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8498951Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8499207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8499308Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8499313Z 2025-09-07T07:13:19.8499424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8499627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8499702Z return mod(**inputs) 2025-09-07T07:13:19.8499923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8499999Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8500272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8500345Z outputs = self.model.decoder( 2025-09-07T07:13:19.8500571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8500644Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8500890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8500971Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8501207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8501299Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8501568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8501664Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8501677Z 2025-09-07T07:13:19.8501778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8501979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8502055Z return mod(**inputs) 2025-09-07T07:13:19.8502278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8502378Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8502638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8502713Z outputs = self.model.decoder( 2025-09-07T07:13:19.8502941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8503014Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8503271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8503346Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8503570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8503656Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8503903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-09-07T07:13:19.8504047Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-09-07T07:13:19.8504051Z 2025-09-07T07:13:19.8504155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8504361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8504427Z return mod(**inputs) 2025-09-07T07:13:19.8504649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8504734Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8504984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8505082Z outputs = self.model.decoder( 2025-09-07T07:13:19.8505304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8505381Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8505635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8505772Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8506015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8506094Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8506339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8506484Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8506749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8506879Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8506883Z 2025-09-07T07:13:19.8506997Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8507222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8507309Z return mod(**inputs) 2025-09-07T07:13:19.8507531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8507615Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8507861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8507945Z outputs = self.model.decoder( 2025-09-07T07:13:19.8508165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8508242Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8508497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8508588Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8508835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8508918Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8509164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8509271Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8509517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8509609Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8509615Z 2025-09-07T07:13:19.8509721Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8509930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8509996Z return mod(**inputs) 2025-09-07T07:13:19.8510217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8510303Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8510549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8510632Z outputs = self.model.decoder( 2025-09-07T07:13:19.8510852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8510926Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8511179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8511272Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8511504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8511586Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8511830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8511939Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8512185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8512278Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8512281Z 2025-09-07T07:13:19.8512379Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8512467Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8512545Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8512626Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8512738Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8512938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8513011Z return mod(**inputs) 2025-09-07T07:13:19.8513232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8513308Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8513557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8513629Z outputs = self.model.decoder( 2025-09-07T07:13:19.8513856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8513932Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8514175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8514257Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8514481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8514587Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8514853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8514963Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8515210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8515313Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8515622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8515760Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8515764Z 2025-09-07T07:13:19.8515874Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8516081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8516146Z return mod(**inputs) 2025-09-07T07:13:19.8516378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8516454Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8516708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8516780Z outputs = self.model.decoder( 2025-09-07T07:13:19.8516998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8517078Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8517341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8517419Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8517641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8517727Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8517984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8518079Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8518322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8518434Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8518729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8518840Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8518844Z 2025-09-07T07:13:19.8518943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8519147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8519211Z return mod(**inputs) 2025-09-07T07:13:19.8519431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8519504Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8519926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8520005Z outputs = self.model.decoder( 2025-09-07T07:13:19.8520224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8520306Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8520547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8520626Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8520888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8520968Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8521240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8521339Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8521582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8521665Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8521668Z 2025-09-07T07:13:19.8521770Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8521979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8522044Z return mod(**inputs) 2025-09-07T07:13:19.8522271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8522346Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8522591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8522664Z outputs = self.model.decoder( 2025-09-07T07:13:19.8522878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8522958Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8523196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8523275Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8523554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8523631Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8523880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8523961Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8523966Z 2025-09-07T07:13:19.8524077Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8524271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8524335Z return mod(**inputs) 2025-09-07T07:13:19.8524556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8524659Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8524902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8524974Z outputs = self.model.decoder( 2025-09-07T07:13:19.8525195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8525268Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8525507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8525585Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8525803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8525892Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8526130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8526230Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8526235Z 2025-09-07T07:13:19.8526346Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8526541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8526611Z return mod(**inputs) 2025-09-07T07:13:19.8526845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8526933Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8527179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8527249Z outputs = self.model.decoder( 2025-09-07T07:13:19.8527469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8527543Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8527789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8527861Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8528079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8528165Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8528404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8528491Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8528494Z 2025-09-07T07:13:19.8528594Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8528791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8528866Z return mod(**inputs) 2025-09-07T07:13:19.8529086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8529184Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8529441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8529518Z outputs = self.model.decoder( 2025-09-07T07:13:19.8529733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8529807Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8530050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8530121Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8530343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8530441Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8530677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8530784Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8531021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8531140Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8531144Z 2025-09-07T07:13:19.8531243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8531440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8531512Z return mod(**inputs) 2025-09-07T07:13:19.8531726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8531806Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8532044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8532123Z outputs = self.model.decoder( 2025-09-07T07:13:19.8532337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8532409Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8532673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8532748Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8532997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8533077Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8533327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8533436Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8533672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8533762Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8533766Z 2025-09-07T07:13:19.8533869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8534077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8534143Z return mod(**inputs) 2025-09-07T07:13:19.8534365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8534446Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8534689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8534768Z outputs = self.model.decoder( 2025-09-07T07:13:19.8534989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8535062Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8535330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8535401Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8535633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8535712Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8535955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8536063Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8536309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8536415Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8536419Z 2025-09-07T07:13:19.8536502Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8536584Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8536670Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8536749Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8536861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8537063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8537131Z return mod(**inputs) 2025-09-07T07:13:19.8537359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8537434Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8537685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8537762Z outputs = self.model.decoder( 2025-09-07T07:13:19.8537994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8538075Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8538333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8538417Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8538672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8538782Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8539045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8539151Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8539427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8539541Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8539867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8540016Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8540020Z 2025-09-07T07:13:19.8540137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8540356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8540430Z return mod(**inputs) 2025-09-07T07:13:19.8540673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8540753Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8541068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8541149Z outputs = self.model.decoder( 2025-09-07T07:13:19.8541387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8541493Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8541766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8541851Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8542092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8542176Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8542454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8542560Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8542837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8542957Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8543281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8543400Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8543403Z 2025-09-07T07:13:19.8543514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8543737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8543807Z return mod(**inputs) 2025-09-07T07:13:19.8544046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8544126Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8544400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8544488Z outputs = self.model.decoder( 2025-09-07T07:13:19.8544721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8544809Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8545082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8545184Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8545438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8545523Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8545859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8545970Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8546260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8546350Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8546355Z 2025-09-07T07:13:19.8546470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8546701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8546777Z return mod(**inputs) 2025-09-07T07:13:19.8547025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8547109Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8547373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8547456Z outputs = self.model.decoder( 2025-09-07T07:13:19.8547675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8547758Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8548007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8548119Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8548356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8548433Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8548671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8548750Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8548754Z 2025-09-07T07:13:19.8548862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8549057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8549138Z return mod(**inputs) 2025-09-07T07:13:19.8549367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8549445Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8549699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8549771Z outputs = self.model.decoder( 2025-09-07T07:13:19.8549995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8550078Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8550323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8550403Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8550627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8550717Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8550968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8551067Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8551070Z 2025-09-07T07:13:19.8551180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8551406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8551481Z return mod(**inputs) 2025-09-07T07:13:19.8551705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8551781Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8552024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8552095Z outputs = self.model.decoder( 2025-09-07T07:13:19.8552317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8552389Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8552619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8552696Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8552911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8552996Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8553228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8553313Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8553316Z 2025-09-07T07:13:19.8553413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8553607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8553678Z return mod(**inputs) 2025-09-07T07:13:19.8553910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8553987Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8554227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8554299Z outputs = self.model.decoder( 2025-09-07T07:13:19.8554520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8554592Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8554832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8554901Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8555140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8555226Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8555467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-09-07T07:13:19.8555617Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-09-07T07:13:19.8555620Z 2025-09-07T07:13:19.8555722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8555922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8555986Z return mod(**inputs) 2025-09-07T07:13:19.8556195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8556273Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8556505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8556583Z outputs = self.model.decoder( 2025-09-07T07:13:19.8556792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8556864Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8557122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8557192Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8557432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8557514Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8557766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8557869Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8558117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8558243Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8558248Z 2025-09-07T07:13:19.8558351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8558560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8558626Z return mod(**inputs) 2025-09-07T07:13:19.8558851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8558935Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8559179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8559259Z outputs = self.model.decoder( 2025-09-07T07:13:19.8559482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8559558Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8559807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8559900Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8560133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8560212Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8560464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8560565Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8560809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8560900Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8560920Z 2025-09-07T07:13:19.8561024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8561237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8561304Z return mod(**inputs) 2025-09-07T07:13:19.8561517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8561599Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8561840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8561919Z outputs = self.model.decoder( 2025-09-07T07:13:19.8562133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8562205Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8562450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8562521Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8562747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8562827Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8563073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8563185Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8563441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8563535Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8563538Z 2025-09-07T07:13:19.8563617Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8563701Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8563777Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8563856Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8563964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8564159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8564230Z return mod(**inputs) 2025-09-07T07:13:19.8564443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8564515Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8564766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8564839Z outputs = self.model.decoder( 2025-09-07T07:13:19.8565061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8565134Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8565369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8565450Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8565684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8565770Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8566008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8566112Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8566354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8566448Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8566741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8566889Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8566892Z 2025-09-07T07:13:19.8567000Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8567201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8567267Z return mod(**inputs) 2025-09-07T07:13:19.8567496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8567573Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8567825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8567898Z outputs = self.model.decoder( 2025-09-07T07:13:19.8568124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8568198Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8568444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8568527Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8568748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8568834Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8569091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8569211Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8569461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8569558Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8569862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8569976Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8569980Z 2025-09-07T07:13:19.8570092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8570294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8570360Z return mod(**inputs) 2025-09-07T07:13:19.8570587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8570662Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8570924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8570995Z outputs = self.model.decoder( 2025-09-07T07:13:19.8571211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8571292Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8571535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8571632Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8571850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8571929Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8572175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8572272Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8572518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8572600Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8572603Z 2025-09-07T07:13:19.8572712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8572925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8572992Z return mod(**inputs) 2025-09-07T07:13:19.8573215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8573287Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8573532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8573602Z outputs = self.model.decoder( 2025-09-07T07:13:19.8573815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8573893Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8574129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8574209Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8574428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8574508Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8574752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8574831Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8574852Z 2025-09-07T07:13:19.8574961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8575170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8575241Z return mod(**inputs) 2025-09-07T07:13:19.8575455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8575527Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8575771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8575843Z outputs = self.model.decoder( 2025-09-07T07:13:19.8576063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8576135Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8576375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8576455Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8576673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8576756Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8576993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8577089Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8577102Z 2025-09-07T07:13:19.8577202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8577397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8577491Z return mod(**inputs) 2025-09-07T07:13:19.8577709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8577790Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8578038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8578112Z outputs = self.model.decoder( 2025-09-07T07:13:19.8578344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8578418Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8578675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8578764Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8578995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8579088Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8579333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8579422Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8579425Z 2025-09-07T07:13:19.8579530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8579737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8579804Z return mod(**inputs) 2025-09-07T07:13:19.8580025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8580111Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8580353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8580433Z outputs = self.model.decoder( 2025-09-07T07:13:19.8580653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8580727Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8580997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8581085Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8581316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8581397Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8581644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8581757Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8582005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8582131Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8582134Z 2025-09-07T07:13:19.8582239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8582449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8582517Z return mod(**inputs) 2025-09-07T07:13:19.8582740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8582823Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8583082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8583169Z outputs = self.model.decoder( 2025-09-07T07:13:19.8583401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8583506Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8583788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8583863Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8584109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8584198Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8584482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8584596Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8584879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8584992Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8584995Z 2025-09-07T07:13:19.8585109Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8585342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8585412Z return mod(**inputs) 2025-09-07T07:13:19.8585645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8585811Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8586082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8586167Z outputs = self.model.decoder( 2025-09-07T07:13:19.8586401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8586481Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8586764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8586843Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8587086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8587171Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8587463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8587595Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8587864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8587962Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8587966Z 2025-09-07T07:13:19.8588050Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8588143Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8588224Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8588308Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8588421Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8588642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8588717Z return mod(**inputs) 2025-09-07T07:13:19.8588953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8589033Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8589296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8589372Z outputs = self.model.decoder( 2025-09-07T07:13:19.8589606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8589686Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8589955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8590060Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8590298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8590392Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8590649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8590754Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8591028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8591131Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8591466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8591608Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8591613Z 2025-09-07T07:13:19.8591726Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8591938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8592008Z return mod(**inputs) 2025-09-07T07:13:19.8592248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8592329Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8592591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8592667Z outputs = self.model.decoder( 2025-09-07T07:13:19.8592898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8592987Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8593243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8593327Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8593588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8593680Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8593954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8594060Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8594323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8594425Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8594747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8594865Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8594869Z 2025-09-07T07:13:19.8594977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8595200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8595270Z return mod(**inputs) 2025-09-07T07:13:19.8595510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8595590Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8595848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8595933Z outputs = self.model.decoder( 2025-09-07T07:13:19.8596168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8596255Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8596537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8596620Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8596858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8596943Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8597209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8597322Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8597594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8597695Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8597698Z 2025-09-07T07:13:19.8597801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8598010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8598078Z return mod(**inputs) 2025-09-07T07:13:19.8598314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8598394Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8598661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8598739Z outputs = self.model.decoder( 2025-09-07T07:13:19.8598971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8599055Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8599310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8599394Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8599631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8599715Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8599996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8600084Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8600087Z 2025-09-07T07:13:19.8600218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8600438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8600509Z return mod(**inputs) 2025-09-07T07:13:19.8600748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8600829Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8601096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8601176Z outputs = self.model.decoder( 2025-09-07T07:13:19.8601415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8601495Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8601754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8601837Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8602077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8602168Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8602429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8602536Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8602558Z 2025-09-07T07:13:19.8602675Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8602889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8602966Z return mod(**inputs) 2025-09-07T07:13:19.8603208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8603287Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8603563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8603640Z outputs = self.model.decoder( 2025-09-07T07:13:19.8603886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8603979Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8604241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8604317Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8604551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8604640Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8604900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8604995Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8604999Z 2025-09-07T07:13:19.8605108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8605320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8605396Z return mod(**inputs) 2025-09-07T07:13:19.8605626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8605710Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8605969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8606046Z outputs = self.model.decoder( 2025-09-07T07:13:19.8606298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8606377Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8606657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8606735Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8606978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8607063Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8607323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-09-07T07:13:19.8607480Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-09-07T07:13:19.8607483Z 2025-09-07T07:13:19.8607593Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8607817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8607886Z return mod(**inputs) 2025-09-07T07:13:19.8608118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8608207Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8608472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8608560Z outputs = self.model.decoder( 2025-09-07T07:13:19.8608805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8608886Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8609182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8609260Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8609513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8609599Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8609872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8609981Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8610252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-09-07T07:13:19.8610402Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:13:19.8610406Z 2025-09-07T07:13:19.8610518Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8610743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8610825Z return mod(**inputs) 2025-09-07T07:13:19.8611060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8611147Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8611407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8611491Z outputs = self.model.decoder( 2025-09-07T07:13:19.8611721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8611806Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8612066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8612143Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8612386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8612468Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8612749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8612855Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8613134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-09-07T07:13:19.8613229Z key_states = self.k_proj(hidden_states) 2025-09-07T07:13:19.8613232Z 2025-09-07T07:13:19.8613341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8613559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8613631Z return mod(**inputs) 2025-09-07T07:13:19.8613863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8613950Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8614212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8614298Z outputs = self.model.decoder( 2025-09-07T07:13:19.8614534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8614619Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8614879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8614957Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8615203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8615289Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8615569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8615674Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8615933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-09-07T07:13:19.8616030Z value_states = self.v_proj(hidden_states) 2025-09-07T07:13:19.8616035Z 2025-09-07T07:13:19.8616120Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8616214Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8616296Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8616377Z cudagraph partition due to non gpu ops 2025-09-07T07:13:19.8616494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8616735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8616811Z return mod(**inputs) 2025-09-07T07:13:19.8617048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8617126Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8617397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8617473Z outputs = self.model.decoder( 2025-09-07T07:13:19.8617713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8617790Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8618059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8618136Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8618382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8618478Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8618743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8618856Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8619139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8619291Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8619795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:19.8621108Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:19.8621127Z 2025-09-07T07:13:19.8621331Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8621600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8621707Z return mod(**inputs) 2025-09-07T07:13:19.8621999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8622091Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8622395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8622484Z outputs = self.model.decoder( 2025-09-07T07:13:19.8622753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8622841Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8623122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8623216Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8623474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8623882Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8624165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8624287Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8624573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-09-07T07:13:19.8624688Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:19.8625026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:19.8625152Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:19.8626420Z 2025-09-07T07:13:19.8626566Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8626884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8626970Z return mod(**inputs) 2025-09-07T07:13:19.8627240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8627327Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8627620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8627714Z outputs = self.model.decoder( 2025-09-07T07:13:19.8627970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8628062Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8628344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8628437Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8628687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8628779Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8629054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-09-07T07:13:19.8629206Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:19.8629511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-09-07T07:13:19.8629606Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:19.8629611Z 2025-09-07T07:13:19.8629818Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8630089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8630174Z return mod(**inputs) 2025-09-07T07:13:19.8630425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8630515Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8630831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8630916Z outputs = self.model.decoder( 2025-09-07T07:13:19.8631172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8631256Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8631540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8631624Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8631875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8631974Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8632255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-09-07T07:13:19.8632378Z hidden_states = self.fc1(hidden_states) 2025-09-07T07:13:19.8632383Z 2025-09-07T07:13:19.8632499Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8632736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8632816Z return mod(**inputs) 2025-09-07T07:13:19.8633067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8633155Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8633429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8633520Z outputs = self.model.decoder( 2025-09-07T07:13:19.8633785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8633858Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8634116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8634190Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8634463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8634542Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8634782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-09-07T07:13:19.8634889Z hidden_states = self.activation_fn(hidden_states) 2025-09-07T07:13:19.8634893Z 2025-09-07T07:13:19.8634995Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8635207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8635273Z return mod(**inputs) 2025-09-07T07:13:19.8635488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8635568Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8635825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-09-07T07:13:19.8635907Z outputs = self.model.decoder( 2025-09-07T07:13:19.8636147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8636229Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8636466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-09-07T07:13:19.8636537Z layer_outputs = decoder_layer( 2025-09-07T07:13:19.8636766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:19.8636844Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:19.8637088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-09-07T07:13:19.8637167Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:19.8637171Z 2025-09-07T07:13:19.8637276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8637480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8637546Z return mod(**inputs) 2025-09-07T07:13:19.8637763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8637833Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8638069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 841, in forward 2025-09-07T07:13:19.8638170Z logits = self.lm_head(outputs[0]).contiguous() 2025-09-07T07:13:19.8638174Z 2025-09-07T07:13:19.8638295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:19.8638495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:19.8638560Z return mod(**inputs) 2025-09-07T07:13:19.8638782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-09-07T07:13:19.8638855Z output = func(self, *args, **kwargs) 2025-09-07T07:13:19.8639097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 847, in forward 2025-09-07T07:13:19.8639177Z loss = self.loss_function( 2025-09-07T07:13:19.8639420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-09-07T07:13:19.8639619Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-09-07T07:13:19.8639870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-09-07T07:13:19.8640074Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-09-07T07:13:19.8640087Z 2025-09-07T07:13:32.9284402Z Compilation time (from dynamo_timed): 18.327901636 2025-09-07T07:13:32.9712502Z pass 2025-09-07T07:13:32.9712997Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:32.9713872Z TIMING: _recursive_pre_grad_passes:0.00823 _recursive_joint_graph_passes:0.62613 _recursive_post_grad_passes:0.09706 async_compile.wait:0.86145 code_gen:11.58497 inductor_compile:12.85938 backend_compile:16.04758 gc:0.00032 entire_frame_compile:18.3279 total_wall_time:18.3279 2025-09-07T07:13:32.9714937Z STATS: call_* op count: 415 | FakeTensorMode.__torch_dispatch__:12795 | FakeTensor.__torch_dispatch__:4179 | ProxyTorchDispatchMode.__torch_dispatch__:4707 2025-09-07T07:13:32.9715563Z Dynamo produced 1 graphs covering 415 ops with 0 graph breaks (0 unique) 2025-09-07T07:13:35.6968956Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:13:35.6970129Z import pynvml # type: ignore[import] 2025-09-07T07:13:38.4556854Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:13:38.4557883Z from pkg_resources import resource_filename 2025-09-07T07:13:39.1221421Z 2025-09-07T07:13:40.3489031Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:13:40.3491138Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:13:40.3496397Z cpu eval PLBartForCausalLM 2025-09-07T07:13:41.0540562Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:41.4128255Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:41.7098643Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:46.8197381Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8197713Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8197941Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8198180Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8198425Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8198635Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8198910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8199345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8199981Z return mod(**inputs) 2025-09-07T07:13:46.8200413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8200877Z outputs = self.model.decoder( 2025-09-07T07:13:46.8201353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8201861Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8202279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8202754Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8203210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8203748Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8204212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:13:46.8204743Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:13:46.8204988Z 2025-09-07T07:13:46.8205108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8205503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8205857Z return mod(**inputs) 2025-09-07T07:13:46.8206257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8206713Z outputs = self.model.decoder( 2025-09-07T07:13:46.8207179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8207628Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8208020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8208421Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8208917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8209399Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8209903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:13:46.8210343Z key_states = self.k_proj(current_states) 2025-09-07T07:13:46.8210499Z 2025-09-07T07:13:46.8210614Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8211013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8211379Z return mod(**inputs) 2025-09-07T07:13:46.8211794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8212240Z outputs = self.model.decoder( 2025-09-07T07:13:46.8212687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8213136Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8213520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8213924Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8214344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8214795Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8215259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:13:46.8215696Z value_states = self.v_proj(current_states) 2025-09-07T07:13:46.8215873Z 2025-09-07T07:13:46.8215960Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8216189Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8216415Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8216640Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8216895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8217280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8217640Z return mod(**inputs) 2025-09-07T07:13:46.8218051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8218478Z outputs = self.model.decoder( 2025-09-07T07:13:46.8218897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8219344Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8220115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8220515Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8220942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8221387Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8221838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8222290Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8222784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:46.8223314Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:46.8223516Z 2025-09-07T07:13:46.8223633Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8224028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8224376Z return mod(**inputs) 2025-09-07T07:13:46.8224812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8225247Z outputs = self.model.decoder( 2025-09-07T07:13:46.8225897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8226342Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8226726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8227123Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8227549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8228017Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8228491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8228944Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8229430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:46.8229931Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:46.8230115Z 2025-09-07T07:13:46.8230230Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8230619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8230968Z return mod(**inputs) 2025-09-07T07:13:46.8231334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8231753Z outputs = self.model.decoder( 2025-09-07T07:13:46.8232140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8232532Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8232879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8233235Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8233642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8234067Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8234491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:13:46.8234951Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:46.8235095Z 2025-09-07T07:13:46.8235203Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8235570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8235909Z return mod(**inputs) 2025-09-07T07:13:46.8236280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8236676Z outputs = self.model.decoder( 2025-09-07T07:13:46.8237054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8237444Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8237793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8238156Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8238543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8238995Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8239177Z 2025-09-07T07:13:46.8239285Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8239677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8240015Z return mod(**inputs) 2025-09-07T07:13:46.8240434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8240842Z outputs = self.model.decoder( 2025-09-07T07:13:46.8241239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8241642Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8241996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8242368Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8242778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8243226Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8243621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:13:46.8243967Z return self.act(input) 2025-09-07T07:13:46.8244091Z 2025-09-07T07:13:46.8244198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8244572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8244908Z return mod(**inputs) 2025-09-07T07:13:46.8245290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8245689Z outputs = self.model.decoder( 2025-09-07T07:13:46.8246096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8246493Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8246844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8247203Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8247612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:13:46.8248022Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:46.8248163Z 2025-09-07T07:13:46.8248277Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8248663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8248987Z return mod(**inputs) 2025-09-07T07:13:46.8249359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8249761Z outputs = self.model.decoder( 2025-09-07T07:13:46.8250160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8250564Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8250917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8251297Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8251699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8252129Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8252552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:13:46.8253035Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:13:46.8253252Z 2025-09-07T07:13:46.8253361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8253754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8254084Z return mod(**inputs) 2025-09-07T07:13:46.8254474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8254883Z outputs = self.model.decoder( 2025-09-07T07:13:46.8255278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8255686Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8256051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8256419Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8256832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8257268Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8257700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:13:46.8258111Z key_states = self.k_proj(current_states) 2025-09-07T07:13:46.8258256Z 2025-09-07T07:13:46.8258362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8258731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8259072Z return mod(**inputs) 2025-09-07T07:13:46.8259441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8259835Z outputs = self.model.decoder( 2025-09-07T07:13:46.8260240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8260641Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8261003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8261371Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8261776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8262204Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8262653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:13:46.8263127Z value_states = self.v_proj(current_states) 2025-09-07T07:13:46.8263281Z 2025-09-07T07:13:46.8263371Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8263606Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8263837Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8264063Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8264320Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8264710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8265069Z return mod(**inputs) 2025-09-07T07:13:46.8265452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8265953Z outputs = self.model.decoder( 2025-09-07T07:13:46.8266383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8266830Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8267245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8267646Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8268085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8268540Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8268985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8269415Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8269877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:46.8270379Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:46.8270571Z 2025-09-07T07:13:46.8270679Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8271048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8271382Z return mod(**inputs) 2025-09-07T07:13:46.8271765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8272169Z outputs = self.model.decoder( 2025-09-07T07:13:46.8272583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8273011Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8273397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8273793Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8274218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8274675Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8275139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8275589Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8276076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:46.8276558Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:46.8276743Z 2025-09-07T07:13:46.8276855Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8277243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8277603Z return mod(**inputs) 2025-09-07T07:13:46.8278008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8278465Z outputs = self.model.decoder( 2025-09-07T07:13:46.8278897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8279328Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8279710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8280097Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8280530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8280991Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8281442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:13:46.8281890Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:46.8282039Z 2025-09-07T07:13:46.8282151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8282544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8282894Z return mod(**inputs) 2025-09-07T07:13:46.8283291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8283794Z outputs = self.model.decoder( 2025-09-07T07:13:46.8284294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8284729Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8285109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8285504Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8285934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8286416Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8286614Z 2025-09-07T07:13:46.8286727Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8287120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8287496Z return mod(**inputs) 2025-09-07T07:13:46.8287896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8288325Z outputs = self.model.decoder( 2025-09-07T07:13:46.8288763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8289200Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8289570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8289968Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8290399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8290902Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8291325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:13:46.8291689Z return self.act(input) 2025-09-07T07:13:46.8291817Z 2025-09-07T07:13:46.8291930Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8292323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8292695Z return mod(**inputs) 2025-09-07T07:13:46.8293112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8293552Z outputs = self.model.decoder( 2025-09-07T07:13:46.8293967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8294392Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8294744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8295107Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8295514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:13:46.8295924Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:46.8296063Z 2025-09-07T07:13:46.8296176Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8296542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8296866Z return mod(**inputs) 2025-09-07T07:13:46.8297246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8297653Z outputs = self.model.decoder( 2025-09-07T07:13:46.8298046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8298437Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8298834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8299224Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8299631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8300061Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8300478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:13:46.8300961Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:13:46.8301180Z 2025-09-07T07:13:46.8301286Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8301654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8301989Z return mod(**inputs) 2025-09-07T07:13:46.8302361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8302772Z outputs = self.model.decoder( 2025-09-07T07:13:46.8303186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8303612Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8303989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8304393Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8304835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8305369Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8305912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:13:46.8306369Z key_states = self.k_proj(current_states) 2025-09-07T07:13:46.8306539Z 2025-09-07T07:13:46.8306656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8307052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8307407Z return mod(**inputs) 2025-09-07T07:13:46.8307787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8308212Z outputs = self.model.decoder( 2025-09-07T07:13:46.8308610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8309017Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8309378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8309755Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8310159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8310591Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8311027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:13:46.8311433Z value_states = self.v_proj(current_states) 2025-09-07T07:13:46.8311574Z 2025-09-07T07:13:46.8311656Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8311871Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8312085Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8312300Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8312536Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8312907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8313265Z return mod(**inputs) 2025-09-07T07:13:46.8313666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8314070Z outputs = self.model.decoder( 2025-09-07T07:13:46.8314458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8314860Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8315224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8315588Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8315981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8316392Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8316807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8317225Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8317672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:46.8318147Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:46.8318337Z 2025-09-07T07:13:46.8318440Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8318802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8319125Z return mod(**inputs) 2025-09-07T07:13:46.8319532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8320080Z outputs = self.model.decoder( 2025-09-07T07:13:46.8320493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8320901Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8321275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8321642Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8322045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8322505Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8323035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8323502Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8323951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:46.8324429Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:46.8324599Z 2025-09-07T07:13:46.8324704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8325068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8325394Z return mod(**inputs) 2025-09-07T07:13:46.8325758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8326162Z outputs = self.model.decoder( 2025-09-07T07:13:46.8326554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8326953Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8327301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8327653Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8328082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8328529Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8328943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:13:46.8329363Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:46.8329500Z 2025-09-07T07:13:46.8329603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8329969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8330293Z return mod(**inputs) 2025-09-07T07:13:46.8330666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8331058Z outputs = self.model.decoder( 2025-09-07T07:13:46.8331448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8331841Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8332201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8332574Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8332971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8333419Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8333602Z 2025-09-07T07:13:46.8333707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8334103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8334430Z return mod(**inputs) 2025-09-07T07:13:46.8334802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8335207Z outputs = self.model.decoder( 2025-09-07T07:13:46.8335602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8336004Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8336354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8336723Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8337149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8337601Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8337996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:13:46.8338340Z return self.act(input) 2025-09-07T07:13:46.8338460Z 2025-09-07T07:13:46.8338569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8338939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8339293Z return mod(**inputs) 2025-09-07T07:13:46.8339665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8340068Z outputs = self.model.decoder( 2025-09-07T07:13:46.8340465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8340871Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8341226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8341589Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8342013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:13:46.8342428Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:46.8342568Z 2025-09-07T07:13:46.8342699Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8343083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8343425Z return mod(**inputs) 2025-09-07T07:13:46.8343827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8344253Z outputs = self.model.decoder( 2025-09-07T07:13:46.8344663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8345077Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8345447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8345903Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8346343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8346798Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8347247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:13:46.8347732Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:13:46.8347954Z 2025-09-07T07:13:46.8348062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8348442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8348825Z return mod(**inputs) 2025-09-07T07:13:46.8349223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8349692Z outputs = self.model.decoder( 2025-09-07T07:13:46.8350108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8350526Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8350896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8351289Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8351756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8352271Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8352722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:13:46.8353158Z key_states = self.k_proj(current_states) 2025-09-07T07:13:46.8353310Z 2025-09-07T07:13:46.8353426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8353824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8354173Z return mod(**inputs) 2025-09-07T07:13:46.8354590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8355023Z outputs = self.model.decoder( 2025-09-07T07:13:46.8355459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8355889Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8356268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8356654Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8357099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8357557Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8358023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:13:46.8358466Z value_states = self.v_proj(current_states) 2025-09-07T07:13:46.8358619Z 2025-09-07T07:13:46.8358705Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8358940Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8359173Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8359397Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8359644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8360037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8360388Z return mod(**inputs) 2025-09-07T07:13:46.8360790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8361218Z outputs = self.model.decoder( 2025-09-07T07:13:46.8361631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8362064Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8362443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8362841Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8363278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8363783Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8364229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8364678Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8365165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:46.8365695Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:46.8365903Z 2025-09-07T07:13:46.8366015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8366406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8366772Z return mod(**inputs) 2025-09-07T07:13:46.8367173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8367599Z outputs = self.model.decoder( 2025-09-07T07:13:46.8368015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8368438Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8368817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8369203Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8369601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8370031Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8370458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8370888Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8371338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:46.8371806Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:46.8371979Z 2025-09-07T07:13:46.8372106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8372480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8372828Z return mod(**inputs) 2025-09-07T07:13:46.8373204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8373640Z outputs = self.model.decoder( 2025-09-07T07:13:46.8374068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8374498Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8374878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8375270Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8375680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8376111Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8376545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:13:46.8376976Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:46.8377133Z 2025-09-07T07:13:46.8377246Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8377637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8377989Z return mod(**inputs) 2025-09-07T07:13:46.8378386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8378800Z outputs = self.model.decoder( 2025-09-07T07:13:46.8379191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8379592Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8379949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8380319Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8380720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8381169Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8381352Z 2025-09-07T07:13:46.8381480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8381849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8382174Z return mod(**inputs) 2025-09-07T07:13:46.8382548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8382947Z outputs = self.model.decoder( 2025-09-07T07:13:46.8383363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8383785Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8384150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8384538Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8384968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8385443Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8385945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:13:46.8386331Z return self.act(input) 2025-09-07T07:13:46.8386466Z 2025-09-07T07:13:46.8386581Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8387018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8387370Z return mod(**inputs) 2025-09-07T07:13:46.8387785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8388214Z outputs = self.model.decoder( 2025-09-07T07:13:46.8388634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8389060Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8389445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8389832Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8390262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:13:46.8390698Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:46.8390846Z 2025-09-07T07:13:46.8390967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8391358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8391699Z return mod(**inputs) 2025-09-07T07:13:46.8392115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8392551Z outputs = self.model.decoder( 2025-09-07T07:13:46.8392983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8393409Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8393814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8394202Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8394640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8395091Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8395556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:13:46.8396059Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:13:46.8396288Z 2025-09-07T07:13:46.8396402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8396812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8397182Z return mod(**inputs) 2025-09-07T07:13:46.8397575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8398002Z outputs = self.model.decoder( 2025-09-07T07:13:46.8398441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8398866Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8399237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8399632Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8400062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8400515Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8400959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:13:46.8401384Z key_states = self.k_proj(current_states) 2025-09-07T07:13:46.8401536Z 2025-09-07T07:13:46.8401648Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8402059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8402410Z return mod(**inputs) 2025-09-07T07:13:46.8402822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8403225Z outputs = self.model.decoder( 2025-09-07T07:13:46.8403624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8404034Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8404391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8404758Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8405164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8405592Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8406025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:13:46.8406452Z value_states = self.v_proj(current_states) 2025-09-07T07:13:46.8406596Z 2025-09-07T07:13:46.8406680Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8406905Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8407122Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8407339Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8407577Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8407951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8408318Z return mod(**inputs) 2025-09-07T07:13:46.8408700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8409109Z outputs = self.model.decoder( 2025-09-07T07:13:46.8409497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8409905Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8410264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8410639Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8411058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8411543Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8412000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8412432Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8412890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:46.8413381Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:46.8413579Z 2025-09-07T07:13:46.8413688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8414060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8414400Z return mod(**inputs) 2025-09-07T07:13:46.8414819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8415246Z outputs = self.model.decoder( 2025-09-07T07:13:46.8415672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8416080Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8416438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8416830Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8417256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8417681Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8418102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8418541Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8418979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:46.8419450Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:46.8419841Z 2025-09-07T07:13:46.8419953Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8420332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8420667Z return mod(**inputs) 2025-09-07T07:13:46.8421041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8421452Z outputs = self.model.decoder( 2025-09-07T07:13:46.8421855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8422254Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8422606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8422980Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8423471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8423934Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8424392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:13:46.8424845Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:46.8425002Z 2025-09-07T07:13:46.8425116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8426235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8427134Z return mod(**inputs) 2025-09-07T07:13:46.8427771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8428651Z outputs = self.model.decoder( 2025-09-07T07:13:46.8429141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8429595Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8430001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8430406Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8430851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8431334Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8431537Z 2025-09-07T07:13:46.8431661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8432060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8432426Z return mod(**inputs) 2025-09-07T07:13:46.8432843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8433284Z outputs = self.model.decoder( 2025-09-07T07:13:46.8433776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8434203Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8434644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8435026Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8435440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8435907Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8436344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:13:46.8436722Z return self.act(input) 2025-09-07T07:13:46.8436858Z 2025-09-07T07:13:46.8436979Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8437396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8437768Z return mod(**inputs) 2025-09-07T07:13:46.8438153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8438569Z outputs = self.model.decoder( 2025-09-07T07:13:46.8438966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8439369Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8439724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8440099Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8440508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:13:46.8441034Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:46.8441182Z 2025-09-07T07:13:46.8441303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8441688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8442044Z return mod(**inputs) 2025-09-07T07:13:46.8442469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8442891Z outputs = self.model.decoder( 2025-09-07T07:13:46.8443298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8443714Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8444081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8444453Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8444862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8445301Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8446569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:13:46.8447125Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:13:46.8447357Z 2025-09-07T07:13:46.8447480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8447888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8448260Z return mod(**inputs) 2025-09-07T07:13:46.8448657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8449098Z outputs = self.model.decoder( 2025-09-07T07:13:46.8449526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8450007Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8450383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8450802Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8451237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8451700Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8452164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:13:46.8452609Z key_states = self.k_proj(current_states) 2025-09-07T07:13:46.8452772Z 2025-09-07T07:13:46.8452893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8453287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8453643Z return mod(**inputs) 2025-09-07T07:13:46.8454045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8454481Z outputs = self.model.decoder( 2025-09-07T07:13:46.8454903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8455335Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8455716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8456109Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8456546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8457038Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8457496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:13:46.8457960Z value_states = self.v_proj(current_states) 2025-09-07T07:13:46.8458118Z 2025-09-07T07:13:46.8458213Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8458450Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8458684Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8458909Z cudagraph partition due to non gpu ops 2025-09-07T07:13:46.8459155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8459547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8459946Z return mod(**inputs) 2025-09-07T07:13:46.8460357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8460800Z outputs = self.model.decoder( 2025-09-07T07:13:46.8461227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8461663Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8462067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8462465Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8462893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8463353Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8463813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8464287Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8464798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:13:46.8465374Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:13:46.8465598Z 2025-09-07T07:13:46.8465811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8466328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8466702Z return mod(**inputs) 2025-09-07T07:13:46.8467126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8467563Z outputs = self.model.decoder( 2025-09-07T07:13:46.8467987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8468416Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8468798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8469197Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8469616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8470081Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8470538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:13:46.8470996Z attn_output, attn_weights = attention_interface( 2025-09-07T07:13:46.8471475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:13:46.8471977Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:13:46.8472154Z 2025-09-07T07:13:46.8472303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8472678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8473020Z return mod(**inputs) 2025-09-07T07:13:46.8473420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8473856Z outputs = self.model.decoder( 2025-09-07T07:13:46.8474285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8474729Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8475090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8475511Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8475925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:13:46.8476365Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:13:46.8476793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:13:46.8477199Z attn_output = self.out_proj(attn_output) 2025-09-07T07:13:46.8477347Z 2025-09-07T07:13:46.8477456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8477833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8478173Z return mod(**inputs) 2025-09-07T07:13:46.8478552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8478954Z outputs = self.model.decoder( 2025-09-07T07:13:46.8479348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8479752Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8480110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8480483Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8480936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8481447Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8481625Z 2025-09-07T07:13:46.8481740Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8482126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8482493Z return mod(**inputs) 2025-09-07T07:13:46.8482899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8483335Z outputs = self.model.decoder( 2025-09-07T07:13:46.8483768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8484209Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8484591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8484967Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8485376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:13:46.8485825Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:13:46.8486224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:13:46.8486596Z return self.act(input) 2025-09-07T07:13:46.8486722Z 2025-09-07T07:13:46.8486836Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8487265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8487615Z return mod(**inputs) 2025-09-07T07:13:46.8488029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-09-07T07:13:46.8488458Z outputs = self.model.decoder( 2025-09-07T07:13:46.8488897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:13:46.8489326Z layer_outputs = decoder_layer( 2025-09-07T07:13:46.8489705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:13:46.8490097Z return super().__call__(*args, **kwargs) 2025-09-07T07:13:46.8490559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:13:46.8491000Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:13:46.8491149Z 2025-09-07T07:13:46.8491270Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8491659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8492017Z return mod(**inputs) 2025-09-07T07:13:46.8492426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1694, in forward 2025-09-07T07:13:46.8492860Z logits = self.lm_head(outputs[0]) 2025-09-07T07:13:46.8493003Z 2025-09-07T07:13:46.8493123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:13:46.8493510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:13:46.8493868Z return mod(**inputs) 2025-09-07T07:13:46.8494271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1700, in forward 2025-09-07T07:13:46.8494784Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:13:46.8495000Z 2025-09-07T07:13:57.2904408Z Compilation time (from dynamo_timed): 14.040856358 2025-09-07T07:13:57.3204274Z pass 2025-09-07T07:13:57.3205202Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:13:57.3206305Z TIMING: _recursive_pre_grad_passes:0.00595 _recursive_joint_graph_passes:0.25049 _recursive_post_grad_passes:0.05131 async_compile.wait:0.75037 code_gen:9.84988 inductor_compile:10.87742 backend_compile:12.73616 gc:0.00125 entire_frame_compile:14.04086 total_wall_time:14.04086 2025-09-07T07:13:57.3207433Z STATS: call_* op count: 198 | FakeTensorMode.__torch_dispatch__:7096 | FakeTensor.__torch_dispatch__:2414 | ProxyTorchDispatchMode.__torch_dispatch__:2533 2025-09-07T07:13:57.3208053Z Dynamo produced 1 graphs covering 198 ops with 0 graph breaks (0 unique) 2025-09-07T07:13:59.9789678Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:13:59.9791020Z import pynvml # type: ignore[import] 2025-09-07T07:14:02.7768754Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:14:02.7771174Z from pkg_resources import resource_filename 2025-09-07T07:14:03.4703358Z 2025-09-07T07:14:05.7649734Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:14:05.7650104Z loading model: 0it [00:02, ?it/s] 2025-09-07T07:14:05.7663280Z cpu eval PLBartForConditionalGeneration 2025-09-07T07:14:06.9863384Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:14:07.5391026Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:14:08.0893104Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:14:17.7765522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7767293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7767718Z return mod(**inputs) 2025-09-07T07:14:17.7768196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1357, in forward 2025-09-07T07:14:17.7768745Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-09-07T07:14:17.7769835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1084, in shift_tokens_right 2025-09-07T07:14:17.7770421Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-09-07T07:14:17.7770666Z 2025-09-07T07:14:17.7770759Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7770999Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7771229Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7771457Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7771686Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7771908Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7772169Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7772579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7772948Z return mod(**inputs) 2025-09-07T07:14:17.7773364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7773789Z outputs = self.model( 2025-09-07T07:14:17.7774183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7774664Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7775150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7775575Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7775983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7776346Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7776756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7777200Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7777663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.7778210Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.7778426Z 2025-09-07T07:14:17.7778538Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7778914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7779251Z return mod(**inputs) 2025-09-07T07:14:17.7779633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7780026Z outputs = self.model( 2025-09-07T07:14:17.7780428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7780860Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7781288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7781770Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7782144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7782573Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7783018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7783483Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7783944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.7784385Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.7784541Z 2025-09-07T07:14:17.7784682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7785074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7785448Z return mod(**inputs) 2025-09-07T07:14:17.7785915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7786360Z outputs = self.model( 2025-09-07T07:14:17.7786778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7787222Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7787636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7788029Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7788390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7788767Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7789186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7789796Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7790229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.7790692Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.7790854Z 2025-09-07T07:14:17.7790945Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7791233Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7791453Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7791676Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7791932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7792328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7792682Z return mod(**inputs) 2025-09-07T07:14:17.7793082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7793517Z outputs = self.model( 2025-09-07T07:14:17.7793897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7794320Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7794737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7795160Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7795542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7795938Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7796367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7796821Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7797285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7797746Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7798233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.7798760Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.7798961Z 2025-09-07T07:14:17.7799074Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7799468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7799814Z return mod(**inputs) 2025-09-07T07:14:17.7800232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7800674Z outputs = self.model( 2025-09-07T07:14:17.7801090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7801541Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7801963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7802388Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7802766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7803169Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7803614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7804074Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7804533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7805014Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7805505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.7806034Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.7806212Z 2025-09-07T07:14:17.7806325Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7806742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7807098Z return mod(**inputs) 2025-09-07T07:14:17.7807492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7807892Z outputs = self.model( 2025-09-07T07:14:17.7808266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7808673Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7809069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7809486Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7809859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7810258Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7810685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7811129Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7811583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.7812149Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.7812320Z 2025-09-07T07:14:17.7812469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7812869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7813229Z return mod(**inputs) 2025-09-07T07:14:17.7813609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7813999Z outputs = self.model( 2025-09-07T07:14:17.7814378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7814782Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7815184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7815594Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7815950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7816323Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7816730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.7817181Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.7817364Z 2025-09-07T07:14:17.7817473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7817846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7818181Z return mod(**inputs) 2025-09-07T07:14:17.7818572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7818973Z outputs = self.model( 2025-09-07T07:14:17.7819344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7819943Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7820372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7820818Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7821262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7821671Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7822142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.7822645Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.7823075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.7823449Z return self.act(input) 2025-09-07T07:14:17.7823581Z 2025-09-07T07:14:17.7823696Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7824103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7824472Z return mod(**inputs) 2025-09-07T07:14:17.7824889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7825318Z outputs = self.model( 2025-09-07T07:14:17.7825889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7826352Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7826799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7827214Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7827594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7827988Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7828461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-09-07T07:14:17.7828893Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.7829041Z 2025-09-07T07:14:17.7829155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7829538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7829882Z return mod(**inputs) 2025-09-07T07:14:17.7830278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7830692Z outputs = self.model( 2025-09-07T07:14:17.7831079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7831533Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7831952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7832371Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7832743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7833136Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7833564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7834009Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7834449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.7834945Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.7835165Z 2025-09-07T07:14:17.7835272Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7835645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7835978Z return mod(**inputs) 2025-09-07T07:14:17.7836374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7836769Z outputs = self.model( 2025-09-07T07:14:17.7837165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7837570Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7837975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7838355Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7838707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7839067Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7839473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7839899Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7840358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.7840793Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.7840945Z 2025-09-07T07:14:17.7841058Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7841443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7841783Z return mod(**inputs) 2025-09-07T07:14:17.7842144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7842533Z outputs = self.model( 2025-09-07T07:14:17.7842930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7843337Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7843729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7844129Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7844489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7844863Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7845270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7845681Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7846125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.7846539Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.7846683Z 2025-09-07T07:14:17.7846781Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7846995Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7847203Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7847409Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7847644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7848005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7848326Z return mod(**inputs) 2025-09-07T07:14:17.7848697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7849086Z outputs = self.model( 2025-09-07T07:14:17.7849456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7849846Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7850236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7850635Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7851033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7851425Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7851828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7852244Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7852666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7853097Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7853559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.7854047Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.7854243Z 2025-09-07T07:14:17.7854354Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7854727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7855064Z return mod(**inputs) 2025-09-07T07:14:17.7855434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7855835Z outputs = self.model( 2025-09-07T07:14:17.7856212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7856620Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7857015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7857431Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7857792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7858165Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7858572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7858990Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7859399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7859823Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7860296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.7860793Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.7860969Z 2025-09-07T07:14:17.7861091Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7861474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7861904Z return mod(**inputs) 2025-09-07T07:14:17.7862424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7862937Z outputs = self.model( 2025-09-07T07:14:17.7863379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7863798Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7864212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7864634Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7865012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7865404Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7865970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7866431Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7866923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.7867340Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.7867481Z 2025-09-07T07:14:17.7867599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7867974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7868316Z return mod(**inputs) 2025-09-07T07:14:17.7868688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7869078Z outputs = self.model( 2025-09-07T07:14:17.7869450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7869834Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7870270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7870720Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7871115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7871523Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7871964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.7872454Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.7872666Z 2025-09-07T07:14:17.7872774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7873152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7873477Z return mod(**inputs) 2025-09-07T07:14:17.7873855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7874251Z outputs = self.model( 2025-09-07T07:14:17.7874621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7875024Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7875414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7875836Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7876195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7876574Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7876988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.7877428Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.7877827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.7878178Z return self.act(input) 2025-09-07T07:14:17.7878294Z 2025-09-07T07:14:17.7878414Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7878809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7879156Z return mod(**inputs) 2025-09-07T07:14:17.7879556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7879979Z outputs = self.model( 2025-09-07T07:14:17.7880394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7880835Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7881259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7881662Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7882020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7882396Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7882799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-09-07T07:14:17.7883212Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.7883363Z 2025-09-07T07:14:17.7883469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7883842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7884221Z return mod(**inputs) 2025-09-07T07:14:17.7884597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7884997Z outputs = self.model( 2025-09-07T07:14:17.7885381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7885787Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7886181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7886585Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7886945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7887353Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7887759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7888179Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7888605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.7889089Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.7889300Z 2025-09-07T07:14:17.7889415Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7889793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7890149Z return mod(**inputs) 2025-09-07T07:14:17.7890538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7890961Z outputs = self.model( 2025-09-07T07:14:17.7891351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7891787Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7892230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7892667Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7893059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7893495Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7893898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7894335Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7894767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.7895187Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.7895332Z 2025-09-07T07:14:17.7895481Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7895849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7896208Z return mod(**inputs) 2025-09-07T07:14:17.7896597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7897006Z outputs = self.model( 2025-09-07T07:14:17.7897395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7897815Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7898225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7898644Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7899013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7899392Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7899815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7900249Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7900687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.7901140Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.7901309Z 2025-09-07T07:14:17.7901403Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7901651Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7901908Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7902133Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7902382Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7902777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7903132Z return mod(**inputs) 2025-09-07T07:14:17.7903533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7903949Z outputs = self.model( 2025-09-07T07:14:17.7904348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7904771Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7905279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7905801Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7906222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7906639Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7907091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7907538Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7907976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7908430Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7908921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.7909447Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.7909648Z 2025-09-07T07:14:17.7909771Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7910155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7910509Z return mod(**inputs) 2025-09-07T07:14:17.7910943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7911373Z outputs = self.model( 2025-09-07T07:14:17.7911795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7912220Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7912642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7913080Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7913469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7913842Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7914247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7914670Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7915090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7915524Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7915977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.7916455Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.7916627Z 2025-09-07T07:14:17.7916737Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7917111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7917474Z return mod(**inputs) 2025-09-07T07:14:17.7917846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7918245Z outputs = self.model( 2025-09-07T07:14:17.7918627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7919033Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7919421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7919964Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7920330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7920746Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7921158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7921572Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7921997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.7922408Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.7922552Z 2025-09-07T07:14:17.7922670Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7923042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7923373Z return mod(**inputs) 2025-09-07T07:14:17.7923760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7924173Z outputs = self.model( 2025-09-07T07:14:17.7924535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7924918Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7925297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7925751Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7926111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7926503Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7926901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.7927350Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.7927543Z 2025-09-07T07:14:17.7927655Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7928030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7928350Z return mod(**inputs) 2025-09-07T07:14:17.7928710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7929098Z outputs = self.model( 2025-09-07T07:14:17.7929458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7929850Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7930219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7930605Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7930944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7931302Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7931693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.7932161Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.7932549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.7932887Z return self.act(input) 2025-09-07T07:14:17.7932997Z 2025-09-07T07:14:17.7933107Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7933460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7933769Z return mod(**inputs) 2025-09-07T07:14:17.7934129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7934548Z outputs = self.model( 2025-09-07T07:14:17.7934919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7935314Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7935705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7936099Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7936458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7936839Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7937258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-09-07T07:14:17.7937672Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.7937816Z 2025-09-07T07:14:17.7937921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7938290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7938620Z return mod(**inputs) 2025-09-07T07:14:17.7939000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7939414Z outputs = self.model( 2025-09-07T07:14:17.7939819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7940215Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7940622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7941016Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7941371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7941751Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7942167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7942588Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7943012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.7943495Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.7943704Z 2025-09-07T07:14:17.7943821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7944196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7944523Z return mod(**inputs) 2025-09-07T07:14:17.7944917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7945616Z outputs = self.model( 2025-09-07T07:14:17.7946203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7946784Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7947346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7947852Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7948240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7948705Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7949177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7949651Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7950200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.7950719Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.7950915Z 2025-09-07T07:14:17.7951031Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7951489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7951891Z return mod(**inputs) 2025-09-07T07:14:17.7952340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7952777Z outputs = self.model( 2025-09-07T07:14:17.7953225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7953761Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7954227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7954668Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7955119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7955565Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7956042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7968887Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7969520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.7969997Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.7970153Z 2025-09-07T07:14:17.7970251Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7970475Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7970695Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7970904Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.7971157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7971535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7971872Z return mod(**inputs) 2025-09-07T07:14:17.7972263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7972669Z outputs = self.model( 2025-09-07T07:14:17.7973057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7973463Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7973862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7974267Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7974629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7975006Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7975405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7975855Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7976283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7976712Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7977180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.7977708Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.7977915Z 2025-09-07T07:14:17.7978034Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7978429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7978794Z return mod(**inputs) 2025-09-07T07:14:17.7979170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7979573Z outputs = self.model( 2025-09-07T07:14:17.7979953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7980359Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7980750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7981151Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7981508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7981884Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7982297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7982711Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7983130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.7983555Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.7984028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.7984513Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.7984690Z 2025-09-07T07:14:17.7984798Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7985169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7985502Z return mod(**inputs) 2025-09-07T07:14:17.7985986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7986386Z outputs = self.model( 2025-09-07T07:14:17.7986771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7987198Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7987618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7988045Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7988421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7988828Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7989237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.7989677Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.7990109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.7990574Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.7990734Z 2025-09-07T07:14:17.7990850Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7991243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7991597Z return mod(**inputs) 2025-09-07T07:14:17.7991994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7992418Z outputs = self.model( 2025-09-07T07:14:17.7992825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7993250Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7993711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7994127Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7994505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.7994896Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.7995323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.7995793Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.7995991Z 2025-09-07T07:14:17.7996106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.7996496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.7996848Z return mod(**inputs) 2025-09-07T07:14:17.7997253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.7997669Z outputs = self.model( 2025-09-07T07:14:17.7998070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.7998511Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.7998987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.7999417Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.7999814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8000210Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8000639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.8001112Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8001526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8001900Z return self.act(input) 2025-09-07T07:14:17.8002027Z 2025-09-07T07:14:17.8002140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8002536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8002871Z return mod(**inputs) 2025-09-07T07:14:17.8003248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8003638Z outputs = self.model( 2025-09-07T07:14:17.8004014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8004421Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8004813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8005210Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8005583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8005955Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8006384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-09-07T07:14:17.8006821Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8006981Z 2025-09-07T07:14:17.8007093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8007481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8007835Z return mod(**inputs) 2025-09-07T07:14:17.8008234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8008647Z outputs = self.model( 2025-09-07T07:14:17.8009022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8009428Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8009887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8010273Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8010625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8010987Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8011390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8011808Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8012219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8012701Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8012921Z 2025-09-07T07:14:17.8013027Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8013396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8013763Z return mod(**inputs) 2025-09-07T07:14:17.8014144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8014545Z outputs = self.model( 2025-09-07T07:14:17.8014924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8015333Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8015729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8016162Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8016544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8016950Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8017383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8017824Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8018273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8018718Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8018866Z 2025-09-07T07:14:17.8018989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8019395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8019918Z return mod(**inputs) 2025-09-07T07:14:17.8020322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8020833Z outputs = self.model( 2025-09-07T07:14:17.8021250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8021668Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8022088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8022511Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8022892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8023290Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8023747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8024205Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8024665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8025123Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8025282Z 2025-09-07T07:14:17.8025382Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8025620Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8025929Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8026164Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8026426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8026826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8027194Z return mod(**inputs) 2025-09-07T07:14:17.8027609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8028037Z outputs = self.model( 2025-09-07T07:14:17.8028451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8028893Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8029380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8029809Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8030213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8030601Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8031033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8031486Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8031937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8032393Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8032873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8033373Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8033570Z 2025-09-07T07:14:17.8033681Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8034053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8034380Z return mod(**inputs) 2025-09-07T07:14:17.8034759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8035165Z outputs = self.model( 2025-09-07T07:14:17.8035550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8035981Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8036374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8036801Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8037180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8037577Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8038007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8038446Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8038862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8039309Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8039767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8040232Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8040396Z 2025-09-07T07:14:17.8040503Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8040874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8041204Z return mod(**inputs) 2025-09-07T07:14:17.8041587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8041987Z outputs = self.model( 2025-09-07T07:14:17.8042363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8042770Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8043164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8043557Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8043937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8044310Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8044741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8045152Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8045579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8046001Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8046150Z 2025-09-07T07:14:17.8046275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8046670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8047023Z return mod(**inputs) 2025-09-07T07:14:17.8047431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8047861Z outputs = self.model( 2025-09-07T07:14:17.8048271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8048685Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8049080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8049499Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8049882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8050281Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8050739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.8051216Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8051409Z 2025-09-07T07:14:17.8051526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8051917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8052266Z return mod(**inputs) 2025-09-07T07:14:17.8052670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8053101Z outputs = self.model( 2025-09-07T07:14:17.8053514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8053975Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8054396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8054813Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8055194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8055587Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8056020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.8056491Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8056914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8057289Z return self.act(input) 2025-09-07T07:14:17.8057411Z 2025-09-07T07:14:17.8057534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8057925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8058273Z return mod(**inputs) 2025-09-07T07:14:17.8058685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8059119Z outputs = self.model( 2025-09-07T07:14:17.8059544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8060000Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8060440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8060868Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8061242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8061635Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8062058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-09-07T07:14:17.8062495Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8062652Z 2025-09-07T07:14:17.8062765Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8063157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8063506Z return mod(**inputs) 2025-09-07T07:14:17.8063919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8064347Z outputs = self.model( 2025-09-07T07:14:17.8064767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8065195Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8065614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8066162Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8066553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8066984Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8067422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8067877Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8068337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8068872Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8069132Z 2025-09-07T07:14:17.8069258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8069665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8070031Z return mod(**inputs) 2025-09-07T07:14:17.8070465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8070920Z outputs = self.model( 2025-09-07T07:14:17.8071337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8071780Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8072218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8072657Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8073055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8073469Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8073912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8074381Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8074866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8075321Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8075471Z 2025-09-07T07:14:17.8075613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8076017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8076385Z return mod(**inputs) 2025-09-07T07:14:17.8076801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8077246Z outputs = self.model( 2025-09-07T07:14:17.8077656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8078111Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8078535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8078964Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8079346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8079730Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8080160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8080609Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8081054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8081497Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8081675Z 2025-09-07T07:14:17.8081766Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8082001Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8082231Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8082457Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8082707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8083106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8083454Z return mod(**inputs) 2025-09-07T07:14:17.8083846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8084266Z outputs = self.model( 2025-09-07T07:14:17.8084672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8085122Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8085548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8085988Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8086358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8086757Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8087171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8087594Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8088007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8088428Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8088889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8089383Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8089571Z 2025-09-07T07:14:17.8089685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8090075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8090403Z return mod(**inputs) 2025-09-07T07:14:17.8090795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8091201Z outputs = self.model( 2025-09-07T07:14:17.8091582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8091982Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8092378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8092779Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8093135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8093510Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8093934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8094386Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8094831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8095254Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8095701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8096170Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8096365Z 2025-09-07T07:14:17.8096470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8096837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8097170Z return mod(**inputs) 2025-09-07T07:14:17.8097539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8097942Z outputs = self.model( 2025-09-07T07:14:17.8098325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8098733Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8099129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8099542Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8099898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8100271Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8100727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-09-07T07:14:17.8101151Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:14:17.8101568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8101981Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8102121Z 2025-09-07T07:14:17.8102238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8102624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8102971Z return mod(**inputs) 2025-09-07T07:14:17.8103389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8103811Z outputs = self.model( 2025-09-07T07:14:17.8104213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8104651Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8105081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8105537Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8106007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8106428Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8106860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.8107327Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8107508Z 2025-09-07T07:14:17.8107617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8107981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8108308Z return mod(**inputs) 2025-09-07T07:14:17.8108672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8109069Z outputs = self.model( 2025-09-07T07:14:17.8109443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8109841Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8110228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8110614Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8110960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8111347Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8111741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-09-07T07:14:17.8112173Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8112569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8112910Z return self.act(input) 2025-09-07T07:14:17.8113019Z 2025-09-07T07:14:17.8113131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8113490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8113810Z return mod(**inputs) 2025-09-07T07:14:17.8114211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8114616Z outputs = self.model( 2025-09-07T07:14:17.8114994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-09-07T07:14:17.8115384Z encoder_outputs = self.encoder( 2025-09-07T07:14:17.8115763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-09-07T07:14:17.8116152Z layer_outputs = encoder_layer( 2025-09-07T07:14:17.8116503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8116866Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8117254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-09-07T07:14:17.8117656Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8117801Z 2025-09-07T07:14:17.8117905Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8118269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8118589Z return mod(**inputs) 2025-09-07T07:14:17.8118972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8119361Z outputs = self.model( 2025-09-07T07:14:17.8119886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8120287Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8120671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8121071Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8121429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8121799Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8122209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8122619Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8123035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8123499Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8123703Z 2025-09-07T07:14:17.8123817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8124176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8124496Z return mod(**inputs) 2025-09-07T07:14:17.8124874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8125312Z outputs = self.model( 2025-09-07T07:14:17.8125687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8126094Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8126487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8126877Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8127226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8127589Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8127974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8128422Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8128836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8129235Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8129368Z 2025-09-07T07:14:17.8129480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8129832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8130157Z return mod(**inputs) 2025-09-07T07:14:17.8130525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8130915Z outputs = self.model( 2025-09-07T07:14:17.8131279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8131684Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8132086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8132493Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8132847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8133215Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8133638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8134076Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8134503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8134925Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8135073Z 2025-09-07T07:14:17.8135161Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8135394Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8135618Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8135838Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8136093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8136459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8136792Z return mod(**inputs) 2025-09-07T07:14:17.8137169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8137573Z outputs = self.model( 2025-09-07T07:14:17.8137941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8138382Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8138777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8139179Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8139527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8139948Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8140355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8140794Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8141260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8141717Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8142202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8142723Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8142946Z 2025-09-07T07:14:17.8143066Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8143464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8143811Z return mod(**inputs) 2025-09-07T07:14:17.8144224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8144655Z outputs = self.model( 2025-09-07T07:14:17.8145065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8145483Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8145979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8146464Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8146862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8147273Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8147715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8148154Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8148606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8149044Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8149495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8149967Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8150143Z 2025-09-07T07:14:17.8150252Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8150625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8150964Z return mod(**inputs) 2025-09-07T07:14:17.8151340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8151747Z outputs = self.model( 2025-09-07T07:14:17.8152142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8152540Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8152943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8153342Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8153704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8154082Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8154494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8154942Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8155359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8155773Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8155919Z 2025-09-07T07:14:17.8156024Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8156392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8156715Z return mod(**inputs) 2025-09-07T07:14:17.8157091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8157485Z outputs = self.model( 2025-09-07T07:14:17.8157886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8158291Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8158678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8159077Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8159431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8159801Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8160202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8160631Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8161064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8161543Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8161752Z 2025-09-07T07:14:17.8161866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8162234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8162559Z return mod(**inputs) 2025-09-07T07:14:17.8162968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8163394Z outputs = self.model( 2025-09-07T07:14:17.8163780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8164174Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8164574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8164984Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8165339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8165715Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8166116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8166555Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8166993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8167404Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8167550Z 2025-09-07T07:14:17.8167921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8168279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8168608Z return mod(**inputs) 2025-09-07T07:14:17.8168980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8169401Z outputs = self.model( 2025-09-07T07:14:17.8169758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8170153Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8170543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8170947Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8171303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8171662Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8172069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8172511Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8172928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8173332Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8173472Z 2025-09-07T07:14:17.8173554Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8173771Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8173980Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8174187Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8174412Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8174770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8175091Z return mod(**inputs) 2025-09-07T07:14:17.8175455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8175835Z outputs = self.model( 2025-09-07T07:14:17.8176200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8176586Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8176992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8177386Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8177741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8178103Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8178497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8178973Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8179406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8179836Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8180309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8180833Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8181029Z 2025-09-07T07:14:17.8181148Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8181542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8181898Z return mod(**inputs) 2025-09-07T07:14:17.8182309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8182733Z outputs = self.model( 2025-09-07T07:14:17.8183145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8183602Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8184034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8184470Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8184854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8185250Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8185771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8186260Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8186733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8186872Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8187220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8187342Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8187346Z 2025-09-07T07:14:17.8187466Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8187685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8187760Z return mod(**inputs) 2025-09-07T07:14:17.8188051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8188123Z outputs = self.model( 2025-09-07T07:14:17.8188410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8188493Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8188771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8188859Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8189094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8189209Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8189510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8189634Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8189915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8190005Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8190011Z 2025-09-07T07:14:17.8190130Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8190345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8190426Z return mod(**inputs) 2025-09-07T07:14:17.8190710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8190782Z outputs = self.model( 2025-09-07T07:14:17.8191073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8191152Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8191440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8191517Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8191762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8191848Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8192124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8192290Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8192293Z 2025-09-07T07:14:17.8192402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8192622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8192694Z return mod(**inputs) 2025-09-07T07:14:17.8192975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8193055Z outputs = self.model( 2025-09-07T07:14:17.8193333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8193437Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8193720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8193799Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8194043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8194128Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8194411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8194540Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8194774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8194849Z return self.act(input) 2025-09-07T07:14:17.8194852Z 2025-09-07T07:14:17.8194967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8195192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8195266Z return mod(**inputs) 2025-09-07T07:14:17.8195552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8195632Z outputs = self.model( 2025-09-07T07:14:17.8195919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8196004Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8196286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8196369Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8196596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8196677Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8196958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:14:17.8197043Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8197047Z 2025-09-07T07:14:17.8197157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8197356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8197432Z return mod(**inputs) 2025-09-07T07:14:17.8197691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8197758Z outputs = self.model( 2025-09-07T07:14:17.8198023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8198094Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8198361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8198450Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8198669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8198754Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8199011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8199124Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8199379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8199534Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8199538Z 2025-09-07T07:14:17.8199663Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8199868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8199945Z return mod(**inputs) 2025-09-07T07:14:17.8200211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8200286Z outputs = self.model( 2025-09-07T07:14:17.8200561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8200638Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8200910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8200982Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8201213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8201295Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8201564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8201669Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8201931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8202041Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8202045Z 2025-09-07T07:14:17.8202170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8202382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8202449Z return mod(**inputs) 2025-09-07T07:14:17.8202716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8202796Z outputs = self.model( 2025-09-07T07:14:17.8203060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8203142Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8203406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8203478Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8203714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8203796Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8204067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8204169Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8204439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8204532Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8204536Z 2025-09-07T07:14:17.8204639Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8204730Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8204810Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8204896Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8205004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8205209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8205285Z return mod(**inputs) 2025-09-07T07:14:17.8205552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8205629Z outputs = self.model( 2025-09-07T07:14:17.8205895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8205988Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8206264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8206337Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8206568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8206653Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8206915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8207020Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8207281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8207386Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8207689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8207832Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8207837Z 2025-09-07T07:14:17.8207942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8208163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8208238Z return mod(**inputs) 2025-09-07T07:14:17.8208530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8208604Z outputs = self.model( 2025-09-07T07:14:17.8208859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8208931Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8209195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8209266Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8209488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8209566Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8209843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8209948Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8210229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8210340Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8210659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8210785Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8210789Z 2025-09-07T07:14:17.8210919Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8211136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8211215Z return mod(**inputs) 2025-09-07T07:14:17.8211494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8211573Z outputs = self.model( 2025-09-07T07:14:17.8211849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8211930Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8212192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8212291Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8212516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8212596Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8212868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8212975Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8213251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8213346Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8213350Z 2025-09-07T07:14:17.8213460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8213681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8213752Z return mod(**inputs) 2025-09-07T07:14:17.8214039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8214113Z outputs = self.model( 2025-09-07T07:14:17.8214395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8214479Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8214791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8214896Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8215137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8215219Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8215504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8215622Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8215909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8216072Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8216076Z 2025-09-07T07:14:17.8216194Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8216408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8216480Z return mod(**inputs) 2025-09-07T07:14:17.8216767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8216840Z outputs = self.model( 2025-09-07T07:14:17.8217123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8217202Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8217483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8217589Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8217831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8217924Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8218211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8218326Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8218629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8218717Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8218739Z 2025-09-07T07:14:17.8218858Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8219072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8219151Z return mod(**inputs) 2025-09-07T07:14:17.8219436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8219502Z outputs = self.model( 2025-09-07T07:14:17.8219908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8219987Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8220266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8220344Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8220580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8220677Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8220954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8221078Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8221405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8221501Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8221513Z 2025-09-07T07:14:17.8221622Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8221710Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8221799Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8221880Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8221991Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8222215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8222284Z return mod(**inputs) 2025-09-07T07:14:17.8222584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8222656Z outputs = self.model( 2025-09-07T07:14:17.8222963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8223042Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8223343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8223430Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8223667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8223760Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8224038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8224195Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8224479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8224583Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8224908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8225055Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8225058Z 2025-09-07T07:14:17.8225177Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8225401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8225496Z return mod(**inputs) 2025-09-07T07:14:17.8225851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8225936Z outputs = self.model( 2025-09-07T07:14:17.8226250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8226331Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8226688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8226781Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8227023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8227115Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8227392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8227496Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8227753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8227848Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8228159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8228265Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8228269Z 2025-09-07T07:14:17.8228397Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8228591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8228657Z return mod(**inputs) 2025-09-07T07:14:17.8228924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8228994Z outputs = self.model( 2025-09-07T07:14:17.8229260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8229335Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8229590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8229670Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8229892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8229979Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8230241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8230354Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8230615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8230699Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8230722Z 2025-09-07T07:14:17.8230836Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8231037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8231114Z return mod(**inputs) 2025-09-07T07:14:17.8231375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8231445Z outputs = self.model( 2025-09-07T07:14:17.8231716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8231791Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8232059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8232150Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8232382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8232464Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8232741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8232871Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8232875Z 2025-09-07T07:14:17.8232981Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8233193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8233257Z return mod(**inputs) 2025-09-07T07:14:17.8233505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8233581Z outputs = self.model( 2025-09-07T07:14:17.8233829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8233907Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8234156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8235056Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8235321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8235398Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8235654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8235770Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8235983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8236054Z return self.act(input) 2025-09-07T07:14:17.8236059Z 2025-09-07T07:14:17.8236161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8236356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8236420Z return mod(**inputs) 2025-09-07T07:14:17.8236686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8236754Z outputs = self.model( 2025-09-07T07:14:17.8237008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8237098Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8237347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8237424Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8237638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8237733Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8237990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:14:17.8238071Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8238075Z 2025-09-07T07:14:17.8238182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8238375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8238444Z return mod(**inputs) 2025-09-07T07:14:17.8238696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8238781Z outputs = self.model( 2025-09-07T07:14:17.8239044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8239115Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8239379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8239447Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8239665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8239747Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8239999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8240104Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8240370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8240535Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8240540Z 2025-09-07T07:14:17.8240644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8240845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8240920Z return mod(**inputs) 2025-09-07T07:14:17.8241206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8241296Z outputs = self.model( 2025-09-07T07:14:17.8241562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8241635Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8241907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8241981Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8242211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8242292Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8242564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8242667Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8242930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8243019Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8243023Z 2025-09-07T07:14:17.8243128Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8243338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8243408Z return mod(**inputs) 2025-09-07T07:14:17.8243672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8243768Z outputs = self.model( 2025-09-07T07:14:17.8244032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8244113Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8244377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8244452Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8244682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8244760Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8245032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8245150Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8245418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8245509Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8245513Z 2025-09-07T07:14:17.8245596Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8245685Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8245763Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8245849Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8245953Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8246153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8246227Z return mod(**inputs) 2025-09-07T07:14:17.8246489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8246566Z outputs = self.model( 2025-09-07T07:14:17.8246825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8246908Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8247194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8247268Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8247518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8247601Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8247861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8247969Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8248231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8248340Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8248636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8248778Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8248781Z 2025-09-07T07:14:17.8248883Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8249087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8249162Z return mod(**inputs) 2025-09-07T07:14:17.8249426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8249501Z outputs = self.model( 2025-09-07T07:14:17.8249765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8249857Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8250128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8250201Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8250429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8250509Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8250782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8250881Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8251141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8251262Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8251560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8251679Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8251683Z 2025-09-07T07:14:17.8251786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8251995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8252072Z return mod(**inputs) 2025-09-07T07:14:17.8252348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8252422Z outputs = self.model( 2025-09-07T07:14:17.8252690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8252772Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8253036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8253111Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8253346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8253440Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8253729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8253828Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8254085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8254175Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8254180Z 2025-09-07T07:14:17.8254284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8254491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8254561Z return mod(**inputs) 2025-09-07T07:14:17.8254832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8254901Z outputs = self.model( 2025-09-07T07:14:17.8255165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8255258Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8255513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8255589Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8255805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8255883Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8256146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8256274Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8256535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8256683Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8256686Z 2025-09-07T07:14:17.8256795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8256994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8257059Z return mod(**inputs) 2025-09-07T07:14:17.8257324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8257410Z outputs = self.model( 2025-09-07T07:14:17.8257677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8257752Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8258010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8258091Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8258311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8258397Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8258654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8258761Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8259026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8259106Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8259110Z 2025-09-07T07:14:17.8259221Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8259421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8259514Z return mod(**inputs) 2025-09-07T07:14:17.8259799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8259869Z outputs = self.model( 2025-09-07T07:14:17.8260144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8260222Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8260512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8260589Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8260826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8260921Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8261201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8261323Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8261603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8261695Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8261707Z 2025-09-07T07:14:17.8261792Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8261878Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8261972Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8262054Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8262172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8262404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8262474Z return mod(**inputs) 2025-09-07T07:14:17.8262766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8262837Z outputs = self.model( 2025-09-07T07:14:17.8263123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8263200Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8263477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8263563Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8263818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8263910Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8264186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8264301Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8264590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8264693Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8265017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8265159Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8265164Z 2025-09-07T07:14:17.8265282Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8265503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8265578Z return mod(**inputs) 2025-09-07T07:14:17.8265966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8266048Z outputs = self.model( 2025-09-07T07:14:17.8266376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8266521Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8266811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8266909Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8267127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8267214Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8267468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8267583Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8267837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8267932Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8268227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8268334Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8268338Z 2025-09-07T07:14:17.8268444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8268641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8268707Z return mod(**inputs) 2025-09-07T07:14:17.8268968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8269051Z outputs = self.model( 2025-09-07T07:14:17.8269319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8269392Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8269657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8269736Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8269964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8270051Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8270331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8270443Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8270704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8270787Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8270791Z 2025-09-07T07:14:17.8270904Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8271110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8271184Z return mod(**inputs) 2025-09-07T07:14:17.8271448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8271515Z outputs = self.model( 2025-09-07T07:14:17.8271793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8271869Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8272133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8272206Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8272468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8272546Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8272809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8272934Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8272938Z 2025-09-07T07:14:17.8273037Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8273237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8273301Z return mod(**inputs) 2025-09-07T07:14:17.8273549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8273625Z outputs = self.model( 2025-09-07T07:14:17.8273871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8273950Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8274199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8274268Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8274488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8274565Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8274820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8274936Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8275166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8275235Z return self.act(input) 2025-09-07T07:14:17.8275238Z 2025-09-07T07:14:17.8275339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8275538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8275603Z return mod(**inputs) 2025-09-07T07:14:17.8275858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8275922Z outputs = self.model( 2025-09-07T07:14:17.8276170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8276264Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8276516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8276596Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8276812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8276899Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8277157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:14:17.8277239Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8277242Z 2025-09-07T07:14:17.8277352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8277549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8277634Z return mod(**inputs) 2025-09-07T07:14:17.8277881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8277946Z outputs = self.model( 2025-09-07T07:14:17.8278201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8278271Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8278538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8278625Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8278841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8278925Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8279173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8279278Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8279540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8279703Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8279707Z 2025-09-07T07:14:17.8279814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8280020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8280099Z return mod(**inputs) 2025-09-07T07:14:17.8280383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8280462Z outputs = self.model( 2025-09-07T07:14:17.8280753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8280831Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8281122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8281216Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8281460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8281539Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8281806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8281917Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8282169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8282254Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8282274Z 2025-09-07T07:14:17.8282378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8282581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8282649Z return mod(**inputs) 2025-09-07T07:14:17.8282906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8282982Z outputs = self.model( 2025-09-07T07:14:17.8283236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8283316Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8283571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8283642Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8283867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8283945Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8284202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8284303Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8284582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8284671Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8284675Z 2025-09-07T07:14:17.8284769Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8284858Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8284947Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8285028Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8285130Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8285327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8285398Z return mod(**inputs) 2025-09-07T07:14:17.8285657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8285729Z outputs = self.model( 2025-09-07T07:14:17.8285988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8286060Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8286325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8286396Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8286620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8286698Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8286952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8287083Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8287335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8287438Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8287730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8287868Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8287872Z 2025-09-07T07:14:17.8287974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8288170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8288263Z return mod(**inputs) 2025-09-07T07:14:17.8288524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8288603Z outputs = self.model( 2025-09-07T07:14:17.8288863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8288939Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8289217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8289292Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8289524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8289605Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8289889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8289999Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8290277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8290392Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8290726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8290854Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8290858Z 2025-09-07T07:14:17.8290983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8291198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8291276Z return mod(**inputs) 2025-09-07T07:14:17.8291554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8291635Z outputs = self.model( 2025-09-07T07:14:17.8291913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8291999Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8292283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8292357Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8292588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8292668Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8292935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8293032Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8293295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8293385Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8293409Z 2025-09-07T07:14:17.8293514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8293724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8293792Z return mod(**inputs) 2025-09-07T07:14:17.8294065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8294135Z outputs = self.model( 2025-09-07T07:14:17.8294415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8294500Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8294777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8294896Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8295140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8295230Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8295528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8295647Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8295998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8296170Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8296174Z 2025-09-07T07:14:17.8296295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8296514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8296589Z return mod(**inputs) 2025-09-07T07:14:17.8296884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8296959Z outputs = self.model( 2025-09-07T07:14:17.8297272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8297354Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8297659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8297750Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8298000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8298097Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8298385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8298503Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8298811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8298898Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8298902Z 2025-09-07T07:14:17.8299021Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8299240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8299315Z return mod(**inputs) 2025-09-07T07:14:17.8299595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8299665Z outputs = self.model( 2025-09-07T07:14:17.8299950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8300031Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8300345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8300424Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8300667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8300761Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8301059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8301185Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8301468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8301582Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8301595Z 2025-09-07T07:14:17.8301684Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8301775Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8301868Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8301951Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8302072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8302293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8302367Z return mod(**inputs) 2025-09-07T07:14:17.8302664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8302739Z outputs = self.model( 2025-09-07T07:14:17.8303049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8303131Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8303432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8303521Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8303764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8303857Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8304162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8304300Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8304594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8304703Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8305038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8305187Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8305193Z 2025-09-07T07:14:17.8305317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8305541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8305614Z return mod(**inputs) 2025-09-07T07:14:17.8305985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8306068Z outputs = self.model( 2025-09-07T07:14:17.8306371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8306453Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8306744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8306834Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8307078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8307197Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8307483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8307609Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8307896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8308004Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8308346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8308505Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8308509Z 2025-09-07T07:14:17.8308631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8308855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8308928Z return mod(**inputs) 2025-09-07T07:14:17.8309239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8309313Z outputs = self.model( 2025-09-07T07:14:17.8309619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8309698Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8310001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8310086Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8310333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8310431Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8310729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8310854Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8311163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8311273Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8311278Z 2025-09-07T07:14:17.8311403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8311627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8311709Z return mod(**inputs) 2025-09-07T07:14:17.8311995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8312072Z outputs = self.model( 2025-09-07T07:14:17.8312373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8312452Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8312749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8312828Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8313083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8313170Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8313456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8313596Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8313601Z 2025-09-07T07:14:17.8313716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8313972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8314046Z return mod(**inputs) 2025-09-07T07:14:17.8314337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8314419Z outputs = self.model( 2025-09-07T07:14:17.8314705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8314790Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8315079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8315157Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8315427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8315515Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8315810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8315940Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8316186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8316263Z return self.act(input) 2025-09-07T07:14:17.8316269Z 2025-09-07T07:14:17.8316382Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8316611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8316684Z return mod(**inputs) 2025-09-07T07:14:17.8316977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8317051Z outputs = self.model( 2025-09-07T07:14:17.8317339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8317428Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8317734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8317822Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8318090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8318196Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8318472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:14:17.8318561Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8318566Z 2025-09-07T07:14:17.8318683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8318898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8318976Z return mod(**inputs) 2025-09-07T07:14:17.8319255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8319328Z outputs = self.model( 2025-09-07T07:14:17.8319778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8319866Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8320158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8320236Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8320475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8320570Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8320846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8321000Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8321279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8321454Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8321459Z 2025-09-07T07:14:17.8321570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8321786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8321866Z return mod(**inputs) 2025-09-07T07:14:17.8322145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8322251Z outputs = self.model( 2025-09-07T07:14:17.8322538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8322618Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8322914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8322991Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8323242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8323326Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8323621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8323728Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8324012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8324107Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8324111Z 2025-09-07T07:14:17.8324223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8324445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8324563Z return mod(**inputs) 2025-09-07T07:14:17.8324869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8324951Z outputs = self.model( 2025-09-07T07:14:17.8325229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8325313Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8325592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8325670Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8325918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8326003Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8326292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8326395Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8326680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8326773Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8326777Z 2025-09-07T07:14:17.8326865Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8326961Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8327046Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8327136Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8327247Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8327492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8327571Z return mod(**inputs) 2025-09-07T07:14:17.8327854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8327934Z outputs = self.model( 2025-09-07T07:14:17.8328215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8328293Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8328582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8328659Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8328928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8329010Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8329271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8329369Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8329635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8329744Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8330049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8330198Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8330203Z 2025-09-07T07:14:17.8330313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8330526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8330607Z return mod(**inputs) 2025-09-07T07:14:17.8330886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8330965Z outputs = self.model( 2025-09-07T07:14:17.8331261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8331358Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8331643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8331717Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8331947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8332027Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8332298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8332398Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8332675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8332777Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8333069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8333182Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8333185Z 2025-09-07T07:14:17.8333286Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8333490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8333557Z return mod(**inputs) 2025-09-07T07:14:17.8333816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8333909Z outputs = self.model( 2025-09-07T07:14:17.8334167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8334247Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8334505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8334576Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8334803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8334881Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8335162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8335261Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8335518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8335605Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8335608Z 2025-09-07T07:14:17.8335711Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8335916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8335981Z return mod(**inputs) 2025-09-07T07:14:17.8336241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8336307Z outputs = self.model( 2025-09-07T07:14:17.8336563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8336644Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8336909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8336990Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8337223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8337301Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8337575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8337684Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8337948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8338097Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8338102Z 2025-09-07T07:14:17.8338213Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8338415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8338479Z return mod(**inputs) 2025-09-07T07:14:17.8338750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8338817Z outputs = self.model( 2025-09-07T07:14:17.8339086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8339160Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8339423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8339503Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8339752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8339838Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8340155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8340275Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8340565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8340657Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8340661Z 2025-09-07T07:14:17.8340783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8341003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8341082Z return mod(**inputs) 2025-09-07T07:14:17.8341366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8341465Z outputs = self.model( 2025-09-07T07:14:17.8341761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8341841Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8342133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8342213Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8342458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8342551Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8342834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8342961Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8343245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8343353Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8343357Z 2025-09-07T07:14:17.8343447Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8343536Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8343648Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8343734Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8343891Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8344111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8344182Z return mod(**inputs) 2025-09-07T07:14:17.8344470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8344545Z outputs = self.model( 2025-09-07T07:14:17.8344836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8344918Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8345200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8345285Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8345535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8345629Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8346140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8346265Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8346562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8346672Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8347033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8347180Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8347184Z 2025-09-07T07:14:17.8347307Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8347538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8347610Z return mod(**inputs) 2025-09-07T07:14:17.8347900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8347974Z outputs = self.model( 2025-09-07T07:14:17.8348262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8348361Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8348636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8348724Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8348961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8349052Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8349329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8349448Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8349728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8349831Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8350155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8350273Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8350276Z 2025-09-07T07:14:17.8350392Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8350624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8350697Z return mod(**inputs) 2025-09-07T07:14:17.8351003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8351076Z outputs = self.model( 2025-09-07T07:14:17.8351362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8351439Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8351722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8351810Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8352054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8352147Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8352425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8352549Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8352829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8352917Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8352920Z 2025-09-07T07:14:17.8353040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8353261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8353339Z return mod(**inputs) 2025-09-07T07:14:17.8353635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8353708Z outputs = self.model( 2025-09-07T07:14:17.8353996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8354074Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8354358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8354437Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8354681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8354764Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8355069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8355195Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8355198Z 2025-09-07T07:14:17.8355301Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8355505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8355569Z return mod(**inputs) 2025-09-07T07:14:17.8355826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8355899Z outputs = self.model( 2025-09-07T07:14:17.8356153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8356230Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8356489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8356559Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8356785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8356862Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8357138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8357275Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8357492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8357562Z return self.act(input) 2025-09-07T07:14:17.8357565Z 2025-09-07T07:14:17.8357665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8357867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8357937Z return mod(**inputs) 2025-09-07T07:14:17.8358211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8358278Z outputs = self.model( 2025-09-07T07:14:17.8358526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8358603Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8358854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8358931Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8359143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8359226Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8359484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:14:17.8359567Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8359590Z 2025-09-07T07:14:17.8359702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8359903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8359978Z return mod(**inputs) 2025-09-07T07:14:17.8360245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8360314Z outputs = self.model( 2025-09-07T07:14:17.8360585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8360658Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8360930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8361022Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8361247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8361336Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8361605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8361712Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8361976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8362138Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8362142Z 2025-09-07T07:14:17.8362248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8362451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8362524Z return mod(**inputs) 2025-09-07T07:14:17.8362792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8362867Z outputs = self.model( 2025-09-07T07:14:17.8363157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8363230Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8363511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8363584Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8363813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8363891Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8364154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8364253Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8364507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8364594Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8364597Z 2025-09-07T07:14:17.8364698Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8364903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8364969Z return mod(**inputs) 2025-09-07T07:14:17.8365227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8365302Z outputs = self.model( 2025-09-07T07:14:17.8365564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8365648Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8365931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8366003Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8366238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8366316Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8366591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8366690Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8366959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8367065Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8367069Z 2025-09-07T07:14:17.8367154Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8367248Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8367331Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8367420Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8367530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8367744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8367825Z return mod(**inputs) 2025-09-07T07:14:17.8368105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8368186Z outputs = self.model( 2025-09-07T07:14:17.8368473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8368561Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8368829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8368904Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8369137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8369217Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8369545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8369664Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8369929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8370035Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8370330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8370478Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8370483Z 2025-09-07T07:14:17.8370586Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8370790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8370869Z return mod(**inputs) 2025-09-07T07:14:17.8371150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8371234Z outputs = self.model( 2025-09-07T07:14:17.8371515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8371603Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8371882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8371962Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8372215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8372311Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8372573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8372668Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8372928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8373033Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8373329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8373468Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8373471Z 2025-09-07T07:14:17.8373576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8373783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8373851Z return mod(**inputs) 2025-09-07T07:14:17.8374114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8374189Z outputs = self.model( 2025-09-07T07:14:17.8374459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8374539Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8374802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8374875Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8375107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8375186Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8375455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-09-07T07:14:17.8375553Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:17.8375827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8375932Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8375936Z 2025-09-07T07:14:17.8376041Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8376253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8376320Z return mod(**inputs) 2025-09-07T07:14:17.8376592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8376663Z outputs = self.model( 2025-09-07T07:14:17.8376929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8377014Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8377284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8377366Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8377590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8377669Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8377942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8378052Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8378328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-09-07T07:14:17.8378498Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:17.8378502Z 2025-09-07T07:14:17.8378609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8378816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8378884Z return mod(**inputs) 2025-09-07T07:14:17.8379153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8379221Z outputs = self.model( 2025-09-07T07:14:17.8379505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8379582Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8379878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8379964Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8380199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8380290Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8380566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8380685Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8380984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-09-07T07:14:17.8381073Z key_states = self.k_proj(current_states) 2025-09-07T07:14:17.8381077Z 2025-09-07T07:14:17.8381198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8381420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8381501Z return mod(**inputs) 2025-09-07T07:14:17.8381796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8381869Z outputs = self.model( 2025-09-07T07:14:17.8382187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8382270Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8382583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8382664Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8382911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8383010Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8383295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8383423Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8383707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-09-07T07:14:17.8383811Z value_states = self.v_proj(current_states) 2025-09-07T07:14:17.8383815Z 2025-09-07T07:14:17.8383906Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8383996Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8384090Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8384174Z cudagraph partition due to non gpu ops 2025-09-07T07:14:17.8384297Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8384519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8384592Z return mod(**inputs) 2025-09-07T07:14:17.8384888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8384981Z outputs = self.model( 2025-09-07T07:14:17.8385296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8385379Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8385683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8385851Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8386106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8386204Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8386499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8386656Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8386946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8387051Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8387375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:17.8387521Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:17.8387526Z 2025-09-07T07:14:17.8387644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8387859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8387929Z return mod(**inputs) 2025-09-07T07:14:17.8388236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8388311Z outputs = self.model( 2025-09-07T07:14:17.8388609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8388689Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8389009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8389097Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8389359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8389453Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8389731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8389855Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8390138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-09-07T07:14:17.8390246Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:17.8390568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:17.8390684Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:17.8390687Z 2025-09-07T07:14:17.8390806Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8391024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8391095Z return mod(**inputs) 2025-09-07T07:14:17.8391405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8391476Z outputs = self.model( 2025-09-07T07:14:17.8391766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8391865Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8392156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8392232Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8392470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8392564Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8392840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-09-07T07:14:17.8392963Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:14:17.8393237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-09-07T07:14:17.8393353Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:17.8393357Z 2025-09-07T07:14:17.8393476Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8393688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8393765Z return mod(**inputs) 2025-09-07T07:14:17.8394063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8394135Z outputs = self.model( 2025-09-07T07:14:17.8394441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8394518Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8394817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8394896Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8395139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8395225Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8395501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8395655Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8395662Z 2025-09-07T07:14:17.8395774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8396016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8396088Z return mod(**inputs) 2025-09-07T07:14:17.8396366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8396446Z outputs = self.model( 2025-09-07T07:14:17.8396727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8396814Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8397098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8397182Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8397426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8397510Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8397795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-09-07T07:14:17.8397923Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:17.8398159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:17.8398235Z return self.act(input) 2025-09-07T07:14:17.8398238Z 2025-09-07T07:14:17.8398349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8398590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8398661Z return mod(**inputs) 2025-09-07T07:14:17.8398946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-09-07T07:14:17.8399019Z outputs = self.model( 2025-09-07T07:14:17.8399296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-09-07T07:14:17.8399382Z decoder_outputs = self.decoder( 2025-09-07T07:14:17.8399657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-09-07T07:14:17.8399740Z layer_outputs = decoder_layer( 2025-09-07T07:14:17.8399996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:17.8400087Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:17.8400366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-09-07T07:14:17.8400457Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:17.8400461Z 2025-09-07T07:14:17.8400584Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8400800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8400879Z return mod(**inputs) 2025-09-07T07:14:17.8401160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1377, in forward 2025-09-07T07:14:17.8401245Z lm_logits = self.lm_head(outputs[0]) 2025-09-07T07:14:17.8401249Z 2025-09-07T07:14:17.8401369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:17.8401581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:17.8401660Z return mod(**inputs) 2025-09-07T07:14:17.8401938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1383, in forward 2025-09-07T07:14:17.8402148Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:14:17.8402153Z 2025-09-07T07:14:30.0300989Z Compilation time (from dynamo_timed): 20.191677519 2025-09-07T07:14:30.0543977Z pass 2025-09-07T07:14:30.0544459Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:14:30.0545381Z TIMING: _recursive_pre_grad_passes:0.00947 _recursive_joint_graph_passes:0.46092 _recursive_post_grad_passes:0.10515 async_compile.wait:0.77373 code_gen:11.52402 inductor_compile:13.24083 backend_compile:17.22417 gc:0.00211 entire_frame_compile:20.19168 total_wall_time:20.19168 2025-09-07T07:14:30.0546567Z STATS: call_* op count: 517 | FakeTensorMode.__torch_dispatch__:17508 | FakeTensor.__torch_dispatch__:5831 | ProxyTorchDispatchMode.__torch_dispatch__:6406 2025-09-07T07:14:30.0547181Z Dynamo produced 1 graphs covering 517 ops with 0 graph breaks (0 unique) 2025-09-07T07:14:32.8664046Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:14:32.8665070Z import pynvml # type: ignore[import] 2025-09-07T07:14:35.6245644Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:14:35.6251415Z from pkg_resources import resource_filename 2025-09-07T07:14:36.2730136Z 2025-09-07T07:14:39.7146016Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:14:39.7146356Z loading model: 0it [00:03, ?it/s] 2025-09-07T07:14:39.7158183Z cpu eval PegasusForCausalLM 2025-09-07T07:14:40.1307620Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:14:40.3220343Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:14:40.4571350Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:14:48.1983565Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1983929Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1984164Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1984397Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1985022Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1985269Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1985513Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1985938Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1986191Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1987478Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1987861Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1988785Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.1989078Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.1989589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.1989969Z return mod(**inputs) 2025-09-07T07:14:48.1990422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.1990939Z outputs = self.model.decoder( 2025-09-07T07:14:48.1991412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.1991891Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.1992287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.1992694Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.1993313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.1993846Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.1994320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.1994862Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.1995130Z 2025-09-07T07:14:48.1995259Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.1995678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.1996052Z return mod(**inputs) 2025-09-07T07:14:48.1996473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.1996929Z outputs = self.model.decoder( 2025-09-07T07:14:48.1997366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.1997826Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.1998217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.1998640Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.1999083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.1999556Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2000022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2000540Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2000693Z 2025-09-07T07:14:48.2000810Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2001225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2001581Z return mod(**inputs) 2025-09-07T07:14:48.2001990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2002430Z outputs = self.model.decoder( 2025-09-07T07:14:48.2002855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2003325Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2003708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2004107Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2004537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2005002Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2005460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2005910Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2006065Z 2025-09-07T07:14:48.2006161Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2006387Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2006616Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2006847Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2007103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2007498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2007870Z return mod(**inputs) 2025-09-07T07:14:48.2008289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2008753Z outputs = self.model.decoder( 2025-09-07T07:14:48.2009202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2009630Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2010014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2010413Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2010858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2011323Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2011775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2012237Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2012740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2013277Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2013481Z 2025-09-07T07:14:48.2013605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2013992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2014342Z return mod(**inputs) 2025-09-07T07:14:48.2014770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2015214Z outputs = self.model.decoder( 2025-09-07T07:14:48.2015668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2016106Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2016499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2016906Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2017345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2017799Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2018261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2018745Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2019228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2020121Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2020313Z 2025-09-07T07:14:48.2020436Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2020852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2021216Z return mod(**inputs) 2025-09-07T07:14:48.2021652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2022110Z outputs = self.model.decoder( 2025-09-07T07:14:48.2022543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2022984Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2023385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2023797Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2024248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2024784Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2025289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2025818Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2025984Z 2025-09-07T07:14:48.2026115Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2026524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2026894Z return mod(**inputs) 2025-09-07T07:14:48.2027323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2027781Z outputs = self.model.decoder( 2025-09-07T07:14:48.2028228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2028673Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2029068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2029482Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2029934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2030428Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2030635Z 2025-09-07T07:14:48.2030750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2031153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2031554Z return mod(**inputs) 2025-09-07T07:14:48.2031977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2032420Z outputs = self.model.decoder( 2025-09-07T07:14:48.2032865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2033315Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2033707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2034116Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2034562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2035099Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2035538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2035923Z return self.act(input) 2025-09-07T07:14:48.2036048Z 2025-09-07T07:14:48.2036170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2036548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2036885Z return mod(**inputs) 2025-09-07T07:14:48.2037263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2037668Z outputs = self.model.decoder( 2025-09-07T07:14:48.2038055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2038455Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2038807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2039172Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2039568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2039976Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2040163Z 2025-09-07T07:14:48.2040274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2040657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2040986Z return mod(**inputs) 2025-09-07T07:14:48.2041361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2041769Z outputs = self.model.decoder( 2025-09-07T07:14:48.2042163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2042563Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2042914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2043336Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2043744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2044169Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2044590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2045059Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2045270Z 2025-09-07T07:14:48.2045376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2045741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2046069Z return mod(**inputs) 2025-09-07T07:14:48.2046477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2046867Z outputs = self.model.decoder( 2025-09-07T07:14:48.2047263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2047666Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2048019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2048393Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2048812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2049259Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2049681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2050087Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2050226Z 2025-09-07T07:14:48.2050337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2050699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2051027Z return mod(**inputs) 2025-09-07T07:14:48.2051408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2051806Z outputs = self.model.decoder( 2025-09-07T07:14:48.2052198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2052584Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2052933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2053301Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2053712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2054122Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2054551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2054985Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2055132Z 2025-09-07T07:14:48.2055223Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2055697Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2055906Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2056119Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2056377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2056742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2057059Z return mod(**inputs) 2025-09-07T07:14:48.2057435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2057830Z outputs = self.model.decoder( 2025-09-07T07:14:48.2058230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2058641Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2058991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2059362Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2059770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2060206Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2060630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2061087Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2061572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2062104Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2062306Z 2025-09-07T07:14:48.2062449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2062837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2063194Z return mod(**inputs) 2025-09-07T07:14:48.2063583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2064036Z outputs = self.model.decoder( 2025-09-07T07:14:48.2064465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2064890Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2065272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2065669Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2066231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2066713Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2067158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2067587Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2068033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2068512Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2068689Z 2025-09-07T07:14:48.2068795Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2069183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2069524Z return mod(**inputs) 2025-09-07T07:14:48.2069934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2070345Z outputs = self.model.decoder( 2025-09-07T07:14:48.2070740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2071151Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2071512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2071891Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2072300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2072727Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2073158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2073575Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2073715Z 2025-09-07T07:14:48.2073829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2074189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2074525Z return mod(**inputs) 2025-09-07T07:14:48.2074914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2075327Z outputs = self.model.decoder( 2025-09-07T07:14:48.2075755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2076158Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2076521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2076905Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2077342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2077831Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2078019Z 2025-09-07T07:14:48.2078132Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2078550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2078902Z return mod(**inputs) 2025-09-07T07:14:48.2079314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2079735Z outputs = self.model.decoder( 2025-09-07T07:14:48.2080163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2080593Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2080978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2081378Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2081788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2082245Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2082649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2083004Z return self.act(input) 2025-09-07T07:14:48.2083119Z 2025-09-07T07:14:48.2083235Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2083599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2083960Z return mod(**inputs) 2025-09-07T07:14:48.2084365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2084779Z outputs = self.model.decoder( 2025-09-07T07:14:48.2085189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2085627Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2085984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2086374Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2086810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2087244Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2087399Z 2025-09-07T07:14:48.2087520Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2087911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2088263Z return mod(**inputs) 2025-09-07T07:14:48.2088664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2089096Z outputs = self.model.decoder( 2025-09-07T07:14:48.2089515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2089941Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2090317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2090726Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2091168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2091627Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2092089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2092607Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2092829Z 2025-09-07T07:14:48.2092942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2093335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2093724Z return mod(**inputs) 2025-09-07T07:14:48.2094132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2094565Z outputs = self.model.decoder( 2025-09-07T07:14:48.2094980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2095410Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2095796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2096193Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2096651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2097137Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2097598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2098039Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2098185Z 2025-09-07T07:14:48.2098305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2098688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2099058Z return mod(**inputs) 2025-09-07T07:14:48.2099516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2099950Z outputs = self.model.decoder( 2025-09-07T07:14:48.2100373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2100792Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2101169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2101565Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2101997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2102476Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2102970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2103414Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2103571Z 2025-09-07T07:14:48.2103667Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2103903Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2104128Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2104354Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2104614Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2105020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2105374Z return mod(**inputs) 2025-09-07T07:14:48.2105917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2106381Z outputs = self.model.decoder( 2025-09-07T07:14:48.2106825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2107274Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2107650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2108045Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2108480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2108981Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2109424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2109885Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2110365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2110894Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2111093Z 2025-09-07T07:14:48.2111215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2111598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2111949Z return mod(**inputs) 2025-09-07T07:14:48.2112354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2112791Z outputs = self.model.decoder( 2025-09-07T07:14:48.2113215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2113644Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2114026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2114441Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2114897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2115362Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2115839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2116309Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2116797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2117270Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2117443Z 2025-09-07T07:14:48.2117559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2117945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2118311Z return mod(**inputs) 2025-09-07T07:14:48.2118734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2119182Z outputs = self.model.decoder( 2025-09-07T07:14:48.2119719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2120142Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2120508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2120889Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2121366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2121797Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2122231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2122653Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2122797Z 2025-09-07T07:14:48.2122912Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2123288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2123616Z return mod(**inputs) 2025-09-07T07:14:48.2124002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2124452Z outputs = self.model.decoder( 2025-09-07T07:14:48.2124848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2125245Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2125606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2125977Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2126386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2126838Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2127015Z 2025-09-07T07:14:48.2127121Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2127487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2127820Z return mod(**inputs) 2025-09-07T07:14:48.2128214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2128639Z outputs = self.model.decoder( 2025-09-07T07:14:48.2129065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2129529Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2129939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2130340Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2130754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2131220Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2131627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2131992Z return self.act(input) 2025-09-07T07:14:48.2132112Z 2025-09-07T07:14:48.2132233Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2132603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2132947Z return mod(**inputs) 2025-09-07T07:14:48.2133355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2133801Z outputs = self.model.decoder( 2025-09-07T07:14:48.2134229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2134667Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2135114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2135536Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2135994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2136496Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2136657Z 2025-09-07T07:14:48.2136773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2137167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2137532Z return mod(**inputs) 2025-09-07T07:14:48.2137938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2138376Z outputs = self.model.decoder( 2025-09-07T07:14:48.2138803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2139272Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2139647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2140045Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2140496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:14:48.2140949Z hidden_states = residual + hidden_states 2025-09-07T07:14:48.2141096Z 2025-09-07T07:14:48.2141222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2141616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2141966Z return mod(**inputs) 2025-09-07T07:14:48.2142389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2142844Z outputs = self.model.decoder( 2025-09-07T07:14:48.2143281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2143933Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2144325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2144746Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2145249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2145813Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2146290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2146819Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2147055Z 2025-09-07T07:14:48.2147175Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2147578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2147929Z return mod(**inputs) 2025-09-07T07:14:48.2148331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2148765Z outputs = self.model.decoder( 2025-09-07T07:14:48.2149194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2149631Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2150014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2150404Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2150852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2151326Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2151790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2152323Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2152457Z 2025-09-07T07:14:48.2152562Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2152924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2153257Z return mod(**inputs) 2025-09-07T07:14:48.2153635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2154030Z outputs = self.model.decoder( 2025-09-07T07:14:48.2154435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2154867Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2155218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2155586Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2156014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2156507Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2156971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2157413Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2157563Z 2025-09-07T07:14:48.2157658Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2157883Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2158111Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2158320Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2158552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2158903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2159262Z return mod(**inputs) 2025-09-07T07:14:48.2159683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2160141Z outputs = self.model.decoder( 2025-09-07T07:14:48.2160577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2161009Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2161386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2161755Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2162162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2162578Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2162996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2163421Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2163878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2164374Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2164562Z 2025-09-07T07:14:48.2164669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2165035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2165365Z return mod(**inputs) 2025-09-07T07:14:48.2165766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2166202Z outputs = self.model.decoder( 2025-09-07T07:14:48.2166629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2167035Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2167397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2167770Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2168199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2168659Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2169118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2169599Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2170078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2170559Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2170744Z 2025-09-07T07:14:48.2170860Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2171258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2171612Z return mod(**inputs) 2025-09-07T07:14:48.2172025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2172453Z outputs = self.model.decoder( 2025-09-07T07:14:48.2172876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2173309Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2173691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2174084Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2174525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2175008Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2175481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2175923Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2176072Z 2025-09-07T07:14:48.2176187Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2176576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2176927Z return mod(**inputs) 2025-09-07T07:14:48.2177335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2177769Z outputs = self.model.decoder( 2025-09-07T07:14:48.2178186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2178613Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2178996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2179393Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2179820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2180316Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2180512Z 2025-09-07T07:14:48.2180626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2181018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2181368Z return mod(**inputs) 2025-09-07T07:14:48.2181797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2182226Z outputs = self.model.decoder( 2025-09-07T07:14:48.2182650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2183077Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2183455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2183840Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2184294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2184811Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2185244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2185620Z return self.act(input) 2025-09-07T07:14:48.2185840Z 2025-09-07T07:14:48.2185963Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2186375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2186757Z return mod(**inputs) 2025-09-07T07:14:48.2187187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2187633Z outputs = self.model.decoder( 2025-09-07T07:14:48.2188078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2188523Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2188924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2189344Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2189798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2190266Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2190428Z 2025-09-07T07:14:48.2190580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2191003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2191359Z return mod(**inputs) 2025-09-07T07:14:48.2191780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2192222Z outputs = self.model.decoder( 2025-09-07T07:14:48.2192659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2193099Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2193485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2193887Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2194342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2194812Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2195346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2195835Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2196055Z 2025-09-07T07:14:48.2196168Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2196549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2196888Z return mod(**inputs) 2025-09-07T07:14:48.2197304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2197708Z outputs = self.model.decoder( 2025-09-07T07:14:48.2198115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2198540Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2198911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2199273Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2199688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2200147Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2200586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2201003Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2201141Z 2025-09-07T07:14:48.2201245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2201614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2201947Z return mod(**inputs) 2025-09-07T07:14:48.2202329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2202728Z outputs = self.model.decoder( 2025-09-07T07:14:48.2203128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2203532Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2204026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2204429Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2204864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2205338Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2205839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2206326Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2206473Z 2025-09-07T07:14:48.2206566Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2206785Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2207001Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2207214Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2207458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2207822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2208160Z return mod(**inputs) 2025-09-07T07:14:48.2208542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2208945Z outputs = self.model.decoder( 2025-09-07T07:14:48.2209354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2209790Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2210172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2210547Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2210957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2211382Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2211836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2212320Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2212811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2213348Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2213546Z 2025-09-07T07:14:48.2213661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2214058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2214417Z return mod(**inputs) 2025-09-07T07:14:48.2214826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2215290Z outputs = self.model.decoder( 2025-09-07T07:14:48.2215705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2216133Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2216516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2216916Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2217357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2217879Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2218342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2218806Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2219289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2219991Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2220180Z 2025-09-07T07:14:48.2220295Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2220750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2221114Z return mod(**inputs) 2025-09-07T07:14:48.2221554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2221984Z outputs = self.model.decoder( 2025-09-07T07:14:48.2222412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2222842Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2223223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2223615Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2224048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2224524Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2225009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2225468Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2225620Z 2025-09-07T07:14:48.2225783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2226205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2226572Z return mod(**inputs) 2025-09-07T07:14:48.2227003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2227440Z outputs = self.model.decoder( 2025-09-07T07:14:48.2227921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2228364Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2228758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2229161Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2229599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2230099Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2230301Z 2025-09-07T07:14:48.2230421Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2230821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2231219Z return mod(**inputs) 2025-09-07T07:14:48.2231630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2232079Z outputs = self.model.decoder( 2025-09-07T07:14:48.2232516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2232959Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2233352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2233761Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2234223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2234669Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2235081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2235457Z return self.act(input) 2025-09-07T07:14:48.2235580Z 2025-09-07T07:14:48.2235694Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2236099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2236454Z return mod(**inputs) 2025-09-07T07:14:48.2236859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2237259Z outputs = self.model.decoder( 2025-09-07T07:14:48.2237656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2238061Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2238437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2238831Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2239261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2239711Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2239864Z 2025-09-07T07:14:48.2239984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2240347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2240676Z return mod(**inputs) 2025-09-07T07:14:48.2241065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2241488Z outputs = self.model.decoder( 2025-09-07T07:14:48.2241909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2242338Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2242708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2243103Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2243528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:14:48.2243946Z hidden_states = residual + hidden_states 2025-09-07T07:14:48.2244085Z 2025-09-07T07:14:48.2244202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2244579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2244929Z return mod(**inputs) 2025-09-07T07:14:48.2245334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2245785Z outputs = self.model.decoder( 2025-09-07T07:14:48.2246197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2246624Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2246999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2247371Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2247770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2248185Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2248603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2249075Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2249283Z 2025-09-07T07:14:48.2249394Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2249755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2250072Z return mod(**inputs) 2025-09-07T07:14:48.2250447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2250841Z outputs = self.model.decoder( 2025-09-07T07:14:48.2251330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2251757Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2252099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2252459Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2252862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2253292Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2253716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2254135Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2254281Z 2025-09-07T07:14:48.2254390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2254762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2255095Z return mod(**inputs) 2025-09-07T07:14:48.2255487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2255928Z outputs = self.model.decoder( 2025-09-07T07:14:48.2256346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2256751Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2257113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2257502Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2257919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2258360Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2258797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2259212Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2259364Z 2025-09-07T07:14:48.2259451Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2259679Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2259897Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2260136Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2260379Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2260758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2261094Z return mod(**inputs) 2025-09-07T07:14:48.2261483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2261884Z outputs = self.model.decoder( 2025-09-07T07:14:48.2262381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2262793Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2263152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2263524Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2263962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2264434Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2264899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2265368Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2266166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2266744Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2266953Z 2025-09-07T07:14:48.2267081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2267481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2267846Z return mod(**inputs) 2025-09-07T07:14:48.2268239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2268655Z outputs = self.model.decoder( 2025-09-07T07:14:48.2269067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2269515Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2269896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2270303Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2270750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2271180Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2271613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2272052Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2272508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2274125Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2274298Z 2025-09-07T07:14:48.2274414Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2274815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2275175Z return mod(**inputs) 2025-09-07T07:14:48.2275566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2275975Z outputs = self.model.decoder( 2025-09-07T07:14:48.2276384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2276819Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2277182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2277557Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2277978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2278413Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2278841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2279310Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2279466Z 2025-09-07T07:14:48.2279579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2279970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2280295Z return mod(**inputs) 2025-09-07T07:14:48.2280681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2281098Z outputs = self.model.decoder( 2025-09-07T07:14:48.2281506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2281934Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2282289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2282683Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2283096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2283551Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2283728Z 2025-09-07T07:14:48.2283843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2284217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2284571Z return mod(**inputs) 2025-09-07T07:14:48.2284984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2285415Z outputs = self.model.decoder( 2025-09-07T07:14:48.2285833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2286259Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2286638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2287174Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2287691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2288299Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2288858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2289275Z return self.act(input) 2025-09-07T07:14:48.2289397Z 2025-09-07T07:14:48.2289523Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2289919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2290267Z return mod(**inputs) 2025-09-07T07:14:48.2290677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2291110Z outputs = self.model.decoder( 2025-09-07T07:14:48.2291537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2291986Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2292366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2292773Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2293211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2293665Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2293815Z 2025-09-07T07:14:48.2293928Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2294325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2294686Z return mod(**inputs) 2025-09-07T07:14:48.2295092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2295523Z outputs = self.model.decoder( 2025-09-07T07:14:48.2295944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2296378Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2296756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2297158Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2297625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2298129Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2298602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2299139Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2299365Z 2025-09-07T07:14:48.2299486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2299876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2300245Z return mod(**inputs) 2025-09-07T07:14:48.2300656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2301185Z outputs = self.model.decoder( 2025-09-07T07:14:48.2301617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2302130Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2302522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2302916Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2303351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2303809Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2304284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2304755Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2304902Z 2025-09-07T07:14:48.2305020Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2305409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2305825Z return mod(**inputs) 2025-09-07T07:14:48.2306252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2306682Z outputs = self.model.decoder( 2025-09-07T07:14:48.2307130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2307610Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2307985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2308394Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2308809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2309273Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2309728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2310193Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2310357Z 2025-09-07T07:14:48.2310449Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2310694Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2310936Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2311155Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2311417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2314457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2314845Z return mod(**inputs) 2025-09-07T07:14:48.2315244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2315697Z outputs = self.model.decoder( 2025-09-07T07:14:48.2316110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2316524Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2316877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2317257Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2317673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2318111Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2318573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2319012Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2319474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2320116Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2320306Z 2025-09-07T07:14:48.2320423Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2320787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2321123Z return mod(**inputs) 2025-09-07T07:14:48.2321512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2321928Z outputs = self.model.decoder( 2025-09-07T07:14:48.2322383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2322781Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2323145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2323518Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2323929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2324353Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2324787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2325275Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2325731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2326205Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2326368Z 2025-09-07T07:14:48.2326474Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2326859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2327211Z return mod(**inputs) 2025-09-07T07:14:48.2327621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2328054Z outputs = self.model.decoder( 2025-09-07T07:14:48.2328473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2328909Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2329293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2329767Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2330193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2330644Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2331125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2331579Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2331728Z 2025-09-07T07:14:48.2331850Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2332238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2332601Z return mod(**inputs) 2025-09-07T07:14:48.2333020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2333460Z outputs = self.model.decoder( 2025-09-07T07:14:48.2333892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2334325Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2334706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2335112Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2335552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2336060Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2336250Z 2025-09-07T07:14:48.2336367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2336764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2337115Z return mod(**inputs) 2025-09-07T07:14:48.2337564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2337996Z outputs = self.model.decoder( 2025-09-07T07:14:48.2338434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2338874Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2339266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2339692Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2340150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2340676Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2341108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2341519Z return self.act(input) 2025-09-07T07:14:48.2341640Z 2025-09-07T07:14:48.2341761Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2342160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2342530Z return mod(**inputs) 2025-09-07T07:14:48.2342949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2343399Z outputs = self.model.decoder( 2025-09-07T07:14:48.2343987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2344502Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2344904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2345308Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2345925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2346411Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2346576Z 2025-09-07T07:14:48.2346716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2347127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2347494Z return mod(**inputs) 2025-09-07T07:14:48.2347905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2348355Z outputs = self.model.decoder( 2025-09-07T07:14:48.2348790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2349233Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2349625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2350024Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2350477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:14:48.2350931Z hidden_states = residual + hidden_states 2025-09-07T07:14:48.2351083Z 2025-09-07T07:14:48.2351210Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2351624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2351982Z return mod(**inputs) 2025-09-07T07:14:48.2352405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2352859Z outputs = self.model.decoder( 2025-09-07T07:14:48.2353302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2353775Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2354164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2354556Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2354969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2355407Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2355841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2356330Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2356566Z 2025-09-07T07:14:48.2356674Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2357043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2357378Z return mod(**inputs) 2025-09-07T07:14:48.2357757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2358168Z outputs = self.model.decoder( 2025-09-07T07:14:48.2358572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2358979Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2359334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2359697Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2360106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2360545Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2361000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2361415Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2361559Z 2025-09-07T07:14:48.2361688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2362064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2362398Z return mod(**inputs) 2025-09-07T07:14:48.2362781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2363182Z outputs = self.model.decoder( 2025-09-07T07:14:48.2363585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2363989Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2364344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2364713Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2365117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2365550Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2365985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2366404Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2366549Z 2025-09-07T07:14:48.2366632Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2366854Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2367074Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2367287Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2367517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2367910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2368246Z return mod(**inputs) 2025-09-07T07:14:48.2368652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2369086Z outputs = self.model.decoder( 2025-09-07T07:14:48.2369500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2369931Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2370313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2370686Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2371147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2371607Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2372088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2372566Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2373055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2373591Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2373795Z 2025-09-07T07:14:48.2373910Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2374310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2374671Z return mod(**inputs) 2025-09-07T07:14:48.2375089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2375547Z outputs = self.model.decoder( 2025-09-07T07:14:48.2375983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2376417Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2376818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2377217Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2377648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2378107Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2378567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2379027Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2379516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2380008Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2380191Z 2025-09-07T07:14:48.2380306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2380698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2381052Z return mod(**inputs) 2025-09-07T07:14:48.2381448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2381859Z outputs = self.model.decoder( 2025-09-07T07:14:48.2382264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2382675Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2383036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2383440Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2383876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2384340Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2384774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2385214Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2385362Z 2025-09-07T07:14:48.2385475Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2385955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2386357Z return mod(**inputs) 2025-09-07T07:14:48.2386778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2387208Z outputs = self.model.decoder( 2025-09-07T07:14:48.2387631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2388065Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2388460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2388870Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2389301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2389782Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2389979Z 2025-09-07T07:14:48.2390093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2390511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2390867Z return mod(**inputs) 2025-09-07T07:14:48.2391247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2391695Z outputs = self.model.decoder( 2025-09-07T07:14:48.2392125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2392533Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2392884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2393255Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2393661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2394118Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2394521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2394862Z return self.act(input) 2025-09-07T07:14:48.2394984Z 2025-09-07T07:14:48.2395090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2395467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2395820Z return mod(**inputs) 2025-09-07T07:14:48.2396225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2396646Z outputs = self.model.decoder( 2025-09-07T07:14:48.2397071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2397503Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2397862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2398274Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2398709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2399148Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2399294Z 2025-09-07T07:14:48.2399414Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2399799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2400140Z return mod(**inputs) 2025-09-07T07:14:48.2400544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2401001Z outputs = self.model.decoder( 2025-09-07T07:14:48.2401429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2401863Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2402234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2402625Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2403062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2403517Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2403966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2404479Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2404711Z 2025-09-07T07:14:48.2404826Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2405220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2405612Z return mod(**inputs) 2025-09-07T07:14:48.2406014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2406448Z outputs = self.model.decoder( 2025-09-07T07:14:48.2406891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2407327Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2407716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2408125Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2408585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2409056Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2409534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2409987Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2410137Z 2025-09-07T07:14:48.2410254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2410654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2411015Z return mod(**inputs) 2025-09-07T07:14:48.2411425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2411859Z outputs = self.model.decoder( 2025-09-07T07:14:48.2412294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2412747Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2413119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2413515Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2413946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2414421Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2414877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2415332Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2415486Z 2025-09-07T07:14:48.2415582Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2415809Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2416042Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2416287Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2416540Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2416923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2417274Z return mod(**inputs) 2025-09-07T07:14:48.2417683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2418117Z outputs = self.model.decoder( 2025-09-07T07:14:48.2418536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2418965Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2419340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2419958Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2420401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2420861Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2421376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2421842Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2422359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2422891Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2423092Z 2025-09-07T07:14:48.2423206Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2423600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2423952Z return mod(**inputs) 2025-09-07T07:14:48.2424369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2424816Z outputs = self.model.decoder( 2025-09-07T07:14:48.2425264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2425755Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2426165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2426568Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2427022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2427490Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2427921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2428351Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2428851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2429589Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2429846Z 2025-09-07T07:14:48.2430008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2430558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2431054Z return mod(**inputs) 2025-09-07T07:14:48.2431643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2432078Z outputs = self.model.decoder( 2025-09-07T07:14:48.2432505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2432972Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2433348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2433738Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2434175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2434637Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2435089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2435571Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2435719Z 2025-09-07T07:14:48.2435832Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2436225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2436572Z return mod(**inputs) 2025-09-07T07:14:48.2436973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2437430Z outputs = self.model.decoder( 2025-09-07T07:14:48.2437849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2438343Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2438851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2439257Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2439690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2452194Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2452558Z 2025-09-07T07:14:48.2452728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2453143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2453508Z return mod(**inputs) 2025-09-07T07:14:48.2453939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2454398Z outputs = self.model.decoder( 2025-09-07T07:14:48.2454849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2455296Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2455693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2456101Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2456558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2457016Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2457426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2457879Z return self.act(input) 2025-09-07T07:14:48.2458006Z 2025-09-07T07:14:48.2458135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2458545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2458901Z return mod(**inputs) 2025-09-07T07:14:48.2459319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2459766Z outputs = self.model.decoder( 2025-09-07T07:14:48.2460203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2460630Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2461055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2461457Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2461904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2462343Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2462503Z 2025-09-07T07:14:48.2462622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2463020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2463377Z return mod(**inputs) 2025-09-07T07:14:48.2463783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2464221Z outputs = self.model.decoder( 2025-09-07T07:14:48.2464655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2465090Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2465516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2466122Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2466624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:14:48.2467081Z hidden_states = residual + hidden_states 2025-09-07T07:14:48.2467234Z 2025-09-07T07:14:48.2467363Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2467781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2468130Z return mod(**inputs) 2025-09-07T07:14:48.2468541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2468976Z outputs = self.model.decoder( 2025-09-07T07:14:48.2469401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2469829Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2470217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2470612Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2471052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2471518Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2471970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2472491Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2472724Z 2025-09-07T07:14:48.2472839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2473266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2473618Z return mod(**inputs) 2025-09-07T07:14:48.2474025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2474459Z outputs = self.model.decoder( 2025-09-07T07:14:48.2474891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2475324Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2475697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2476093Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2476551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2477015Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2477479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2477910Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2478068Z 2025-09-07T07:14:48.2478181Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2478586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2478926Z return mod(**inputs) 2025-09-07T07:14:48.2479311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2479719Z outputs = self.model.decoder( 2025-09-07T07:14:48.2480122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2480532Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2480915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2481289Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2481711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2482148Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2482580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2482997Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2483142Z 2025-09-07T07:14:48.2483227Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2483456Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2483674Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2483888Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2484127Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2484499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2484832Z return mod(**inputs) 2025-09-07T07:14:48.2485217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2485626Z outputs = self.model.decoder( 2025-09-07T07:14:48.2486020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2486425Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2486811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2487207Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2487617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2488068Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2488524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2489035Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2489498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2489997Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2490188Z 2025-09-07T07:14:48.2490296Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2490668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2491041Z return mod(**inputs) 2025-09-07T07:14:48.2491430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2491841Z outputs = self.model.decoder( 2025-09-07T07:14:48.2492247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2492658Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2493017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2493392Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2493798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2494236Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2494676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2495113Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2495592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2496058Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2496233Z 2025-09-07T07:14:48.2496357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2496729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2497065Z return mod(**inputs) 2025-09-07T07:14:48.2497446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2497859Z outputs = self.model.decoder( 2025-09-07T07:14:48.2498283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2498716Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2499076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2499435Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2499847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2500278Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2500706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2501125Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2501265Z 2025-09-07T07:14:48.2501373Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2501743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2502076Z return mod(**inputs) 2025-09-07T07:14:48.2502467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2502905Z outputs = self.model.decoder( 2025-09-07T07:14:48.2503331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2503761Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2504137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2504536Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2504969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2505452Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2505664Z 2025-09-07T07:14:48.2505918Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2506334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2506691Z return mod(**inputs) 2025-09-07T07:14:48.2507105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2507544Z outputs = self.model.decoder( 2025-09-07T07:14:48.2507985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2508418Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2508803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2509210Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2509670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2510165Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2510626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2510991Z return self.act(input) 2025-09-07T07:14:48.2511121Z 2025-09-07T07:14:48.2511259Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2511662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2512015Z return mod(**inputs) 2025-09-07T07:14:48.2512424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2512851Z outputs = self.model.decoder( 2025-09-07T07:14:48.2513283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2513734Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2514129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2514529Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2514982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2515437Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2515591Z 2025-09-07T07:14:48.2515716Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2516117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2516478Z return mod(**inputs) 2025-09-07T07:14:48.2516884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2517319Z outputs = self.model.decoder( 2025-09-07T07:14:48.2517744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2518205Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2518580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2518985Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2519431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2520280Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2520773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2521304Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2521622Z 2025-09-07T07:14:48.2521736Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2522129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2522483Z return mod(**inputs) 2025-09-07T07:14:48.2522882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2523316Z outputs = self.model.decoder( 2025-09-07T07:14:48.2523740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2524183Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2524571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2524965Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2525413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2525885Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2526395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2526846Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2527005Z 2025-09-07T07:14:48.2527165Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2527568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2527931Z return mod(**inputs) 2025-09-07T07:14:48.2528345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2528793Z outputs = self.model.decoder( 2025-09-07T07:14:48.2529229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2529673Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2530064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2530464Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2530913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2531383Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2531850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2532311Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2532468Z 2025-09-07T07:14:48.2532562Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2532803Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2533040Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2533274Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2533529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2533967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2534331Z return mod(**inputs) 2025-09-07T07:14:48.2534750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2535191Z outputs = self.model.decoder( 2025-09-07T07:14:48.2535628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2536070Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2536457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2536857Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2537316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2537798Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2538269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2538743Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2539249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2539788Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2540002Z 2025-09-07T07:14:48.2540119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2540526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2540895Z return mod(**inputs) 2025-09-07T07:14:48.2541308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2541780Z outputs = self.model.decoder( 2025-09-07T07:14:48.2542224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2542673Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2543083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2543481Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2543933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2544403Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2544874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2545339Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2545910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2546442Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2546632Z 2025-09-07T07:14:48.2546750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2547155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2547508Z return mod(**inputs) 2025-09-07T07:14:48.2547911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2548362Z outputs = self.model.decoder( 2025-09-07T07:14:48.2548804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2549249Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2549635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2550075Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2550521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2550993Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2551457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2551903Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2552065Z 2025-09-07T07:14:48.2552180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2552591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2552969Z return mod(**inputs) 2025-09-07T07:14:48.2553372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2553799Z outputs = self.model.decoder( 2025-09-07T07:14:48.2554217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2554643Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2555014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2555402Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2555850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2556331Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2556517Z 2025-09-07T07:14:48.2556636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2557027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2557398Z return mod(**inputs) 2025-09-07T07:14:48.2557806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2558242Z outputs = self.model.decoder( 2025-09-07T07:14:48.2558695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2559127Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2559499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2559891Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2560327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2560813Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2561233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2561608Z return self.act(input) 2025-09-07T07:14:48.2561736Z 2025-09-07T07:14:48.2561851Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2562259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2562629Z return mod(**inputs) 2025-09-07T07:14:48.2563029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2563465Z outputs = self.model.decoder( 2025-09-07T07:14:48.2563891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2564323Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2564695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2565121Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2565563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2566012Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2566161Z 2025-09-07T07:14:48.2566281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2566673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2567065Z return mod(**inputs) 2025-09-07T07:14:48.2567485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2567915Z outputs = self.model.decoder( 2025-09-07T07:14:48.2568374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2568814Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2569203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2569621Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2570058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:14:48.2570500Z hidden_states = residual + hidden_states 2025-09-07T07:14:48.2570658Z 2025-09-07T07:14:48.2570774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2571189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2571553Z return mod(**inputs) 2025-09-07T07:14:48.2571968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2572409Z outputs = self.model.decoder( 2025-09-07T07:14:48.2572882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2573332Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2573746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2574159Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2574599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2575080Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2575563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:14:48.2576093Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:14:48.2576320Z 2025-09-07T07:14:48.2576445Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2576842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2577204Z return mod(**inputs) 2025-09-07T07:14:48.2577623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2578072Z outputs = self.model.decoder( 2025-09-07T07:14:48.2578501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2578945Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2579332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2579747Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2580193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2580707Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2581196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:14:48.2581647Z key_states = self.k_proj(current_states) 2025-09-07T07:14:48.2581798Z 2025-09-07T07:14:48.2581925Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2582321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2582674Z return mod(**inputs) 2025-09-07T07:14:48.2583086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2583524Z outputs = self.model.decoder( 2025-09-07T07:14:48.2583986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2584434Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2584811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2585202Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2585646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2586213Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2586679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:14:48.2587144Z value_states = self.v_proj(current_states) 2025-09-07T07:14:48.2587306Z 2025-09-07T07:14:48.2587395Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2587635Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2587857Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2588086Z cudagraph partition due to non gpu ops 2025-09-07T07:14:48.2588401Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2588797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2589152Z return mod(**inputs) 2025-09-07T07:14:48.2589575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2590009Z outputs = self.model.decoder( 2025-09-07T07:14:48.2590436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2590868Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2591239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2591636Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2592074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2592541Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2592994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2593440Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2593927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:14:48.2594451Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:14:48.2594652Z 2025-09-07T07:14:48.2594774Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2595164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2595514Z return mod(**inputs) 2025-09-07T07:14:48.2595925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2596379Z outputs = self.model.decoder( 2025-09-07T07:14:48.2596808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2597249Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2597623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2598021Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2598470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2598939Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2599409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:14:48.2599872Z attn_output, attn_weights = attention_interface( 2025-09-07T07:14:48.2600355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:14:48.2600853Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:14:48.2601028Z 2025-09-07T07:14:48.2601148Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2601530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2601884Z return mod(**inputs) 2025-09-07T07:14:48.2602290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2602736Z outputs = self.model.decoder( 2025-09-07T07:14:48.2603163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2603586Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2603987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2604380Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2604841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:14:48.2605293Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:14:48.2605747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:14:48.2606187Z attn_output = self.out_proj(attn_output) 2025-09-07T07:14:48.2606336Z 2025-09-07T07:14:48.2606458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2606849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2607191Z return mod(**inputs) 2025-09-07T07:14:48.2607599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2608033Z outputs = self.model.decoder( 2025-09-07T07:14:48.2608472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2608916Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2609285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2609679Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2610113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2610596Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2610781Z 2025-09-07T07:14:48.2610899Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2611301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2611653Z return mod(**inputs) 2025-09-07T07:14:48.2612060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2612493Z outputs = self.model.decoder( 2025-09-07T07:14:48.2612905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2613338Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2613715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2614104Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2614568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:14:48.2615048Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:14:48.2615473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:14:48.2615848Z return self.act(input) 2025-09-07T07:14:48.2615971Z 2025-09-07T07:14:48.2616093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2616479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2616833Z return mod(**inputs) 2025-09-07T07:14:48.2617239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-09-07T07:14:48.2617677Z outputs = self.model.decoder( 2025-09-07T07:14:48.2618100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:14:48.2618523Z layer_outputs = decoder_layer( 2025-09-07T07:14:48.2618927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:14:48.2619327Z return super().__call__(*args, **kwargs) 2025-09-07T07:14:48.2620195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:14:48.2620665Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:14:48.2620815Z 2025-09-07T07:14:48.2620931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2621341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2621709Z return mod(**inputs) 2025-09-07T07:14:48.2622134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1650, in forward 2025-09-07T07:14:48.2622585Z logits = self.lm_head(outputs[0]) 2025-09-07T07:14:48.2622743Z 2025-09-07T07:14:48.2622862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:14:48.2623269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:14:48.2623346Z return mod(**inputs) 2025-09-07T07:14:48.2623659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1656, in forward 2025-09-07T07:14:48.2623827Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:14:48.2623831Z 2025-09-07T07:15:00.0770656Z Compilation time (from dynamo_timed): 18.349135244 2025-09-07T07:15:00.0786204Z pass 2025-09-07T07:15:00.0786685Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:15:00.0787639Z TIMING: _recursive_pre_grad_passes:0.00854 _recursive_joint_graph_passes:0.65573 _recursive_post_grad_passes:0.07959 async_compile.wait:0.73322 code_gen:11.46861 inductor_compile:12.79389 backend_compile:16.06074 gc:0.00219 entire_frame_compile:18.34914 total_wall_time:18.34914 2025-09-07T07:15:00.0788943Z STATS: call_* op count: 369 | FakeTensorMode.__torch_dispatch__:13164 | FakeTensor.__torch_dispatch__:4526 | ProxyTorchDispatchMode.__torch_dispatch__:4803 2025-09-07T07:15:00.0789502Z Dynamo produced 1 graphs covering 369 ops with 0 graph breaks (0 unique) 2025-09-07T07:15:02.8650383Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:15:02.8657036Z import pynvml # type: ignore[import] 2025-09-07T07:15:05.6843905Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:15:05.6845083Z from pkg_resources import resource_filename 2025-09-07T07:15:06.4312261Z 2025-09-07T07:15:12.1556671Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:15:12.1560278Z loading model: 0it [00:05, ?it/s] 2025-09-07T07:15:12.1583605Z cpu eval PegasusForConditionalGeneration 2025-09-07T07:15:12.8828140Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:15:13.2344511Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:15:13.5253357Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:15:30.6935842Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6941401Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6943486Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6944235Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6944860Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6945110Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6947050Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6947314Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6947647Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6947868Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6948088Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6948297Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6948556Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6949016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6949429Z return mod(**inputs) 2025-09-07T07:15:30.6949936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6950432Z outputs = self.model( 2025-09-07T07:15:30.6950901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6951357Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6951823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6952268Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6952662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.6953072Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.6953523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.6953986Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.6954558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.6955158Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.6955396Z 2025-09-07T07:15:30.6955535Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6955951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6956309Z return mod(**inputs) 2025-09-07T07:15:30.6956731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6957183Z outputs = self.model( 2025-09-07T07:15:30.6957604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6958064Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6958560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6959004Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6959392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.6959798Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.6960246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.6960715Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.6961168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.6961619Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.6961767Z 2025-09-07T07:15:30.6961884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6962279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6962638Z return mod(**inputs) 2025-09-07T07:15:30.6963078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6963510Z outputs = self.model( 2025-09-07T07:15:30.6963937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6964394Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6964846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6965291Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6965683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.6966087Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.6966552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.6967033Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.6967479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.6967928Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.6968090Z 2025-09-07T07:15:30.6968181Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6968619Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6968850Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6969076Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.6969329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6969724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6970086Z return mod(**inputs) 2025-09-07T07:15:30.6970493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6970946Z outputs = self.model( 2025-09-07T07:15:30.6971357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6971799Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6972231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6972672Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6973047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.6973458Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.6973913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.6974385Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.6974834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.6975287Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.6975777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.6976312Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.6976513Z 2025-09-07T07:15:30.6976634Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6977029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6977379Z return mod(**inputs) 2025-09-07T07:15:30.6977794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6978224Z outputs = self.model( 2025-09-07T07:15:30.6978657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6979094Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6979537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6979982Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6980374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.6980782Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.6981226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.6981692Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.6982154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.6982631Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.6983137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.6983652Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.6983842Z 2025-09-07T07:15:30.6983959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6984366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6984730Z return mod(**inputs) 2025-09-07T07:15:30.6985146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6985602Z outputs = self.model( 2025-09-07T07:15:30.6986302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6986798Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6987240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6987690Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6988085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.6988494Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.6988946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.6989419Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.6989888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.6990371Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.6990533Z 2025-09-07T07:15:30.6990656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6991063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6991431Z return mod(**inputs) 2025-09-07T07:15:30.6991847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6992297Z outputs = self.model( 2025-09-07T07:15:30.6992728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6993175Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6993602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6994051Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6994447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.6994880Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.6995335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.6995848Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.6996054Z 2025-09-07T07:15:30.6996170Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.6996575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.6996934Z return mod(**inputs) 2025-09-07T07:15:30.6997347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.6997799Z outputs = self.model( 2025-09-07T07:15:30.6998226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.6998675Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.6999116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.6999556Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.6999958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7000365Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7000807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7001302Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7001734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7002136Z return self.act(input) 2025-09-07T07:15:30.7002264Z 2025-09-07T07:15:30.7002377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7002819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7003169Z return mod(**inputs) 2025-09-07T07:15:30.7003588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7004027Z outputs = self.model( 2025-09-07T07:15:30.7004435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7004870Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7005306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7005792Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7006178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7006581Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7007011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7007442Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7007601Z 2025-09-07T07:15:30.7007718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7008112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7008460Z return mod(**inputs) 2025-09-07T07:15:30.7008857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7009298Z outputs = self.model( 2025-09-07T07:15:30.7009711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7010152Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7010611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7011042Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7011468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7011869Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7012302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7012757Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7013201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7013726Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7013968Z 2025-09-07T07:15:30.7014086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7014485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7014846Z return mod(**inputs) 2025-09-07T07:15:30.7015258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7015699Z outputs = self.model( 2025-09-07T07:15:30.7016120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7016571Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7016997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7017429Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7017805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7018220Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7018657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7019113Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7019848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7020322Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7020479Z 2025-09-07T07:15:30.7020605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7021015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7021373Z return mod(**inputs) 2025-09-07T07:15:30.7021862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7022299Z outputs = self.model( 2025-09-07T07:15:30.7022713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7023149Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7023594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7024034Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7024430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7024847Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7025286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7025838Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7026361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7026829Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7026989Z 2025-09-07T07:15:30.7027088Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7027358Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7027596Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7027832Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7028098Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7028505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7028881Z return mod(**inputs) 2025-09-07T07:15:30.7029306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7029757Z outputs = self.model( 2025-09-07T07:15:30.7030175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7030633Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7031082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7031532Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7031928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7032329Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7032774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7033243Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7033710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7034184Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7034719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7035275Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7035494Z 2025-09-07T07:15:30.7035616Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7036026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7036398Z return mod(**inputs) 2025-09-07T07:15:30.7036814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7037258Z outputs = self.model( 2025-09-07T07:15:30.7037692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7038117Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7038536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7038967Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7039352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7039747Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7040184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7040639Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7041105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7041556Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7042053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7042558Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7042733Z 2025-09-07T07:15:30.7042845Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7043262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7043616Z return mod(**inputs) 2025-09-07T07:15:30.7044023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7044465Z outputs = self.model( 2025-09-07T07:15:30.7044884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7045322Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7045749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7046186Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7046556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7046951Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7047387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7047851Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7048308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7048762Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7048919Z 2025-09-07T07:15:30.7049033Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7049429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7049804Z return mod(**inputs) 2025-09-07T07:15:30.7050210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7050628Z outputs = self.model( 2025-09-07T07:15:30.7051030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7051467Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7051907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7052340Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7052733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7053180Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7053622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7054137Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7054340Z 2025-09-07T07:15:30.7054461Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7054874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7055249Z return mod(**inputs) 2025-09-07T07:15:30.7055682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7056140Z outputs = self.model( 2025-09-07T07:15:30.7056569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7057029Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7057476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7057951Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7058334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7058747Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7059255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7059775Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7060209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7060607Z return self.act(input) 2025-09-07T07:15:30.7060739Z 2025-09-07T07:15:30.7060856Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7061261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7061641Z return mod(**inputs) 2025-09-07T07:15:30.7062074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7062509Z outputs = self.model( 2025-09-07T07:15:30.7062931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7063376Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7063812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7064265Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7064664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7065078Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7065529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7066120Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7066276Z 2025-09-07T07:15:30.7066396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7066802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7067173Z return mod(**inputs) 2025-09-07T07:15:30.7067590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7068035Z outputs = self.model( 2025-09-07T07:15:30.7068446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7068894Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7069375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7069825Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7070208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7070628Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7071096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7071565Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7072046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7072584Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7072824Z 2025-09-07T07:15:30.7072942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7073350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7073718Z return mod(**inputs) 2025-09-07T07:15:30.7075103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7075546Z outputs = self.model( 2025-09-07T07:15:30.7075997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7076447Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7076897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7077335Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7077725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7078127Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7078576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7079040Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7079492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7079961Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7080124Z 2025-09-07T07:15:30.7080241Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7080644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7081006Z return mod(**inputs) 2025-09-07T07:15:30.7081421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7081864Z outputs = self.model( 2025-09-07T07:15:30.7082282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7082753Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7083184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7083629Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7084018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7084422Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7084871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7085322Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7085791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7086250Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7086400Z 2025-09-07T07:15:30.7086501Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7086736Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7086961Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7087192Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7087578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7088005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7088372Z return mod(**inputs) 2025-09-07T07:15:30.7088799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7089244Z outputs = self.model( 2025-09-07T07:15:30.7089666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7090118Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7090584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7091042Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7091443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7091879Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7092349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7092851Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7093318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7093809Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7094314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7094872Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7095090Z 2025-09-07T07:15:30.7095204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7095611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7095986Z return mod(**inputs) 2025-09-07T07:15:30.7096404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7096834Z outputs = self.model( 2025-09-07T07:15:30.7097247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7097773Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7098211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7098654Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7099037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7099424Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7099862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7100306Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7100763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7101215Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7101694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7102220Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7102400Z 2025-09-07T07:15:30.7102526Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7102921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7103284Z return mod(**inputs) 2025-09-07T07:15:30.7103692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7104115Z outputs = self.model( 2025-09-07T07:15:30.7104528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7104972Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7105411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7105929Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7106327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7106754Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7107206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7107689Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7108155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7108609Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7108764Z 2025-09-07T07:15:30.7108883Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7109288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7109655Z return mod(**inputs) 2025-09-07T07:15:30.7110077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7110515Z outputs = self.model( 2025-09-07T07:15:30.7110936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7111383Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7111838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7112290Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7112662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7113063Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7113474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7113934Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7114116Z 2025-09-07T07:15:30.7114258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7114624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7114962Z return mod(**inputs) 2025-09-07T07:15:30.7115349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7115756Z outputs = self.model( 2025-09-07T07:15:30.7116139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7116555Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7117001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7117458Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7117837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7118226Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7118650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7119186Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7119823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7120207Z return self.act(input) 2025-09-07T07:15:30.7120330Z 2025-09-07T07:15:30.7120444Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7120848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7121182Z return mod(**inputs) 2025-09-07T07:15:30.7121569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7122042Z outputs = self.model( 2025-09-07T07:15:30.7122426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7122832Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7123266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7123681Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7124037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7124417Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7124832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7125261Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7125401Z 2025-09-07T07:15:30.7125514Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7125888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7126226Z return mod(**inputs) 2025-09-07T07:15:30.7126615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7127024Z outputs = self.model( 2025-09-07T07:15:30.7127409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7127819Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7128221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7128636Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7128998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7129409Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7129826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-09-07T07:15:30.7130247Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7130389Z 2025-09-07T07:15:30.7130506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7130881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7131215Z return mod(**inputs) 2025-09-07T07:15:30.7131608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7132015Z outputs = self.model( 2025-09-07T07:15:30.7132402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7132837Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7133244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7133650Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7134013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7134389Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7134804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7135226Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7135638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7136123Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7136334Z 2025-09-07T07:15:30.7136467Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7136835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7137168Z return mod(**inputs) 2025-09-07T07:15:30.7137580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7137983Z outputs = self.model( 2025-09-07T07:15:30.7138366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7138769Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7139170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7139576Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7139936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7140305Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7140719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7141149Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7141576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7141994Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7142133Z 2025-09-07T07:15:30.7142240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7142613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7142949Z return mod(**inputs) 2025-09-07T07:15:30.7143338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7143801Z outputs = self.model( 2025-09-07T07:15:30.7144189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7144598Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7145001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7145408Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7145835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7146237Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7146671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7147139Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7147569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7147970Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7148120Z 2025-09-07T07:15:30.7148204Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7148427Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7148643Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7148859Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7149092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7149447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7149771Z return mod(**inputs) 2025-09-07T07:15:30.7150136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7150529Z outputs = self.model( 2025-09-07T07:15:30.7150939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7151343Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7151733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7152155Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7152535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7152930Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7153363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7153811Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7154244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7154676Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7155147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7155626Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7155809Z 2025-09-07T07:15:30.7155921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7156280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7156610Z return mod(**inputs) 2025-09-07T07:15:30.7156998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7157398Z outputs = self.model( 2025-09-07T07:15:30.7157772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7158179Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7158610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7159018Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7159376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7159739Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7160152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7160578Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7161001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7161452Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7161904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7162381Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7162553Z 2025-09-07T07:15:30.7162660Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7163037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7163365Z return mod(**inputs) 2025-09-07T07:15:30.7163755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7164158Z outputs = self.model( 2025-09-07T07:15:30.7164541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7164950Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7165348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7165785Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7166155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7166542Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7166984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7167383Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7167794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7168202Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7168337Z 2025-09-07T07:15:30.7168447Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7168801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7169130Z return mod(**inputs) 2025-09-07T07:15:30.7169512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7169893Z outputs = self.model( 2025-09-07T07:15:30.7170266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7170658Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7171048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7171441Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7171787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7172148Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7172545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7173007Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7173187Z 2025-09-07T07:15:30.7173289Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7173647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7173965Z return mod(**inputs) 2025-09-07T07:15:30.7174339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7174730Z outputs = self.model( 2025-09-07T07:15:30.7175101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7175531Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7175925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7176370Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7176762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7177158Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7177595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7178073Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7178460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7178802Z return self.act(input) 2025-09-07T07:15:30.7178912Z 2025-09-07T07:15:30.7179023Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7179406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7179760Z return mod(**inputs) 2025-09-07T07:15:30.7180190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7180630Z outputs = self.model( 2025-09-07T07:15:30.7181071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7181508Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7181948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7182402Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7182782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7183171Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7183607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7184047Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7184194Z 2025-09-07T07:15:30.7184317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7184707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7185053Z return mod(**inputs) 2025-09-07T07:15:30.7185463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7185970Z outputs = self.model( 2025-09-07T07:15:30.7186381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7186814Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7187239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7187672Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7188084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7188478Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7188907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7189363Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7189816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7190331Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7190556Z 2025-09-07T07:15:30.7190678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7191100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7191452Z return mod(**inputs) 2025-09-07T07:15:30.7191861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7192289Z outputs = self.model( 2025-09-07T07:15:30.7192697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7193123Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7193549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7193979Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7194357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7194745Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7195218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7195682Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7196151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7196625Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7196772Z 2025-09-07T07:15:30.7196887Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7197277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7197635Z return mod(**inputs) 2025-09-07T07:15:30.7198043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7198444Z outputs = self.model( 2025-09-07T07:15:30.7198841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7199285Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7199689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7200097Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7200447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7200821Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7201229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7201656Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7202074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7202485Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7202637Z 2025-09-07T07:15:30.7202752Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7202973Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7203188Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7203393Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7203634Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7204005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7204338Z return mod(**inputs) 2025-09-07T07:15:30.7204725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7205127Z outputs = self.model( 2025-09-07T07:15:30.7205515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7205944Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7206345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7206754Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7207104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7207476Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7207888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7208385Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7208890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7209317Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7209793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7210344Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7210546Z 2025-09-07T07:15:30.7210662Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7211072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7211436Z return mod(**inputs) 2025-09-07T07:15:30.7211832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7212233Z outputs = self.model( 2025-09-07T07:15:30.7212615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7213015Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7213422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7213828Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7214192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7214564Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7215014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7215452Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7215879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7216307Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7216755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7217256Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7217438Z 2025-09-07T07:15:30.7217580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7217973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7218325Z return mod(**inputs) 2025-09-07T07:15:30.7218706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7219107Z outputs = self.model( 2025-09-07T07:15:30.7219494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7220107Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7220525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7221016Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7221396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7221791Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7222224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7222676Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7223128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7223578Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7223725Z 2025-09-07T07:15:30.7223848Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7224240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7224593Z return mod(**inputs) 2025-09-07T07:15:30.7224998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7225457Z outputs = self.model( 2025-09-07T07:15:30.7225929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7226375Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7226867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7227295Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7227676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7228070Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7228494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7228953Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7229137Z 2025-09-07T07:15:30.7229248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7229619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7229961Z return mod(**inputs) 2025-09-07T07:15:30.7230363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7230788Z outputs = self.model( 2025-09-07T07:15:30.7231192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7231620Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7232034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7232449Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7232806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7233250Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7233679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7234152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7234571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7234944Z return self.act(input) 2025-09-07T07:15:30.7235063Z 2025-09-07T07:15:30.7235182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7235576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7235921Z return mod(**inputs) 2025-09-07T07:15:30.7236348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7236777Z outputs = self.model( 2025-09-07T07:15:30.7237188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7237610Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7238055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7238481Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7238862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7239251Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7239698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7240145Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7240299Z 2025-09-07T07:15:30.7240412Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7240850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7241202Z return mod(**inputs) 2025-09-07T07:15:30.7241612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7242037Z outputs = self.model( 2025-09-07T07:15:30.7242433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7242841Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7243236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7243644Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7244005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7244380Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7244794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-09-07T07:15:30.7245200Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7245345Z 2025-09-07T07:15:30.7245453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7245818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7246148Z return mod(**inputs) 2025-09-07T07:15:30.7246530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7246921Z outputs = self.model( 2025-09-07T07:15:30.7247303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7247709Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7248127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7248526Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7248975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7249350Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7249770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7250219Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7250674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7251219Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7251449Z 2025-09-07T07:15:30.7251564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7251958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7252291Z return mod(**inputs) 2025-09-07T07:15:30.7252671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7253104Z outputs = self.model( 2025-09-07T07:15:30.7253514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7253949Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7254367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7254804Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7255178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7255597Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7256121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7256577Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7257050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7257503Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7257666Z 2025-09-07T07:15:30.7257773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7258146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7258492Z return mod(**inputs) 2025-09-07T07:15:30.7258893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7259322Z outputs = self.model( 2025-09-07T07:15:30.7259735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7260173Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7260600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7261018Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7261395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7261795Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7262227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7262671Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7263120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7263586Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7263741Z 2025-09-07T07:15:30.7263841Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7264075Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7264296Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7264525Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7264789Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7265185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7265529Z return mod(**inputs) 2025-09-07T07:15:30.7266019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7266497Z outputs = self.model( 2025-09-07T07:15:30.7266926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7267373Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7267804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7268251Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7268655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7269074Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7269527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7269996Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7270465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7270949Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7271469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7271999Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7272207Z 2025-09-07T07:15:30.7272339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7272748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7273101Z return mod(**inputs) 2025-09-07T07:15:30.7273529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7273965Z outputs = self.model( 2025-09-07T07:15:30.7274387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7274832Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7275275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7275710Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7276096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7276494Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7276912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7277338Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7277760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7278196Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7278654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7279172Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7279347Z 2025-09-07T07:15:30.7279468Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7279858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7280215Z return mod(**inputs) 2025-09-07T07:15:30.7280625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7281056Z outputs = self.model( 2025-09-07T07:15:30.7281467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7281931Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7282354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7282766Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7283126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7283495Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7283913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7284347Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7284776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7285197Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7285340Z 2025-09-07T07:15:30.7285447Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7285823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7286171Z return mod(**inputs) 2025-09-07T07:15:30.7286585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7286997Z outputs = self.model( 2025-09-07T07:15:30.7287428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7287854Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7288269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7288714Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7289087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7289475Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7289890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7290355Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7290532Z 2025-09-07T07:15:30.7290649Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7291021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7291369Z return mod(**inputs) 2025-09-07T07:15:30.7291771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7292205Z outputs = self.model( 2025-09-07T07:15:30.7292610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7293048Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7293475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7293953Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7294333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7294722Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7295158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7295641Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7296065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7296438Z return self.act(input) 2025-09-07T07:15:30.7296561Z 2025-09-07T07:15:30.7296671Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7297090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7297443Z return mod(**inputs) 2025-09-07T07:15:30.7297849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7298273Z outputs = self.model( 2025-09-07T07:15:30.7298684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7299123Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7299545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7299973Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7300349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7300751Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7301185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7301650Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7301807Z 2025-09-07T07:15:30.7301927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7302329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7302682Z return mod(**inputs) 2025-09-07T07:15:30.7303088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7303518Z outputs = self.model( 2025-09-07T07:15:30.7303920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7304351Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7304779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7305209Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7305590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7306075Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7306542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7306996Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7307451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7307942Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7308163Z 2025-09-07T07:15:30.7308271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7308644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7308970Z return mod(**inputs) 2025-09-07T07:15:30.7309402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7309832Z outputs = self.model( 2025-09-07T07:15:30.7310261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7310707Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7311144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7311590Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7311975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7312403Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7312856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7313306Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7313728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7314150Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7314287Z 2025-09-07T07:15:30.7314403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7314766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7315099Z return mod(**inputs) 2025-09-07T07:15:30.7315482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7315888Z outputs = self.model( 2025-09-07T07:15:30.7316281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7316742Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7317189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7317627Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7318027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7318429Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7318876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7319324Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7319985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7320444Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7320600Z 2025-09-07T07:15:30.7320691Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7320934Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7321164Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7321392Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7321643Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7322043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7322398Z return mod(**inputs) 2025-09-07T07:15:30.7322807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7323238Z outputs = self.model( 2025-09-07T07:15:30.7323641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7324078Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7324506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7325013Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7325394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7325790Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7326223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7326680Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7327131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7327580Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7328094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7328626Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7328827Z 2025-09-07T07:15:30.7328948Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7329343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7329691Z return mod(**inputs) 2025-09-07T07:15:30.7330117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7330566Z outputs = self.model( 2025-09-07T07:15:30.7330982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7331420Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7331849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7332284Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7332699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7333094Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7333549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7334003Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7334445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7334900Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7335355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7335830Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7336009Z 2025-09-07T07:15:30.7336126Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7336518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7336877Z return mod(**inputs) 2025-09-07T07:15:30.7337287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7337713Z outputs = self.model( 2025-09-07T07:15:30.7338128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7338565Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7338991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7339398Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7339751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7340172Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7340606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7341063Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7341511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7341969Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7342130Z 2025-09-07T07:15:30.7342248Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7342651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7343012Z return mod(**inputs) 2025-09-07T07:15:30.7343448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7343884Z outputs = self.model( 2025-09-07T07:15:30.7344293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7344722Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7345138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7345567Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7346033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7346447Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7346897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7347395Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7347599Z 2025-09-07T07:15:30.7347762Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7348140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7348479Z return mod(**inputs) 2025-09-07T07:15:30.7348883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7349278Z outputs = self.model( 2025-09-07T07:15:30.7349677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7350106Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7350526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7350961Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7351332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7351730Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7352164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7352647Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7353061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7353435Z return self.act(input) 2025-09-07T07:15:30.7353564Z 2025-09-07T07:15:30.7353673Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7354064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7354413Z return mod(**inputs) 2025-09-07T07:15:30.7354810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7355234Z outputs = self.model( 2025-09-07T07:15:30.7355660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7356095Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7356515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7356943Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7357324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7357716Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7358147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7358610Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7358767Z 2025-09-07T07:15:30.7358879Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7359273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7359626Z return mod(**inputs) 2025-09-07T07:15:30.7360033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7360453Z outputs = self.model( 2025-09-07T07:15:30.7360859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7361286Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7361710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7362139Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7362517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7362940Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7363379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-09-07T07:15:30.7363819Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7363968Z 2025-09-07T07:15:30.7364099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7364487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7364836Z return mod(**inputs) 2025-09-07T07:15:30.7365237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7365660Z outputs = self.model( 2025-09-07T07:15:30.7366066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7366472Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7366894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7367321Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7367694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7368094Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7368532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7368983Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7369430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7369940Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7370169Z 2025-09-07T07:15:30.7370281Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7370700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7371030Z return mod(**inputs) 2025-09-07T07:15:30.7371415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7371826Z outputs = self.model( 2025-09-07T07:15:30.7372231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7372660Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7373084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7373485Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7373865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7374239Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7374651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7375080Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7375503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7375927Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7376082Z 2025-09-07T07:15:30.7376195Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7376589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7376956Z return mod(**inputs) 2025-09-07T07:15:30.7377370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7377806Z outputs = self.model( 2025-09-07T07:15:30.7378216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7378633Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7379049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7379462Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7379846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7380240Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7380689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7381148Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7381612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7382060Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7382211Z 2025-09-07T07:15:30.7382309Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7382543Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7382765Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7382992Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7383247Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7383638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7383983Z return mod(**inputs) 2025-09-07T07:15:30.7384388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7384817Z outputs = self.model( 2025-09-07T07:15:30.7385221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7385669Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7386168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7386616Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7387011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7387424Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7387873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7388324Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7388782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7389262Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7389747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7390258Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7390466Z 2025-09-07T07:15:30.7390579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7390970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7391322Z return mod(**inputs) 2025-09-07T07:15:30.7391727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7392148Z outputs = self.model( 2025-09-07T07:15:30.7392555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7392990Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7393460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7393897Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7394305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7394701Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7395140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7395606Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7396065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7396535Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7397024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7397534Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7397708Z 2025-09-07T07:15:30.7397827Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7398217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7398586Z return mod(**inputs) 2025-09-07T07:15:30.7399004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7399445Z outputs = self.model( 2025-09-07T07:15:30.7399859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7400304Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7400748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7401210Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7401568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7401935Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7402352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7402778Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7403203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7403617Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7403758Z 2025-09-07T07:15:30.7403865Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7404272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7404602Z return mod(**inputs) 2025-09-07T07:15:30.7404990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7405387Z outputs = self.model( 2025-09-07T07:15:30.7405775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7406186Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7406585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7406988Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7407345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7407741Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7408165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7408641Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7408823Z 2025-09-07T07:15:30.7408943Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7409344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7409700Z return mod(**inputs) 2025-09-07T07:15:30.7410110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7410538Z outputs = self.model( 2025-09-07T07:15:30.7410917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7411324Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7411730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7412144Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7412502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7412892Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7413329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7413816Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7414229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7414580Z return self.act(input) 2025-09-07T07:15:30.7414694Z 2025-09-07T07:15:30.7414803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7415178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7415513Z return mod(**inputs) 2025-09-07T07:15:30.7415917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7416317Z outputs = self.model( 2025-09-07T07:15:30.7416740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7417172Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7417597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7418034Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7418408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7418784Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7419259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7419900Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7420054Z 2025-09-07T07:15:30.7420175Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7420561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7420914Z return mod(**inputs) 2025-09-07T07:15:30.7421318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7421750Z outputs = self.model( 2025-09-07T07:15:30.7422160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7422602Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7423042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7423483Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7423952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7424372Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7424871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7425326Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7425829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7426374Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7426604Z 2025-09-07T07:15:30.7426718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7427129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7427492Z return mod(**inputs) 2025-09-07T07:15:30.7427921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7428348Z outputs = self.model( 2025-09-07T07:15:30.7428759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7429191Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7429617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7430046Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7430421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7430814Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7431250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7431750Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7432191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7432594Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7432741Z 2025-09-07T07:15:30.7432847Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7433215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7433543Z return mod(**inputs) 2025-09-07T07:15:30.7433914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7434309Z outputs = self.model( 2025-09-07T07:15:30.7434709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7435111Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7435502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7435888Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7436241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7436613Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7437023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7437449Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7437908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7438352Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7438508Z 2025-09-07T07:15:30.7438591Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7438846Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7439635Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7439852Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7440120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7440487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7440817Z return mod(**inputs) 2025-09-07T07:15:30.7441186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7441580Z outputs = self.model( 2025-09-07T07:15:30.7441958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7442359Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7442750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7443150Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7443501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7443867Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7444268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7444672Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7445086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7445506Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7445954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7446508Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7446707Z 2025-09-07T07:15:30.7446821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7447216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7447589Z return mod(**inputs) 2025-09-07T07:15:30.7448015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7448403Z outputs = self.model( 2025-09-07T07:15:30.7448778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7449174Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7449584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7449975Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7450317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7450676Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7451080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7451505Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7451928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7452350Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7452807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7453277Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7453448Z 2025-09-07T07:15:30.7453579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7453946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7454272Z return mod(**inputs) 2025-09-07T07:15:30.7454672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7455074Z outputs = self.model( 2025-09-07T07:15:30.7455453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7455853Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7456251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7456679Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7457055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7457450Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7457875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7458301Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7458724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7459138Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7459274Z 2025-09-07T07:15:30.7459387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7459750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7460085Z return mod(**inputs) 2025-09-07T07:15:30.7460470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7460909Z outputs = self.model( 2025-09-07T07:15:30.7461311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7461762Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7462203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7462644Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7463029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7463424Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7463872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7464390Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7464578Z 2025-09-07T07:15:30.7464697Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7465096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7465449Z return mod(**inputs) 2025-09-07T07:15:30.7465943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7466392Z outputs = self.model( 2025-09-07T07:15:30.7466801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7467240Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7467667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7468077Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7468439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7468841Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7469251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7469729Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7470131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7470483Z return self.act(input) 2025-09-07T07:15:30.7470597Z 2025-09-07T07:15:30.7470703Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7471085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7471418Z return mod(**inputs) 2025-09-07T07:15:30.7471814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7472244Z outputs = self.model( 2025-09-07T07:15:30.7472650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7473087Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7473515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7473948Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7474327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7474715Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7475147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7475570Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7475711Z 2025-09-07T07:15:30.7475825Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7476233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7476591Z return mod(**inputs) 2025-09-07T07:15:30.7476998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7477428Z outputs = self.model( 2025-09-07T07:15:30.7477847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7478283Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7478713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7479155Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7479580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7479976Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7480419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-09-07T07:15:30.7480857Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7481008Z 2025-09-07T07:15:30.7481123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7481512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7481866Z return mod(**inputs) 2025-09-07T07:15:30.7482275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7482707Z outputs = self.model( 2025-09-07T07:15:30.7483112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7483555Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7484008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7484464Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7484883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7485301Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7485766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7486228Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7486690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7487207Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7487428Z 2025-09-07T07:15:30.7487549Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7487940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7488290Z return mod(**inputs) 2025-09-07T07:15:30.7488697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7489123Z outputs = self.model( 2025-09-07T07:15:30.7489528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7489948Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7490374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7490805Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7491183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7491604Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7492032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7492482Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7492930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7493381Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7493528Z 2025-09-07T07:15:30.7493640Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7494032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7494397Z return mod(**inputs) 2025-09-07T07:15:30.7494824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7495258Z outputs = self.model( 2025-09-07T07:15:30.7495642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7496049Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7496462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7496897Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7497288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7497681Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7498127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7498590Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7499052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7499487Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7499648Z 2025-09-07T07:15:30.7499737Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7499969Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7500213Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7500442Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7500691Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7501091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7501453Z return mod(**inputs) 2025-09-07T07:15:30.7501866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7502298Z outputs = self.model( 2025-09-07T07:15:30.7502711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7503155Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7503588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7504024Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7504402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7504797Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7505237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7505770Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7506231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7506724Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7507257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7507791Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7507981Z 2025-09-07T07:15:30.7508102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7508470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7508815Z return mod(**inputs) 2025-09-07T07:15:30.7509205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7509617Z outputs = self.model( 2025-09-07T07:15:30.7510006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7510430Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7510836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7511237Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7511597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7511961Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7512373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7512797Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7513222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7513655Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7514110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7514606Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7514778Z 2025-09-07T07:15:30.7514884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7515278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7515613Z return mod(**inputs) 2025-09-07T07:15:30.7515990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7516392Z outputs = self.model( 2025-09-07T07:15:30.7516774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7517180Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7517581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7517994Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7518382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7518785Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7519225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7519808Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7520239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7520688Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7520836Z 2025-09-07T07:15:30.7520957Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7521361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7521710Z return mod(**inputs) 2025-09-07T07:15:30.7522189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7522626Z outputs = self.model( 2025-09-07T07:15:30.7523043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7523487Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7523918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7524352Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7524742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7525183Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7525613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7526119Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7526299Z 2025-09-07T07:15:30.7526404Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7526770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7527109Z return mod(**inputs) 2025-09-07T07:15:30.7527496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7527902Z outputs = self.model( 2025-09-07T07:15:30.7528287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7528702Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7529097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7529515Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7529865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7530230Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7530664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7531102Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7531490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7531830Z return self.act(input) 2025-09-07T07:15:30.7531940Z 2025-09-07T07:15:30.7532051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7532414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7532733Z return mod(**inputs) 2025-09-07T07:15:30.7533111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7533506Z outputs = self.model( 2025-09-07T07:15:30.7533880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7534275Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7534666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7535063Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7535410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7535781Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7536194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7536656Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7536815Z 2025-09-07T07:15:30.7536928Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7537319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7537671Z return mod(**inputs) 2025-09-07T07:15:30.7538069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7538472Z outputs = self.model( 2025-09-07T07:15:30.7538859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7539263Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7539655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7540104Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7540485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7540877Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7541311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7541755Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7542205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7542716Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7542940Z 2025-09-07T07:15:30.7543059Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7543450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7543793Z return mod(**inputs) 2025-09-07T07:15:30.7544218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7544650Z outputs = self.model( 2025-09-07T07:15:30.7545069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7545502Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7545978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7546420Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7546798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7547196Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7547630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7548088Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7548539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7548984Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7549131Z 2025-09-07T07:15:30.7549252Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7549640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7549998Z return mod(**inputs) 2025-09-07T07:15:30.7550410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7550838Z outputs = self.model( 2025-09-07T07:15:30.7551242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7551677Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7552132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7552565Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7552944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7553329Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7553758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7554210Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7554635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7555056Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7555196Z 2025-09-07T07:15:30.7555277Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7555493Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7555702Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7555910Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7556139Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7556509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7556843Z return mod(**inputs) 2025-09-07T07:15:30.7557227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7557627Z outputs = self.model( 2025-09-07T07:15:30.7558015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7558409Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7558828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7559229Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7559581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7559991Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7560408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7560835Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7561262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7561692Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7562185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7562701Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7562892Z 2025-09-07T07:15:30.7563007Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7563384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7563722Z return mod(**inputs) 2025-09-07T07:15:30.7564120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7564519Z outputs = self.model( 2025-09-07T07:15:30.7564902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7564979Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7565258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7565343Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7565588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7565677Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7565942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7566043Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7566315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7566416Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7566720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7566853Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7566857Z 2025-09-07T07:15:30.7566969Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7567182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7567262Z return mod(**inputs) 2025-09-07T07:15:30.7567566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7567640Z outputs = self.model( 2025-09-07T07:15:30.7567936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7568011Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7568289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7568364Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7568589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7568705Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7568998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7569105Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7569422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7569511Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7569523Z 2025-09-07T07:15:30.7569633Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7569856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7569934Z return mod(**inputs) 2025-09-07T07:15:30.7570225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7570305Z outputs = self.model( 2025-09-07T07:15:30.7570594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7570672Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7570970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7571043Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7571278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7571357Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7571630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7571760Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7571764Z 2025-09-07T07:15:30.7571868Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7572108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7572178Z return mod(**inputs) 2025-09-07T07:15:30.7572473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7572545Z outputs = self.model( 2025-09-07T07:15:30.7572826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7572910Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7573191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7573292Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7573531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7573617Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7573910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7574042Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7574279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7574353Z return self.act(input) 2025-09-07T07:15:30.7574357Z 2025-09-07T07:15:30.7574469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7574694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7574765Z return mod(**inputs) 2025-09-07T07:15:30.7575062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7575134Z outputs = self.model( 2025-09-07T07:15:30.7575448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7575527Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7575834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7575921Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7576161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7576255Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7576541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7576631Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7576635Z 2025-09-07T07:15:30.7576752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7576970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7577050Z return mod(**inputs) 2025-09-07T07:15:30.7577337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7577418Z outputs = self.model( 2025-09-07T07:15:30.7577702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7577777Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7578068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7578145Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7578392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7578476Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7578783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-09-07T07:15:30.7578878Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7578882Z 2025-09-07T07:15:30.7578992Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7579212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7579282Z return mod(**inputs) 2025-09-07T07:15:30.7579568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7579649Z outputs = self.model( 2025-09-07T07:15:30.7579934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7580036Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7580338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7580424Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7580671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7580754Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7581049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7581146Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7581451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7581616Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7581621Z 2025-09-07T07:15:30.7581732Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7581970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7582043Z return mod(**inputs) 2025-09-07T07:15:30.7582360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7582436Z outputs = self.model( 2025-09-07T07:15:30.7582743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7582822Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7583124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7583207Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7583446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7583538Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7583839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7583937Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7584247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7584332Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7584336Z 2025-09-07T07:15:30.7584453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7584678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7584753Z return mod(**inputs) 2025-09-07T07:15:30.7585053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7585125Z outputs = self.model( 2025-09-07T07:15:30.7585439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7585515Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7586075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7586159Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7586420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7586518Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7586819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7586948Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7587246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7587343Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7587355Z 2025-09-07T07:15:30.7587443Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7587529Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7587625Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7587708Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7587819Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7588037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7588108Z return mod(**inputs) 2025-09-07T07:15:30.7588409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7588484Z outputs = self.model( 2025-09-07T07:15:30.7588788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7588885Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7589177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7589260Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7589526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7589618Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7589915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7590011Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7590314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7590421Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7590749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7590893Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7590897Z 2025-09-07T07:15:30.7591017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7591235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7591305Z return mod(**inputs) 2025-09-07T07:15:30.7591601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7591673Z outputs = self.model( 2025-09-07T07:15:30.7591963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7592043Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7592326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7592434Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7592674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7592776Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7593043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7593133Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7593409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7593507Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7593835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7593951Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7593955Z 2025-09-07T07:15:30.7594064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7594273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7594339Z return mod(**inputs) 2025-09-07T07:15:30.7594618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7594685Z outputs = self.model( 2025-09-07T07:15:30.7594958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7595032Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7595310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7595453Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7595693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7595784Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7596089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-09-07T07:15:30.7596196Z hidden_states, attn_weights = self.self_attn( 2025-09-07T07:15:30.7596477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7596563Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7596567Z 2025-09-07T07:15:30.7596684Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7596899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7596976Z return mod(**inputs) 2025-09-07T07:15:30.7597262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7597330Z outputs = self.model( 2025-09-07T07:15:30.7597604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7597677Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7597949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7598022Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7598253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7598335Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7598615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7598773Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7598777Z 2025-09-07T07:15:30.7598887Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7599113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7599184Z return mod(**inputs) 2025-09-07T07:15:30.7599470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7599550Z outputs = self.model( 2025-09-07T07:15:30.7599835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7599920Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7600217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7600302Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7600539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7600623Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7600915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-09-07T07:15:30.7601041Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7601278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7601354Z return self.act(input) 2025-09-07T07:15:30.7601357Z 2025-09-07T07:15:30.7601472Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7601684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7601750Z return mod(**inputs) 2025-09-07T07:15:30.7602045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7602113Z outputs = self.model( 2025-09-07T07:15:30.7602394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-09-07T07:15:30.7602478Z encoder_outputs = self.encoder( 2025-09-07T07:15:30.7602746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-09-07T07:15:30.7602824Z layer_outputs = encoder_layer( 2025-09-07T07:15:30.7603047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7603133Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7603406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-09-07T07:15:30.7603490Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7603493Z 2025-09-07T07:15:30.7603603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7603805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7603882Z return mod(**inputs) 2025-09-07T07:15:30.7604153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7604221Z outputs = self.model( 2025-09-07T07:15:30.7604501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7604572Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7604850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7604925Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7605169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7605256Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7605524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7605636Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7605905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7606065Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7606069Z 2025-09-07T07:15:30.7606172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7607043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7607119Z return mod(**inputs) 2025-09-07T07:15:30.7607397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7607474Z outputs = self.model( 2025-09-07T07:15:30.7607750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7607828Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7608113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7608189Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7608425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7608509Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7608788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7608918Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7609187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7609298Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7609302Z 2025-09-07T07:15:30.7609408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7609618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7609684Z return mod(**inputs) 2025-09-07T07:15:30.7609952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7610029Z outputs = self.model( 2025-09-07T07:15:30.7610301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7610382Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7610653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7610731Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7610960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7611039Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7611315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7611416Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7611693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7611783Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7611786Z 2025-09-07T07:15:30.7611868Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7611978Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7612057Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7612142Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7612247Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7612451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7612526Z return mod(**inputs) 2025-09-07T07:15:30.7612795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7612872Z outputs = self.model( 2025-09-07T07:15:30.7613141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7613238Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7613515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7613590Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7613824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7613908Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7614183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7614284Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7614554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7614670Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7614963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7615118Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7615122Z 2025-09-07T07:15:30.7615224Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7615450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7615517Z return mod(**inputs) 2025-09-07T07:15:30.7615785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7615861Z outputs = self.model( 2025-09-07T07:15:30.7616136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7616215Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7616485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7616557Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7616794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7616875Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7617153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7617253Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7617519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7617623Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7617925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7618043Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7618046Z 2025-09-07T07:15:30.7618175Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7618378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7618443Z return mod(**inputs) 2025-09-07T07:15:30.7618707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7618781Z outputs = self.model( 2025-09-07T07:15:30.7619048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7619132Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7619398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7619486Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7619842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7619933Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7620211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7620315Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7620590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7620676Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7620680Z 2025-09-07T07:15:30.7620785Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7620996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7621066Z return mod(**inputs) 2025-09-07T07:15:30.7621344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7621462Z outputs = self.model( 2025-09-07T07:15:30.7621736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7621818Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7622129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7622216Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7622456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7622540Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7622849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7622970Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7623276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7623441Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7623445Z 2025-09-07T07:15:30.7623566Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7623782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7623852Z return mod(**inputs) 2025-09-07T07:15:30.7624161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7624233Z outputs = self.model( 2025-09-07T07:15:30.7624532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7624612Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7624917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7625030Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7625278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7625372Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7625658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7625837Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7626153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7626243Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7626277Z 2025-09-07T07:15:30.7626400Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7626626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7626708Z return mod(**inputs) 2025-09-07T07:15:30.7627000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7627091Z outputs = self.model( 2025-09-07T07:15:30.7627404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7627482Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7627783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7627858Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7628114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7628201Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7628515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7628635Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7628920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7629016Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7629020Z 2025-09-07T07:15:30.7629102Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7629183Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7629269Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7629346Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7629456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7629662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7629728Z return mod(**inputs) 2025-09-07T07:15:30.7630010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7630080Z outputs = self.model( 2025-09-07T07:15:30.7630358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7630431Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7630703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7630784Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7631013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7631101Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7631374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7631535Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7631810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7631911Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7632220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7632355Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7632358Z 2025-09-07T07:15:30.7632469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7632674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7632766Z return mod(**inputs) 2025-09-07T07:15:30.7633037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7633111Z outputs = self.model( 2025-09-07T07:15:30.7633384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7633455Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7633729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7633800Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7634025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7634110Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7634376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7634492Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7634779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7634879Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7635203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7635313Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7635317Z 2025-09-07T07:15:30.7635426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7635636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7635712Z return mod(**inputs) 2025-09-07T07:15:30.7636001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7636075Z outputs = self.model( 2025-09-07T07:15:30.7636376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7636454Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7636749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7636824Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7637063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7637153Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7637439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7637561Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7637847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7637974Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7637978Z 2025-09-07T07:15:30.7638080Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7638282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7638356Z return mod(**inputs) 2025-09-07T07:15:30.7638623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7638700Z outputs = self.model( 2025-09-07T07:15:30.7638968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7639038Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7639333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7639406Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7639640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7639721Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7639988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7640115Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7640119Z 2025-09-07T07:15:30.7640222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7640430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7640496Z return mod(**inputs) 2025-09-07T07:15:30.7640783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7640852Z outputs = self.model( 2025-09-07T07:15:30.7641141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7641223Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7641514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7641597Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7641828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7641906Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7642180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7642299Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7642522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7642596Z return self.act(input) 2025-09-07T07:15:30.7642599Z 2025-09-07T07:15:30.7642709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7642914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7642985Z return mod(**inputs) 2025-09-07T07:15:30.7643279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7643351Z outputs = self.model( 2025-09-07T07:15:30.7643643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7643718Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7644000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7644085Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7644344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7644435Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7644736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7644824Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7644834Z 2025-09-07T07:15:30.7644949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7645150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7645224Z return mod(**inputs) 2025-09-07T07:15:30.7645495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7645591Z outputs = self.model( 2025-09-07T07:15:30.7645866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7645940Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7646221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7646294Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7646530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7646609Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7646878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7646987Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7647261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7647440Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7647446Z 2025-09-07T07:15:30.7647555Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7647792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7647864Z return mod(**inputs) 2025-09-07T07:15:30.7648157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7648233Z outputs = self.model( 2025-09-07T07:15:30.7648509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7648588Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7648863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7648934Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7649176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7649260Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7649545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7649645Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7649924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7650004Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7650008Z 2025-09-07T07:15:30.7650112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7650327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7650393Z return mod(**inputs) 2025-09-07T07:15:30.7650688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7650756Z outputs = self.model( 2025-09-07T07:15:30.7651026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7651105Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7651373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7651451Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7651677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7651754Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7652049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7652152Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7652427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7652514Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7652519Z 2025-09-07T07:15:30.7652607Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7652688Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7652767Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7652851Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7652955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7653167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7653234Z return mod(**inputs) 2025-09-07T07:15:30.7653506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7653602Z outputs = self.model( 2025-09-07T07:15:30.7653873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7653954Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7654240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7654316Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7654550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7654631Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7654911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7655014Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7655300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7655399Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7655692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7655830Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7655833Z 2025-09-07T07:15:30.7655938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7656148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7656216Z return mod(**inputs) 2025-09-07T07:15:30.7656494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7656577Z outputs = self.model( 2025-09-07T07:15:30.7656865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7656970Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7657253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7657337Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7657582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7657664Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7657954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7658061Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7658372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7658485Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7658783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7658904Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7658907Z 2025-09-07T07:15:30.7659011Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7659220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7659287Z return mod(**inputs) 2025-09-07T07:15:30.7659562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7659634Z outputs = self.model( 2025-09-07T07:15:30.7659922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7660009Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7660316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7660399Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7660654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7660741Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7661033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7661138Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7661429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7661519Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7661523Z 2025-09-07T07:15:30.7661640Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7661856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7661927Z return mod(**inputs) 2025-09-07T07:15:30.7662217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7662290Z outputs = self.model( 2025-09-07T07:15:30.7662579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7662655Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7662943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7663027Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7663262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7663382Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7663647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7663757Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7664036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7664197Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7664201Z 2025-09-07T07:15:30.7664319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7664534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7664627Z return mod(**inputs) 2025-09-07T07:15:30.7664910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7664987Z outputs = self.model( 2025-09-07T07:15:30.7665278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7665354Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7665644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7685306Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7685752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7685854Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7686164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7686307Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7686724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7686828Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7686837Z 2025-09-07T07:15:30.7686972Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7687246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7687328Z return mod(**inputs) 2025-09-07T07:15:30.7687636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7687719Z outputs = self.model( 2025-09-07T07:15:30.7688020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7688109Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7688403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7688499Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7688749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7688848Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7689137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7689258Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7689562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7689654Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7689660Z 2025-09-07T07:15:30.7689755Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7689839Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7689957Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7690035Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7690146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7690373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7690443Z return mod(**inputs) 2025-09-07T07:15:30.7690725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7690798Z outputs = self.model( 2025-09-07T07:15:30.7691073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7691161Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7691462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7691544Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7691777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7691863Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7692145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7692257Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7692528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7692644Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7692947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7693098Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7693102Z 2025-09-07T07:15:30.7693228Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7693448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7693517Z return mod(**inputs) 2025-09-07T07:15:30.7693811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7693892Z outputs = self.model( 2025-09-07T07:15:30.7694164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7694247Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7694519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7694593Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7694834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7694919Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7695196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7695308Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7695576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7695685Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7695983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7696102Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7696108Z 2025-09-07T07:15:30.7696213Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7696427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7696516Z return mod(**inputs) 2025-09-07T07:15:30.7696790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7696871Z outputs = self.model( 2025-09-07T07:15:30.7697146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7697232Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7697521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7697599Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7697847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7697952Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7698245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7698361Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7698651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7698735Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7698739Z 2025-09-07T07:15:30.7698842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7699054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7699120Z return mod(**inputs) 2025-09-07T07:15:30.7699395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7699463Z outputs = self.model( 2025-09-07T07:15:30.7699745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7699828Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7700110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7700192Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7700432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7700515Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7700805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7700938Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7700943Z 2025-09-07T07:15:30.7701062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7701285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7701365Z return mod(**inputs) 2025-09-07T07:15:30.7701654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7701728Z outputs = self.model( 2025-09-07T07:15:30.7702021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7702099Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7702397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7702475Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7702720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7702829Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7703144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7703276Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7703511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7703594Z return self.act(input) 2025-09-07T07:15:30.7703598Z 2025-09-07T07:15:30.7703710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7703922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7704001Z return mod(**inputs) 2025-09-07T07:15:30.7704286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7704382Z outputs = self.model( 2025-09-07T07:15:30.7704670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7704749Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7705044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7705124Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7705371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7705457Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7705840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7705949Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7705956Z 2025-09-07T07:15:30.7706072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7706303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7706401Z return mod(**inputs) 2025-09-07T07:15:30.7706721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7706793Z outputs = self.model( 2025-09-07T07:15:30.7707096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7707185Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7707472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7707562Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7707802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7707890Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7708188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:15:30.7708276Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7708280Z 2025-09-07T07:15:30.7708401Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7708617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7708699Z return mod(**inputs) 2025-09-07T07:15:30.7709001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7709072Z outputs = self.model( 2025-09-07T07:15:30.7709364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7709443Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7709735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7709831Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7710069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7710162Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7710447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7710562Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7710848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7711024Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7711050Z 2025-09-07T07:15:30.7711164Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7711379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7711460Z return mod(**inputs) 2025-09-07T07:15:30.7711747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7711829Z outputs = self.model( 2025-09-07T07:15:30.7712129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7712206Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7712513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7712589Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7712836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7712922Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7713223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7713340Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7713640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7713732Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7713736Z 2025-09-07T07:15:30.7713837Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7714045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7714110Z return mod(**inputs) 2025-09-07T07:15:30.7714380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7714458Z outputs = self.model( 2025-09-07T07:15:30.7714727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7714806Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7715077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7715150Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7715380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7715460Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7715734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7715833Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7716108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7716213Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7716217Z 2025-09-07T07:15:30.7716299Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7716388Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7716466Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7716551Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7716656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7716858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7716933Z return mod(**inputs) 2025-09-07T07:15:30.7717201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7717277Z outputs = self.model( 2025-09-07T07:15:30.7717589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7717667Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7717973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7718045Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7718278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7718358Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7718633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7718741Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7719009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7719118Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7719439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7719728Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7719734Z 2025-09-07T07:15:30.7719845Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7720122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7720204Z return mod(**inputs) 2025-09-07T07:15:30.7720479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7720557Z outputs = self.model( 2025-09-07T07:15:30.7720831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7720908Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7721190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7721263Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7721495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7721578Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7721865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7721966Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7722238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7722348Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7722653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7722812Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7722816Z 2025-09-07T07:15:30.7722921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7723123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7723198Z return mod(**inputs) 2025-09-07T07:15:30.7723467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7723544Z outputs = self.model( 2025-09-07T07:15:30.7723814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7723894Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7724166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7724267Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7724506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7724586Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7724865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7724964Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7725233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7725322Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7725326Z 2025-09-07T07:15:30.7725430Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7725639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7725709Z return mod(**inputs) 2025-09-07T07:15:30.7726012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7726084Z outputs = self.model( 2025-09-07T07:15:30.7726395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7726478Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7726754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7726833Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7727063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7727144Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7727427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7727539Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7727815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7727973Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7727977Z 2025-09-07T07:15:30.7728088Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7728293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7728360Z return mod(**inputs) 2025-09-07T07:15:30.7728641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7728708Z outputs = self.model( 2025-09-07T07:15:30.7728990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7729091Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7729362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7729441Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7729667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7729752Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7730026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7730132Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7730402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7730500Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7730504Z 2025-09-07T07:15:30.7730617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7730813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7730884Z return mod(**inputs) 2025-09-07T07:15:30.7731147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7731213Z outputs = self.model( 2025-09-07T07:15:30.7731480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7731550Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7731818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7731891Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7732112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7732213Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7732474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7732605Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7732875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7732969Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7732972Z 2025-09-07T07:15:30.7733054Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7733136Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7733226Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7733304Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7733417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7733620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7733689Z return mod(**inputs) 2025-09-07T07:15:30.7733966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7734036Z outputs = self.model( 2025-09-07T07:15:30.7734308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7734383Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7734651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7734730Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7734967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7735054Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7735317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7735442Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7735713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7735808Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7736103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7736237Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7736241Z 2025-09-07T07:15:30.7736350Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7736562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7736628Z return mod(**inputs) 2025-09-07T07:15:30.7736901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7736969Z outputs = self.model( 2025-09-07T07:15:30.7737238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7737311Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7737740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7737824Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7738048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7738136Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7738405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7738543Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7738808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7738921Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7739226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7739333Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7739337Z 2025-09-07T07:15:30.7739449Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7739654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7739725Z return mod(**inputs) 2025-09-07T07:15:30.7740008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7740079Z outputs = self.model( 2025-09-07T07:15:30.7740356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7740430Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7740711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7740783Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7741012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7741100Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7741371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7741486Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7741760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7741861Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7741873Z 2025-09-07T07:15:30.7741981Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7742196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7742276Z return mod(**inputs) 2025-09-07T07:15:30.7742560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7742640Z outputs = self.model( 2025-09-07T07:15:30.7742924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7743021Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7743315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7743393Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7743641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7743727Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7744017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7744156Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7744160Z 2025-09-07T07:15:30.7744269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7744487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7744560Z return mod(**inputs) 2025-09-07T07:15:30.7744852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7744963Z outputs = self.model( 2025-09-07T07:15:30.7745250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7745334Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7745634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7745781Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7746030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7746113Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7746406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7746539Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7746784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7746863Z return self.act(input) 2025-09-07T07:15:30.7746867Z 2025-09-07T07:15:30.7746980Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7747215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7747284Z return mod(**inputs) 2025-09-07T07:15:30.7747572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7747643Z outputs = self.model( 2025-09-07T07:15:30.7747935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7748014Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7748309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7748422Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7748659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7748744Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7749034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7749123Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7749127Z 2025-09-07T07:15:30.7749245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7749456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7749525Z return mod(**inputs) 2025-09-07T07:15:30.7749834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7749910Z outputs = self.model( 2025-09-07T07:15:30.7750202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7750279Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7750570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7750647Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7750886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7750976Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7751259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7751372Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7751682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7751851Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7751855Z 2025-09-07T07:15:30.7751976Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7752208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7752287Z return mod(**inputs) 2025-09-07T07:15:30.7752573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7752650Z outputs = self.model( 2025-09-07T07:15:30.7752938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7753017Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7753313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7753390Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7753633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7753717Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7754001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7754115Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7754399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7754491Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7754496Z 2025-09-07T07:15:30.7754604Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7754821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7754913Z return mod(**inputs) 2025-09-07T07:15:30.7755198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7755277Z outputs = self.model( 2025-09-07T07:15:30.7755561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7755642Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7755923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7755998Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7756243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7756343Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7756641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7756748Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7757033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7757136Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7757140Z 2025-09-07T07:15:30.7757225Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7757317Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7757402Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7757483Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7757601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7757803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7757877Z return mod(**inputs) 2025-09-07T07:15:30.7758163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7758241Z outputs = self.model( 2025-09-07T07:15:30.7758533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7758608Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7758883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7758955Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7759187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7759267Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7759536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7759648Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7759917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7760025Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7760328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7760478Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7760481Z 2025-09-07T07:15:30.7760587Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7760790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7760871Z return mod(**inputs) 2025-09-07T07:15:30.7761142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7761236Z outputs = self.model( 2025-09-07T07:15:30.7761510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7761584Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7761876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7761953Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7762204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7762289Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7762581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7762712Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7762999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7763109Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7763425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7763547Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7763551Z 2025-09-07T07:15:30.7763657Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7763874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7763948Z return mod(**inputs) 2025-09-07T07:15:30.7764218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7764293Z outputs = self.model( 2025-09-07T07:15:30.7764578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7764654Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7764932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7765020Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7765258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7765338Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7765616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7765716Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7765984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7766073Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7766080Z 2025-09-07T07:15:30.7766183Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7766392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7766461Z return mod(**inputs) 2025-09-07T07:15:30.7766725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7766802Z outputs = self.model( 2025-09-07T07:15:30.7767071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7767150Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7767417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7767498Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7767743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7767825Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7768101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-09-07T07:15:30.7768182Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7768185Z 2025-09-07T07:15:30.7768296Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7768495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7768562Z return mod(**inputs) 2025-09-07T07:15:30.7768848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7768936Z outputs = self.model( 2025-09-07T07:15:30.7769214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7769287Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7769556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7769640Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7769863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7769947Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7770216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7770333Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7770605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7770774Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7770780Z 2025-09-07T07:15:30.7770890Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7771091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7771181Z return mod(**inputs) 2025-09-07T07:15:30.7771462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7771528Z outputs = self.model( 2025-09-07T07:15:30.7771799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7771870Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7772143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7772220Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7772455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7772534Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7772837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7772955Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7773230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7773319Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7773322Z 2025-09-07T07:15:30.7773427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7773631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7773704Z return mod(**inputs) 2025-09-07T07:15:30.7773982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7774073Z outputs = self.model( 2025-09-07T07:15:30.7774342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7774422Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7774690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7774760Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7774991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7775066Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7775371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7775482Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7775752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7775846Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7775851Z 2025-09-07T07:15:30.7775932Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7776019Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7776096Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7776172Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7776284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7776481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7776558Z return mod(**inputs) 2025-09-07T07:15:30.7776833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7776920Z outputs = self.model( 2025-09-07T07:15:30.7777198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7777271Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7777563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7777636Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7777870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7777957Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7778220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7778337Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7778606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7778713Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7779017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7779149Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7779160Z 2025-09-07T07:15:30.7779261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7779459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7779532Z return mod(**inputs) 2025-09-07T07:15:30.7779799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7779875Z outputs = self.model( 2025-09-07T07:15:30.7780146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7780237Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7780516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7780588Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7780820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7780900Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7781168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7781285Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7781571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7781680Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7781983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7782100Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7782103Z 2025-09-07T07:15:30.7782208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7782410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7782484Z return mod(**inputs) 2025-09-07T07:15:30.7782757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7782837Z outputs = self.model( 2025-09-07T07:15:30.7783126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7783205Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7783521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7783600Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7783858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7783945Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7784235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7784350Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7784634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7784729Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7784733Z 2025-09-07T07:15:30.7784844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7785063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7785132Z return mod(**inputs) 2025-09-07T07:15:30.7785416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7785494Z outputs = self.model( 2025-09-07T07:15:30.7785852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7785944Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7786231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7786313Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7786561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7786673Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7786962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7787092Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7787096Z 2025-09-07T07:15:30.7787214Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7787427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7787498Z return mod(**inputs) 2025-09-07T07:15:30.7787789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7787861Z outputs = self.model( 2025-09-07T07:15:30.7788197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7788277Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7788575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7788654Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7788885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7788970Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7789242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7789377Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7789609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7789688Z return self.act(input) 2025-09-07T07:15:30.7789695Z 2025-09-07T07:15:30.7789811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7790047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7790126Z return mod(**inputs) 2025-09-07T07:15:30.7790429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7790503Z outputs = self.model( 2025-09-07T07:15:30.7790806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7790887Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7791179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7791255Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7791495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7791589Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7791872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7791968Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7791974Z 2025-09-07T07:15:30.7792084Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7792303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7792373Z return mod(**inputs) 2025-09-07T07:15:30.7792657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7792736Z outputs = self.model( 2025-09-07T07:15:30.7793026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7793112Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7793419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7793498Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7793745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7793830Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7794119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7794226Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7794517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7794702Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7794706Z 2025-09-07T07:15:30.7794817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7795039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7795108Z return mod(**inputs) 2025-09-07T07:15:30.7795402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7795474Z outputs = self.model( 2025-09-07T07:15:30.7795758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7795841Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7796127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7796213Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7796448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7796558Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7796842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7796964Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7797258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7797343Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7797346Z 2025-09-07T07:15:30.7797463Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7797677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7797747Z return mod(**inputs) 2025-09-07T07:15:30.7798043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7798116Z outputs = self.model( 2025-09-07T07:15:30.7798410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7798488Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7798781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7798865Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7799101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7799193Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7799476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7799589Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7799871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7799989Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7799993Z 2025-09-07T07:15:30.7800088Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7800174Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7800262Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7800342Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7800453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7800674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7800744Z return mod(**inputs) 2025-09-07T07:15:30.7801030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7801118Z outputs = self.model( 2025-09-07T07:15:30.7801404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7801491Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7801773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7801854Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7802090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7802180Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7802461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7802565Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7802854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7802972Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7803297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7803442Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7803463Z 2025-09-07T07:15:30.7803574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7803794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7803866Z return mod(**inputs) 2025-09-07T07:15:30.7804162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7804236Z outputs = self.model( 2025-09-07T07:15:30.7804531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7804609Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7804897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7804981Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7805219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7805309Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7805591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7805694Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7805984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7806088Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7806409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7806547Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7806550Z 2025-09-07T07:15:30.7806675Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7806875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7806943Z return mod(**inputs) 2025-09-07T07:15:30.7807219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7807287Z outputs = self.model( 2025-09-07T07:15:30.7807562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7807653Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7807927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7808014Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7808251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7808345Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7808631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7808734Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7809023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7809110Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7809114Z 2025-09-07T07:15:30.7809233Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7809444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7809544Z return mod(**inputs) 2025-09-07T07:15:30.7809846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7809916Z outputs = self.model( 2025-09-07T07:15:30.7810217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7810292Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7810576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7810649Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7810876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7810964Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7811242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7811366Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7811653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7811824Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7811828Z 2025-09-07T07:15:30.7811939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7812153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7812231Z return mod(**inputs) 2025-09-07T07:15:30.7812515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7812596Z outputs = self.model( 2025-09-07T07:15:30.7812882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7812979Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7813273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7813352Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7813610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7813689Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7813963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7814073Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7814359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7814449Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7814453Z 2025-09-07T07:15:30.7814558Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7814767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7814835Z return mod(**inputs) 2025-09-07T07:15:30.7815104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7815182Z outputs = self.model( 2025-09-07T07:15:30.7815451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7815529Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7815798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7815871Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7816125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7816205Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7816493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7816603Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7816876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7816963Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7816966Z 2025-09-07T07:15:30.7817047Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7817138Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7817219Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7817306Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7817413Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7817618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7817694Z return mod(**inputs) 2025-09-07T07:15:30.7817969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7818044Z outputs = self.model( 2025-09-07T07:15:30.7818316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7818391Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7818664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7818739Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7818970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7819091Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7819383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7819499Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7819999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7820117Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7820434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7820586Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7820644Z 2025-09-07T07:15:30.7820759Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7820984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7821070Z return mod(**inputs) 2025-09-07T07:15:30.7821366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7821450Z outputs = self.model( 2025-09-07T07:15:30.7821744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7821834Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7822127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7822209Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7822460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7822550Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7822885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7823005Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7823328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7823442Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7823754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7823877Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7823881Z 2025-09-07T07:15:30.7823991Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7824214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7824284Z return mod(**inputs) 2025-09-07T07:15:30.7824574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7824654Z outputs = self.model( 2025-09-07T07:15:30.7824940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7825025Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7825309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7825385Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7825633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7825765Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7826069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7826220Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7826505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7826601Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7826607Z 2025-09-07T07:15:30.7826718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7826940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7827011Z return mod(**inputs) 2025-09-07T07:15:30.7827304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7827376Z outputs = self.model( 2025-09-07T07:15:30.7827682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7827766Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7828056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7828138Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7828379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7828463Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7828756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-09-07T07:15:30.7828842Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7828846Z 2025-09-07T07:15:30.7828964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7829180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7829256Z return mod(**inputs) 2025-09-07T07:15:30.7829557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7829630Z outputs = self.model( 2025-09-07T07:15:30.7829939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7830018Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7830310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7830386Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7830626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7830720Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7831004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7831147Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7831151Z 2025-09-07T07:15:30.7831261Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7831491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7831564Z return mod(**inputs) 2025-09-07T07:15:30.7831866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7831946Z outputs = self.model( 2025-09-07T07:15:30.7832232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7832316Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7832617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7832694Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7832969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7833053Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7833359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7833485Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7833718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7833800Z return self.act(input) 2025-09-07T07:15:30.7833808Z 2025-09-07T07:15:30.7833918Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7834136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7834235Z return mod(**inputs) 2025-09-07T07:15:30.7834513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7834582Z outputs = self.model( 2025-09-07T07:15:30.7834853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7834936Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7835207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7835287Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7835512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7835591Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7835871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7835952Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7835976Z 2025-09-07T07:15:30.7836089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7836291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7836381Z return mod(**inputs) 2025-09-07T07:15:30.7836650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7836719Z outputs = self.model( 2025-09-07T07:15:30.7836998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7837071Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7837348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7837423Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7837650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7837738Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7838010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7838119Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7838389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7838547Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7838558Z 2025-09-07T07:15:30.7838662Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7838862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7838939Z return mod(**inputs) 2025-09-07T07:15:30.7839218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7839319Z outputs = self.model( 2025-09-07T07:15:30.7839593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7839667Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7839948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7840021Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7840253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7840333Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7840620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7840730Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7840995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7841082Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7841088Z 2025-09-07T07:15:30.7841191Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7841396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7841462Z return mod(**inputs) 2025-09-07T07:15:30.7841745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7841825Z outputs = self.model( 2025-09-07T07:15:30.7842108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7842192Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7842495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7842573Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7842834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7842919Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7843209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7843316Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7843609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7843698Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7843701Z 2025-09-07T07:15:30.7843783Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7843875Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7843955Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7844041Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7844145Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7844348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7844423Z return mod(**inputs) 2025-09-07T07:15:30.7844700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7844779Z outputs = self.model( 2025-09-07T07:15:30.7845068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7845149Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7845451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7845552Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7845802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7845888Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7846175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7846282Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7846550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7846654Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7846951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7847134Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7847139Z 2025-09-07T07:15:30.7847243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7847441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7847515Z return mod(**inputs) 2025-09-07T07:15:30.7847784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7847859Z outputs = self.model( 2025-09-07T07:15:30.7848125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7848198Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7848473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7848546Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7848802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7848881Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7849173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7849270Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7849536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7849639Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7849936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7850052Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7850055Z 2025-09-07T07:15:30.7850158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7850360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7850436Z return mod(**inputs) 2025-09-07T07:15:30.7850703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7850777Z outputs = self.model( 2025-09-07T07:15:30.7851059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7851142Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7851425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7851502Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7851745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7851852Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7852142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7852245Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7852528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7852621Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7852625Z 2025-09-07T07:15:30.7852733Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7852951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7853026Z return mod(**inputs) 2025-09-07T07:15:30.7853328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7853407Z outputs = self.model( 2025-09-07T07:15:30.7853696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7853782Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7854067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7854151Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7854388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7854473Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7854766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7854886Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7855197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7855364Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7855368Z 2025-09-07T07:15:30.7855485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7855715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7855788Z return mod(**inputs) 2025-09-07T07:15:30.7856084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7856155Z outputs = self.model( 2025-09-07T07:15:30.7856447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7856528Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7856811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7856896Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7857133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7857227Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7857511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7857627Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7857925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7858012Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7858017Z 2025-09-07T07:15:30.7858137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7858351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7858453Z return mod(**inputs) 2025-09-07T07:15:30.7858737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7858810Z outputs = self.model( 2025-09-07T07:15:30.7859106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7859182Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7859473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7859549Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7859786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7859896Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7860182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7860308Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7860595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7860696Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7860700Z 2025-09-07T07:15:30.7860785Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7860871Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7860964Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7861045Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7861163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7861376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7861447Z return mod(**inputs) 2025-09-07T07:15:30.7861753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7861830Z outputs = self.model( 2025-09-07T07:15:30.7862142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7862223Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7862506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7862590Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7862829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7862922Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7863209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7863331Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7863632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7863737Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7864072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7864213Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7864217Z 2025-09-07T07:15:30.7864331Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7864546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7864620Z return mod(**inputs) 2025-09-07T07:15:30.7864912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7865009Z outputs = self.model( 2025-09-07T07:15:30.7865299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7865377Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7865663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7865816Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7866063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7866156Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7866446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7866598Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7866889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7866997Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7867335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7867451Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7867456Z 2025-09-07T07:15:30.7867574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7867790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7867863Z return mod(**inputs) 2025-09-07T07:15:30.7868155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7868229Z outputs = self.model( 2025-09-07T07:15:30.7868540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7868623Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7868915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7869010Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7869255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7869348Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7869632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7869755Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7870041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7870133Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7870136Z 2025-09-07T07:15:30.7870257Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7870472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7870554Z return mod(**inputs) 2025-09-07T07:15:30.7870838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7870917Z outputs = self.model( 2025-09-07T07:15:30.7871199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7871277Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7871568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7871646Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7871914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7871997Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7872284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7872420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7872424Z 2025-09-07T07:15:30.7872533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7872751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7872820Z return mod(**inputs) 2025-09-07T07:15:30.7873122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7873230Z outputs = self.model( 2025-09-07T07:15:30.7873536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7873628Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7873931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7874014Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7874253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7874335Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7874629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7874756Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7874996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7875070Z return self.act(input) 2025-09-07T07:15:30.7875098Z 2025-09-07T07:15:30.7875213Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7875434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7875503Z return mod(**inputs) 2025-09-07T07:15:30.7875823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7875907Z outputs = self.model( 2025-09-07T07:15:30.7876185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7876260Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7876532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7876616Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7876843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7876931Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7877201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7877286Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7877290Z 2025-09-07T07:15:30.7877402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7877603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7877677Z return mod(**inputs) 2025-09-07T07:15:30.7877948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7878018Z outputs = self.model( 2025-09-07T07:15:30.7878297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7878675Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7878957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7879036Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7879281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7879364Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7879646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:15:30.7879738Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7879741Z 2025-09-07T07:15:30.7879869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7880091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7880165Z return mod(**inputs) 2025-09-07T07:15:30.7880452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7880536Z outputs = self.model( 2025-09-07T07:15:30.7880824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7880908Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7881207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7881291Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7881528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7881612Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7881924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7882036Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7882329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7882517Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7882521Z 2025-09-07T07:15:30.7882631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7882854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7882918Z return mod(**inputs) 2025-09-07T07:15:30.7883199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7883268Z outputs = self.model( 2025-09-07T07:15:30.7883544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7883618Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7883886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7883967Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7884191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7884277Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7884546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7884648Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7884925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7885007Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7885030Z 2025-09-07T07:15:30.7885141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7885348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7885417Z return mod(**inputs) 2025-09-07T07:15:30.7885693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7885760Z outputs = self.model( 2025-09-07T07:15:30.7886038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7886110Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7886391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7886482Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7886706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7886793Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7887080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7887194Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7887479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7887570Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7887582Z 2025-09-07T07:15:30.7887668Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7887754Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7887845Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7887925Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7888035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7888279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7888351Z return mod(**inputs) 2025-09-07T07:15:30.7888662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7888733Z outputs = self.model( 2025-09-07T07:15:30.7889026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7889101Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7889386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7889468Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7889709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7889804Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7890090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7890195Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7890496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7890600Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7890931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7891073Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7891079Z 2025-09-07T07:15:30.7891197Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7891416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7891510Z return mod(**inputs) 2025-09-07T07:15:30.7891803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7891875Z outputs = self.model( 2025-09-07T07:15:30.7892166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7892244Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7892526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7892609Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7892847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7892959Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7893247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7893353Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7893644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7893748Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7894067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7894185Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7894188Z 2025-09-07T07:15:30.7894306Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7894522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7894595Z return mod(**inputs) 2025-09-07T07:15:30.7894909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7894982Z outputs = self.model( 2025-09-07T07:15:30.7895290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7895371Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7895656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7895739Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7895982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7896072Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7896364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7896474Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7896760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7896847Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7896850Z 2025-09-07T07:15:30.7896968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7897179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7897256Z return mod(**inputs) 2025-09-07T07:15:30.7897539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7897610Z outputs = self.model( 2025-09-07T07:15:30.7897901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7897978Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7898301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7898379Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7898620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7898709Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7898993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7899117Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7899405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7899592Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7899596Z 2025-09-07T07:15:30.7899706Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7899920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7899998Z return mod(**inputs) 2025-09-07T07:15:30.7900285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7900365Z outputs = self.model( 2025-09-07T07:15:30.7900656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7900734Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7901034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7901114Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7901363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7901469Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7901767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7901884Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7902192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7902289Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7902293Z 2025-09-07T07:15:30.7902407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7902635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7902708Z return mod(**inputs) 2025-09-07T07:15:30.7903004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7903089Z outputs = self.model( 2025-09-07T07:15:30.7903383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7903472Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7903768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7903855Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7904101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7904186Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7904490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7904608Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7904909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7905026Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7905030Z 2025-09-07T07:15:30.7905120Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7905217Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7905305Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7905397Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7905511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7905809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7905900Z return mod(**inputs) 2025-09-07T07:15:30.7906199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7906311Z outputs = self.model( 2025-09-07T07:15:30.7906610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7906702Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7907004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7907083Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7907338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7907421Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7907714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7907828Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7908114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7908248Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7908648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7908790Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7908809Z 2025-09-07T07:15:30.7908915Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7909123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7909192Z return mod(**inputs) 2025-09-07T07:15:30.7909461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7909539Z outputs = self.model( 2025-09-07T07:15:30.7909828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7909915Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7910204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7910281Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7910530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7910616Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7910911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7911025Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7911310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7911423Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7911739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7911885Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7911889Z 2025-09-07T07:15:30.7912006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7912215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7912281Z return mod(**inputs) 2025-09-07T07:15:30.7912547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7912621Z outputs = self.model( 2025-09-07T07:15:30.7912888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7912986Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7913262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7913337Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7913571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7913652Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7913925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7914033Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7914306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7914387Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7914392Z 2025-09-07T07:15:30.7914495Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7914703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7914786Z return mod(**inputs) 2025-09-07T07:15:30.7915065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7915133Z outputs = self.model( 2025-09-07T07:15:30.7915423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7915504Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7915777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7915854Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7916078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7916167Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7916437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7916561Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7916564Z 2025-09-07T07:15:30.7916677Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7916882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7916957Z return mod(**inputs) 2025-09-07T07:15:30.7917227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7917296Z outputs = self.model( 2025-09-07T07:15:30.7917575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7917650Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7917933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7918025Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7918253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7918341Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7918618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7918746Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7918965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7919043Z return self.act(input) 2025-09-07T07:15:30.7919046Z 2025-09-07T07:15:30.7919171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7919374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7919452Z return mod(**inputs) 2025-09-07T07:15:30.7919867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7919949Z outputs = self.model( 2025-09-07T07:15:30.7920226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7920302Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7920579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7920655Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7920889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7920969Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7921297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7921388Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7921392Z 2025-09-07T07:15:30.7921503Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7921762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7921834Z return mod(**inputs) 2025-09-07T07:15:30.7922122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7922190Z outputs = self.model( 2025-09-07T07:15:30.7922461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7922545Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7922818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7922903Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7923131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7923212Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7923490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7923593Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7923870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7924024Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7924029Z 2025-09-07T07:15:30.7924142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7924344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7924442Z return mod(**inputs) 2025-09-07T07:15:30.7924720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7924787Z outputs = self.model( 2025-09-07T07:15:30.7925062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7925136Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7925403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7925482Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7925704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7925818Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7926087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7926197Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7926466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7926545Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7926549Z 2025-09-07T07:15:30.7926661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7926860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7926932Z return mod(**inputs) 2025-09-07T07:15:30.7927205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7927273Z outputs = self.model( 2025-09-07T07:15:30.7927578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7927656Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7927931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7928020Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7928251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7928330Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7928596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7928701Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7928968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7929064Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7929070Z 2025-09-07T07:15:30.7929153Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7929235Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7929322Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7929399Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7929510Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7929715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7929781Z return mod(**inputs) 2025-09-07T07:15:30.7930057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7930124Z outputs = self.model( 2025-09-07T07:15:30.7930401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7930477Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7930769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7930848Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7931073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7931161Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7931431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7931538Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7931808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7931926Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7932236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7932373Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7932376Z 2025-09-07T07:15:30.7932486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7932690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7932758Z return mod(**inputs) 2025-09-07T07:15:30.7933033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7933103Z outputs = self.model( 2025-09-07T07:15:30.7933380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7933456Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7933730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7933822Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7934049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7934137Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7934423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7934532Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7934803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7934900Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7935207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7935319Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7935325Z 2025-09-07T07:15:30.7935435Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7935633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7935707Z return mod(**inputs) 2025-09-07T07:15:30.7935969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7936037Z outputs = self.model( 2025-09-07T07:15:30.7936304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7936379Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7936650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7936723Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7936941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7937052Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7937316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7937417Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7937676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7937762Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7937765Z 2025-09-07T07:15:30.7937866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7938062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7938153Z return mod(**inputs) 2025-09-07T07:15:30.7938418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7938492Z outputs = self.model( 2025-09-07T07:15:30.7938754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7938827Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7939100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7939171Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7939402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7939480Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7939749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-09-07T07:15:30.7939841Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7939862Z 2025-09-07T07:15:30.7939973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7940196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7940267Z return mod(**inputs) 2025-09-07T07:15:30.7940587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7940662Z outputs = self.model( 2025-09-07T07:15:30.7940962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7941046Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7941344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7941430Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7941672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7941757Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7942052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7942171Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7942463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7942627Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7942632Z 2025-09-07T07:15:30.7942747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7942961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7943033Z return mod(**inputs) 2025-09-07T07:15:30.7943333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7943422Z outputs = self.model( 2025-09-07T07:15:30.7943726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7943803Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7944085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7944168Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7944405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7944496Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7944798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7944922Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7945210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7945295Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7945299Z 2025-09-07T07:15:30.7945415Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7945627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7945774Z return mod(**inputs) 2025-09-07T07:15:30.7946071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7946143Z outputs = self.model( 2025-09-07T07:15:30.7946436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7946518Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7946846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7946925Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7947180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7947272Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7947555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7947678Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7947968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7948067Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7948071Z 2025-09-07T07:15:30.7948154Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7948239Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7948327Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7948404Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7948522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7948718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7948785Z return mod(**inputs) 2025-09-07T07:15:30.7949053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7949119Z outputs = self.model( 2025-09-07T07:15:30.7949404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7949485Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7949769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7949893Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7950129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7950222Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7950505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7950627Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7950911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7951015Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7951342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7951493Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7951498Z 2025-09-07T07:15:30.7951606Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7951806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7951879Z return mod(**inputs) 2025-09-07T07:15:30.7952174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7952246Z outputs = self.model( 2025-09-07T07:15:30.7952541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7952619Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7952910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7952988Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7953250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7953341Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7953641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7953764Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7954052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7954157Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7954481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7954597Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7954601Z 2025-09-07T07:15:30.7954717Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7954936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7955018Z return mod(**inputs) 2025-09-07T07:15:30.7955305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7955377Z outputs = self.model( 2025-09-07T07:15:30.7955669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7955746Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7956034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7956113Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7956350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7956464Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7956748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7956871Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7957154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7957242Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7957254Z 2025-09-07T07:15:30.7957364Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7957579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7957655Z return mod(**inputs) 2025-09-07T07:15:30.7957957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7958041Z outputs = self.model( 2025-09-07T07:15:30.7958323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7958400Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7958693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7958770Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7959012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7959099Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7959357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7959481Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7959484Z 2025-09-07T07:15:30.7959584Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7959804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7959873Z return mod(**inputs) 2025-09-07T07:15:30.7960169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7960237Z outputs = self.model( 2025-09-07T07:15:30.7960505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7960585Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7960853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7960933Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7961156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7961237Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7961512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.7961632Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.7961858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.7961926Z return self.act(input) 2025-09-07T07:15:30.7961929Z 2025-09-07T07:15:30.7962038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7962234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7962299Z return mod(**inputs) 2025-09-07T07:15:30.7962569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7962636Z outputs = self.model( 2025-09-07T07:15:30.7962926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7962999Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7963264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7963343Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7963561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7963643Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7963905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.7964001Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.7964005Z 2025-09-07T07:15:30.7964114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7964311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7964384Z return mod(**inputs) 2025-09-07T07:15:30.7964651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7964727Z outputs = self.model( 2025-09-07T07:15:30.7964990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7965062Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7965331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7965404Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7965637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7965715Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7966014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7966131Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7966431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7966606Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7966610Z 2025-09-07T07:15:30.7966722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7966946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7967016Z return mod(**inputs) 2025-09-07T07:15:30.7967314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7967392Z outputs = self.model( 2025-09-07T07:15:30.7967674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7967755Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7968026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7968099Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7968335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7968413Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7968690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7968791Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7969064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7969170Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7969174Z 2025-09-07T07:15:30.7969276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7969488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7969558Z return mod(**inputs) 2025-09-07T07:15:30.7969847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7969918Z outputs = self.model( 2025-09-07T07:15:30.7970208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7970313Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7970584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7970667Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7970904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7970987Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7971285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7971398Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7971679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7971766Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7971770Z 2025-09-07T07:15:30.7971859Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7971940Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7972019Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7972122Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7972227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7972438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7972508Z return mod(**inputs) 2025-09-07T07:15:30.7972805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7972881Z outputs = self.model( 2025-09-07T07:15:30.7973141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7973219Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7973482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7973556Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7973786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7973865Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7974141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7974240Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7974508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7974613Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7974914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7975057Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7975062Z 2025-09-07T07:15:30.7975166Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7975395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7975463Z return mod(**inputs) 2025-09-07T07:15:30.7975732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7975809Z outputs = self.model( 2025-09-07T07:15:30.7976079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7976158Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7976428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7976500Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7976752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7976833Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7977114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7977216Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7977498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7977595Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7977894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7978011Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7978014Z 2025-09-07T07:15:30.7978119Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7978332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7978417Z return mod(**inputs) 2025-09-07T07:15:30.7978690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7978767Z outputs = self.model( 2025-09-07T07:15:30.7979054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7979138Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7979411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7979492Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7979717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7979798Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7980080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.7980181Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.7980460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7980544Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7980547Z 2025-09-07T07:15:30.7980651Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7980868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7980936Z return mod(**inputs) 2025-09-07T07:15:30.7981215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7981287Z outputs = self.model( 2025-09-07T07:15:30.7981566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7981662Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7981934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7982016Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7982243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7982328Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7982597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7982708Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7982986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.7983159Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.7983166Z 2025-09-07T07:15:30.7983278Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7983483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7983560Z return mod(**inputs) 2025-09-07T07:15:30.7983835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7983907Z outputs = self.model( 2025-09-07T07:15:30.7984185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7984261Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7984540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7984617Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7984870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7984963Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7985255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7985375Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7985681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.7985838Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.7985852Z 2025-09-07T07:15:30.7985968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7986203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7986287Z return mod(**inputs) 2025-09-07T07:15:30.7986583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7986669Z outputs = self.model( 2025-09-07T07:15:30.7986965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7987045Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7987339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7987414Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7987661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7987744Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7988041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7988159Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7988456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.7988549Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.7988552Z 2025-09-07T07:15:30.7988634Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7988720Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7988799Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7988875Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.7988985Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7989188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7989258Z return mod(**inputs) 2025-09-07T07:15:30.7989548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7989619Z outputs = self.model( 2025-09-07T07:15:30.7989895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7989968Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7990242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7990312Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7990536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7990623Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7990891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7991008Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7991295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7991397Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7991722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.7991859Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.7991862Z 2025-09-07T07:15:30.7991974Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7992180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7992253Z return mod(**inputs) 2025-09-07T07:15:30.7992524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7992595Z outputs = self.model( 2025-09-07T07:15:30.7992873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7992949Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7993225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7993298Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7993522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7993608Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7993875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7994003Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7994269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.7994373Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.7994691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.7994798Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.7994803Z 2025-09-07T07:15:30.7994913Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7995110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7995181Z return mod(**inputs) 2025-09-07T07:15:30.7995444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7995510Z outputs = self.model( 2025-09-07T07:15:30.7995782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7995872Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7996142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7996214Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7996440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7996518Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7996781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.7996893Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.7997151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.7997240Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.7997243Z 2025-09-07T07:15:30.7997343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7997557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7997631Z return mod(**inputs) 2025-09-07T07:15:30.7997922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.7998004Z outputs = self.model( 2025-09-07T07:15:30.7998273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.7998351Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.7998612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.7998683Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.7998911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.7998990Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.7999262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-09-07T07:15:30.7999342Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.7999345Z 2025-09-07T07:15:30.7999451Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.7999663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.7999729Z return mod(**inputs) 2025-09-07T07:15:30.8000003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8000071Z outputs = self.model( 2025-09-07T07:15:30.8000339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8000422Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8000714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8000798Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8001030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8001116Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8001390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8001512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8001516Z 2025-09-07T07:15:30.8001630Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8001830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8001924Z return mod(**inputs) 2025-09-07T07:15:30.8002202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8002272Z outputs = self.model( 2025-09-07T07:15:30.8002566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8002637Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8002910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8002981Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8003211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8003286Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8003553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8003698Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8003915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.8003991Z return self.act(input) 2025-09-07T07:15:30.8003994Z 2025-09-07T07:15:30.8004115Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8004317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8004388Z return mod(**inputs) 2025-09-07T07:15:30.8004656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8004729Z outputs = self.model( 2025-09-07T07:15:30.8004997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8005070Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8005354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8005428Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8005658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8005736Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8006010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.8006092Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.8006095Z 2025-09-07T07:15:30.8006198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8006404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8006473Z return mod(**inputs) 2025-09-07T07:15:30.8006749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8006844Z outputs = self.model( 2025-09-07T07:15:30.8007115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8007197Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8007476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8007555Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8007773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8007857Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8008118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8008233Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8008504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.8008658Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.8008661Z 2025-09-07T07:15:30.8008773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8008977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8009043Z return mod(**inputs) 2025-09-07T07:15:30.8009329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8009395Z outputs = self.model( 2025-09-07T07:15:30.8009666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8009738Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8010035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8010111Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8010361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8010452Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8010721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8010831Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8011103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.8011189Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.8011195Z 2025-09-07T07:15:30.8011309Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8011518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8011594Z return mod(**inputs) 2025-09-07T07:15:30.8011868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8011941Z outputs = self.model( 2025-09-07T07:15:30.8012222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8012299Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8012578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8012652Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8012889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8012972Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8013256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8013363Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8013633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.8013725Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.8013729Z 2025-09-07T07:15:30.8013810Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8013889Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8013977Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8014052Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8014180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8014380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8014452Z return mod(**inputs) 2025-09-07T07:15:30.8014728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8014796Z outputs = self.model( 2025-09-07T07:15:30.8015073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8015146Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8015420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8015491Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8015716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8015807Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8016091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8016202Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8016471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8016591Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8016898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.8017041Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.8017045Z 2025-09-07T07:15:30.8017163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8017377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8017457Z return mod(**inputs) 2025-09-07T07:15:30.8017743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8017817Z outputs = self.model( 2025-09-07T07:15:30.8018111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8018190Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8018483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8018558Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8018792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8018886Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8019176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8019286Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8019723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8019837Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8020163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.8020280Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.8020284Z 2025-09-07T07:15:30.8020403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8020619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8020698Z return mod(**inputs) 2025-09-07T07:15:30.8021034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8021108Z outputs = self.model( 2025-09-07T07:15:30.8021410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8021490Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8021792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8021872Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8022117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8022211Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8022502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8022620Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8022956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.8023055Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.8023059Z 2025-09-07T07:15:30.8023171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8023416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8023498Z return mod(**inputs) 2025-09-07T07:15:30.8023782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8023861Z outputs = self.model( 2025-09-07T07:15:30.8024145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8024221Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8024516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8024596Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8024843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8024928Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8025220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8025339Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8025622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.8025846Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.8025852Z 2025-09-07T07:15:30.8025970Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8026196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8026306Z return mod(**inputs) 2025-09-07T07:15:30.8026606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8026690Z outputs = self.model( 2025-09-07T07:15:30.8026984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8027070Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8027361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8027459Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8027696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8027800Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8028095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8028212Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8028505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.8028591Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.8028595Z 2025-09-07T07:15:30.8028705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8028926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8028998Z return mod(**inputs) 2025-09-07T07:15:30.8029289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8029363Z outputs = self.model( 2025-09-07T07:15:30.8029696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8029784Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8030070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8030170Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8030411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8030503Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8030788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8030906Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8031211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.8031309Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.8031315Z 2025-09-07T07:15:30.8031411Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8031508Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8031590Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8031679Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8031789Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8032010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8032079Z return mod(**inputs) 2025-09-07T07:15:30.8032368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8032446Z outputs = self.model( 2025-09-07T07:15:30.8032731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8032817Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8033127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8033212Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8033452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8033536Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8033829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8033942Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8034230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8034357Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8034673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.8034826Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.8034830Z 2025-09-07T07:15:30.8034939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8035162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8035233Z return mod(**inputs) 2025-09-07T07:15:30.8035526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8035599Z outputs = self.model( 2025-09-07T07:15:30.8035883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8035969Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8036262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8036361Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8036598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8036681Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8036993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8037107Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8037396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8037499Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8037815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.8037925Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.8037931Z 2025-09-07T07:15:30.8038035Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8038243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8038314Z return mod(**inputs) 2025-09-07T07:15:30.8038603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8038675Z outputs = self.model( 2025-09-07T07:15:30.8038957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8039043Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8039325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8039414Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8039650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8039753Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8040042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8040156Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8040445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.8040533Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.8040538Z 2025-09-07T07:15:30.8040654Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8040867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8040952Z return mod(**inputs) 2025-09-07T07:15:30.8041229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8041298Z outputs = self.model( 2025-09-07T07:15:30.8041573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8041648Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8041916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8041997Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8042223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8042309Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8042579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8042711Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8042743Z 2025-09-07T07:15:30.8042849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8043051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8043142Z return mod(**inputs) 2025-09-07T07:15:30.8043414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8043489Z outputs = self.model( 2025-09-07T07:15:30.8043761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8043835Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8044112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8044186Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8044423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8044502Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8044774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8044904Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8045122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.8045205Z return self.act(input) 2025-09-07T07:15:30.8045209Z 2025-09-07T07:15:30.8045318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8045539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8045611Z return mod(**inputs) 2025-09-07T07:15:30.8045895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8045993Z outputs = self.model( 2025-09-07T07:15:30.8046281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8046368Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8046654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8046730Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8046978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8047062Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8047356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.8047462Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.8047467Z 2025-09-07T07:15:30.8047587Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8047799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8047869Z return mod(**inputs) 2025-09-07T07:15:30.8048161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8048234Z outputs = self.model( 2025-09-07T07:15:30.8048525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8048601Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8048885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8048970Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8049224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8049317Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8049603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-09-07T07:15:30.8049730Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.8049741Z 2025-09-07T07:15:30.8049853Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8050062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8050140Z return mod(**inputs) 2025-09-07T07:15:30.8050422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8050501Z outputs = self.model( 2025-09-07T07:15:30.8050785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8050866Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8051159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8051235Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8051490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8051568Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8051836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8051944Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8052220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.8052399Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.8052423Z 2025-09-07T07:15:30.8052534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8052759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8052830Z return mod(**inputs) 2025-09-07T07:15:30.8053118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8053196Z outputs = self.model( 2025-09-07T07:15:30.8053484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8053563Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8053833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8053928Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8054172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8054259Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8054553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8054659Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8054947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.8055031Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.8055034Z 2025-09-07T07:15:30.8055144Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8055365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8055437Z return mod(**inputs) 2025-09-07T07:15:30.8055743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8055817Z outputs = self.model( 2025-09-07T07:15:30.8056099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8056201Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8056489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8056574Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8056813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8056897Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8057192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8057298Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8057596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.8057688Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.8057692Z 2025-09-07T07:15:30.8057788Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8057875Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8057958Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8058048Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8058159Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8058379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8058451Z return mod(**inputs) 2025-09-07T07:15:30.8058739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8058819Z outputs = self.model( 2025-09-07T07:15:30.8059125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8059211Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8059500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8059576Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8059838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8059924Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8060234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8060358Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8060666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8060771Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8061093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.8061245Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.8061248Z 2025-09-07T07:15:30.8061358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8061576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8061645Z return mod(**inputs) 2025-09-07T07:15:30.8061942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8062024Z outputs = self.model( 2025-09-07T07:15:30.8062322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8062427Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8062717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8062820Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8063062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8063146Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8063435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8063540Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8063831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8063935Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8064253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.8064379Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.8064383Z 2025-09-07T07:15:30.8064494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8064718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8064788Z return mod(**inputs) 2025-09-07T07:15:30.8065095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8065167Z outputs = self.model( 2025-09-07T07:15:30.8065466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8065552Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8065940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8066031Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8066275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8066363Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8066664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8066772Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8067085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.8067204Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.8067208Z 2025-09-07T07:15:30.8067325Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8067543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8067615Z return mod(**inputs) 2025-09-07T07:15:30.8067921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8067996Z outputs = self.model( 2025-09-07T07:15:30.8068308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8068385Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8068684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8068769Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8069011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8069103Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8069403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8069522Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8069828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.8069991Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.8069995Z 2025-09-07T07:15:30.8070110Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8070323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8070399Z return mod(**inputs) 2025-09-07T07:15:30.8070685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8070756Z outputs = self.model( 2025-09-07T07:15:30.8071048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8071125Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8071415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8071493Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8071728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8071820Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8072102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8072226Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8072510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.8072623Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.8072627Z 2025-09-07T07:15:30.8072737Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8072956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8073035Z return mod(**inputs) 2025-09-07T07:15:30.8073331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8073407Z outputs = self.model( 2025-09-07T07:15:30.8073677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8073767Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8074051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8074128Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8074362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8074440Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8074712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8074828Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8075097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.8075190Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.8075194Z 2025-09-07T07:15:30.8075276Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8075366Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8075446Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8075540Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8075653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8075855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8075926Z return mod(**inputs) 2025-09-07T07:15:30.8076212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8076282Z outputs = self.model( 2025-09-07T07:15:30.8076561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8076633Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8076921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8076999Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8077239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8077331Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8077615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8077731Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8077997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8078105Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8078401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.8078540Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.8078543Z 2025-09-07T07:15:30.8078656Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8078880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8078956Z return mod(**inputs) 2025-09-07T07:15:30.8079225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8079293Z outputs = self.model( 2025-09-07T07:15:30.8079567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8079641Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8079916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8079988Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8080237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8080319Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8080588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8080705Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8080974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8081079Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8081374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.8081483Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.8081494Z 2025-09-07T07:15:30.8081600Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8081800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8081891Z return mod(**inputs) 2025-09-07T07:15:30.8082164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8082239Z outputs = self.model( 2025-09-07T07:15:30.8082522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8082599Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8082874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8082946Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8083178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8083259Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8083529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8083644Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8083912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.8084001Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.8084004Z 2025-09-07T07:15:30.8084107Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8084313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8084380Z return mod(**inputs) 2025-09-07T07:15:30.8084651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8084727Z outputs = self.model( 2025-09-07T07:15:30.8085004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8085104Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8085375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8085448Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8085682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8085760Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8086052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8086182Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8086186Z 2025-09-07T07:15:30.8086313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8086534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8086608Z return mod(**inputs) 2025-09-07T07:15:30.8086910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8086982Z outputs = self.model( 2025-09-07T07:15:30.8087290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8087368Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8087654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8087745Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8087967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8088057Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8088342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8088464Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8088704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.8088777Z return self.act(input) 2025-09-07T07:15:30.8088780Z 2025-09-07T07:15:30.8088893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8089103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8089181Z return mod(**inputs) 2025-09-07T07:15:30.8089476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8089551Z outputs = self.model( 2025-09-07T07:15:30.8089835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8089911Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8090190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8090267Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8090504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8090594Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8090878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.8090974Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.8090978Z 2025-09-07T07:15:30.8091087Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8091300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8091404Z return mod(**inputs) 2025-09-07T07:15:30.8091691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8091770Z outputs = self.model( 2025-09-07T07:15:30.8092058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8092142Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8092427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8092502Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8092747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8092849Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8093140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8093248Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8093531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.8093702Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.8093706Z 2025-09-07T07:15:30.8093817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8094036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8094106Z return mod(**inputs) 2025-09-07T07:15:30.8094396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8094470Z outputs = self.model( 2025-09-07T07:15:30.8094753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8094857Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8095156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8095255Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8095496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8095579Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8095873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8095979Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8096272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.8096360Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.8096365Z 2025-09-07T07:15:30.8096484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8096697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8096767Z return mod(**inputs) 2025-09-07T07:15:30.8097058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8097128Z outputs = self.model( 2025-09-07T07:15:30.8097413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8097490Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8097773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8097857Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8098095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8098204Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8098490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8098596Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8098886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.8098977Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.8098981Z 2025-09-07T07:15:30.8099075Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8099159Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8099247Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8099347Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8099456Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8099677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8099747Z return mod(**inputs) 2025-09-07T07:15:30.8100039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8100111Z outputs = self.model( 2025-09-07T07:15:30.8100396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8100480Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8100763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8100843Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8101081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8101165Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8101473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8101578Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8101888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8101995Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8102320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.8102461Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.8102464Z 2025-09-07T07:15:30.8102574Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8102800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8102874Z return mod(**inputs) 2025-09-07T07:15:30.8103165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8103236Z outputs = self.model( 2025-09-07T07:15:30.8103507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8103587Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8103858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8103938Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8104165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8104248Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8104525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8104642Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8104916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8105014Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8105316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.8105426Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.8105429Z 2025-09-07T07:15:30.8105533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8105806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8105910Z return mod(**inputs) 2025-09-07T07:15:30.8106209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8106283Z outputs = self.model( 2025-09-07T07:15:30.8106573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8106665Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8106959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8107048Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8107293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8107389Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8107681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-09-07T07:15:30.8107791Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:15:30.8108121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.8108211Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.8108215Z 2025-09-07T07:15:30.8108355Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8108570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8108642Z return mod(**inputs) 2025-09-07T07:15:30.8108937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8109009Z outputs = self.model( 2025-09-07T07:15:30.8109299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8109379Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8109675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8109755Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8110004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8110098Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8110390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-09-07T07:15:30.8110485Z hidden_states = residual + hidden_states 2025-09-07T07:15:30.8110488Z 2025-09-07T07:15:30.8110601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8110819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8110903Z return mod(**inputs) 2025-09-07T07:15:30.8111200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8111304Z outputs = self.model( 2025-09-07T07:15:30.8111604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8111684Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8111988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8112063Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8112308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8112391Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8112681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8112815Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8113101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-09-07T07:15:30.8113274Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-09-07T07:15:30.8113278Z 2025-09-07T07:15:30.8113390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8113613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8113683Z return mod(**inputs) 2025-09-07T07:15:30.8113966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8114046Z outputs = self.model( 2025-09-07T07:15:30.8114329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8114415Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8114719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8114803Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8115066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8115150Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8115444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8115562Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8115861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-09-07T07:15:30.8115952Z key_states = self.k_proj(current_states) 2025-09-07T07:15:30.8115956Z 2025-09-07T07:15:30.8116068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8116299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8116371Z return mod(**inputs) 2025-09-07T07:15:30.8116673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8116748Z outputs = self.model( 2025-09-07T07:15:30.8117049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8117128Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8117431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8117515Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8117755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8117845Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8118163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8118279Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8118578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-09-07T07:15:30.8118671Z value_states = self.v_proj(current_states) 2025-09-07T07:15:30.8118675Z 2025-09-07T07:15:30.8118767Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8118852Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8118937Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8119027Z cudagraph partition due to non gpu ops 2025-09-07T07:15:30.8119155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8119378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8119450Z return mod(**inputs) 2025-09-07T07:15:30.8119930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8120020Z outputs = self.model( 2025-09-07T07:15:30.8120309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8120398Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8120693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8120784Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8121031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8121120Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8121476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8122298Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8122817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8123212Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8123650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-09-07T07:15:30.8123835Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:15:30.8123844Z 2025-09-07T07:15:30.8123990Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8124290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8124401Z return mod(**inputs) 2025-09-07T07:15:30.8128666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8128800Z outputs = self.model( 2025-09-07T07:15:30.8129122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8129225Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8129538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8129632Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8129884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8129978Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8130279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8130405Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8130803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-09-07T07:15:30.8130915Z attn_output, attn_weights = attention_interface( 2025-09-07T07:15:30.8131247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-09-07T07:15:30.8131370Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:15:30.8131377Z 2025-09-07T07:15:30.8131497Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8131733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8131810Z return mod(**inputs) 2025-09-07T07:15:30.8132154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8132251Z outputs = self.model( 2025-09-07T07:15:30.8132553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8132644Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8132945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8133031Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8133274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8133370Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8133660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-09-07T07:15:30.8133780Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-09-07T07:15:30.8134099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-09-07T07:15:30.8134194Z attn_output = self.out_proj(attn_output) 2025-09-07T07:15:30.8134198Z 2025-09-07T07:15:30.8134319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8135351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8135431Z return mod(**inputs) 2025-09-07T07:15:30.8135726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8135800Z outputs = self.model( 2025-09-07T07:15:30.8136092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8136172Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8136482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8136564Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8136806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8136897Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8137185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8137327Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8137331Z 2025-09-07T07:15:30.8137443Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8137660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8137738Z return mod(**inputs) 2025-09-07T07:15:30.8138026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8138107Z outputs = self.model( 2025-09-07T07:15:30.8138430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8138509Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8138814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8138890Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8139142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8139225Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8139518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-09-07T07:15:30.8139664Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:15:30.8139894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:15:30.8139980Z return self.act(input) 2025-09-07T07:15:30.8139984Z 2025-09-07T07:15:30.8140096Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8140322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8140393Z return mod(**inputs) 2025-09-07T07:15:30.8140692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-09-07T07:15:30.8140773Z outputs = self.model( 2025-09-07T07:15:30.8141070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-09-07T07:15:30.8141155Z decoder_outputs = self.decoder( 2025-09-07T07:15:30.8141457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-09-07T07:15:30.8141560Z layer_outputs = decoder_layer( 2025-09-07T07:15:30.8141803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:15:30.8141887Z return super().__call__(*args, **kwargs) 2025-09-07T07:15:30.8142204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-09-07T07:15:30.8142295Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:15:30.8142299Z 2025-09-07T07:15:30.8142418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8142633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8142704Z return mod(**inputs) 2025-09-07T07:15:30.8143014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1489, in forward 2025-09-07T07:15:30.8143148Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-09-07T07:15:30.8143155Z 2025-09-07T07:15:30.8143273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:15:30.8143487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:15:30.8143558Z return mod(**inputs) 2025-09-07T07:15:30.8143867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1494, in forward 2025-09-07T07:15:30.8144052Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:15:30.8144055Z 2025-09-07T07:15:45.2576559Z Compilation time (from dynamo_timed): 30.286992195 2025-09-07T07:15:45.2582594Z pass 2025-09-07T07:15:45.2583770Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:15:45.2584975Z TIMING: _recursive_pre_grad_passes:0.01455 _recursive_joint_graph_passes:0.78633 _recursive_post_grad_passes:0.16288 async_compile.wait:0.83085 code_gen:14.03269 inductor_compile:17.14991 backend_compile:24.35972 gc:0.00058 entire_frame_compile:30.28699 total_wall_time:30.28699 2025-09-07T07:15:45.2586511Z STATS: call_* op count: 965 | FakeTensorMode.__torch_dispatch__:33293 | FakeTensor.__torch_dispatch__:11091 | ProxyTorchDispatchMode.__torch_dispatch__:12299 2025-09-07T07:15:45.2590988Z Dynamo produced 1 graphs covering 965 ops with 0 graph breaks (0 unique) 2025-09-07T07:15:48.5068272Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:15:48.5070929Z import pynvml # type: ignore[import] 2025-09-07T07:15:51.3458759Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:15:51.3459991Z from pkg_resources import resource_filename 2025-09-07T07:15:52.0296624Z 2025-09-07T07:15:52.0412491Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-09-07T07:15:52.0416189Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-09-07T07:15:53.3026008Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-09-07T07:15:53.3027051Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-09-07T07:15:53.3028404Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-09-07T07:15:53.3029488Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-09-07T07:15:53.4762776Z 2025-09-07T07:15:53.4763267Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:15:53.4779942Z cpu eval RobertaForCausalLM 2025-09-07T07:15:54.0255516Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:15:54.3014410Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:15:54.5674614Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:02.3618973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3619494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3620048Z return mod(**inputs) 2025-09-07T07:16:02.3620519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3620969Z outputs = self.roberta( 2025-09-07T07:16:02.3621402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-09-07T07:16:02.3621870Z embedding_output = self.embeddings( 2025-09-07T07:16:02.3622327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-09-07T07:16:02.3622969Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:16:02.3623666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-09-07T07:16:02.3624542Z mask = input_ids.ne(padding_idx).int() 2025-09-07T07:16:02.3624706Z 2025-09-07T07:16:02.3624816Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3625069Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3625308Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3625544Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3625870Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3626104Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3626334Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3626566Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3626797Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3627089Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3627321Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3627546Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3627796Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3628177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3628509Z return mod(**inputs) 2025-09-07T07:16:02.3628924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3629363Z outputs = self.roberta( 2025-09-07T07:16:02.3629790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-09-07T07:16:02.3630236Z embedding_output = self.embeddings( 2025-09-07T07:16:02.3630684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-09-07T07:16:02.3631279Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:16:02.3632216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-09-07T07:16:02.3632866Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:16:02.3633119Z 2025-09-07T07:16:02.3633238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3633610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3633948Z return mod(**inputs) 2025-09-07T07:16:02.3634337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3634745Z outputs = self.roberta( 2025-09-07T07:16:02.3635128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-09-07T07:16:02.3635542Z embedding_output = self.embeddings( 2025-09-07T07:16:02.3635965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-09-07T07:16:02.3636512Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:16:02.3637160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-09-07T07:16:02.3637787Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:16:02.3638057Z 2025-09-07T07:16:02.3638180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3638596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3638964Z return mod(**inputs) 2025-09-07T07:16:02.3639377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3639828Z outputs = self.roberta( 2025-09-07T07:16:02.3640245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3640679Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3641104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3641545Z layer_outputs = layer_module( 2025-09-07T07:16:02.3641952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3642347Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3642786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3643274Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3643706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3644113Z return func(*args, **kwargs) 2025-09-07T07:16:02.3644529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3644955Z self_outputs = self.self( 2025-09-07T07:16:02.3645350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3645755Z return func(*args, **kwargs) 2025-09-07T07:16:02.3646171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.3646742Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.3647039Z 2025-09-07T07:16:02.3647154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3647572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3647927Z return mod(**inputs) 2025-09-07T07:16:02.3648342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3648762Z outputs = self.roberta( 2025-09-07T07:16:02.3649166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3649595Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3650020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3650441Z layer_outputs = layer_module( 2025-09-07T07:16:02.3650819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3651218Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3651649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3652087Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3652497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3652903Z return func(*args, **kwargs) 2025-09-07T07:16:02.3653317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3653740Z self_outputs = self.self( 2025-09-07T07:16:02.3654130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3654536Z return func(*args, **kwargs) 2025-09-07T07:16:02.3654948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.3655392Z self.key(current_states) 2025-09-07T07:16:02.3655517Z 2025-09-07T07:16:02.3655640Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3656025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3656377Z return mod(**inputs) 2025-09-07T07:16:02.3656781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3657204Z outputs = self.roberta( 2025-09-07T07:16:02.3657611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3658033Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3658474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3658954Z layer_outputs = layer_module( 2025-09-07T07:16:02.3659330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3659717Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3660157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3660601Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3661020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3661432Z return func(*args, **kwargs) 2025-09-07T07:16:02.3661841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3662275Z self_outputs = self.self( 2025-09-07T07:16:02.3662701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3663122Z return func(*args, **kwargs) 2025-09-07T07:16:02.3663552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.3664003Z self.value(current_states) 2025-09-07T07:16:02.3664141Z 2025-09-07T07:16:02.3664229Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3664493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3664892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3665235Z return mod(**inputs) 2025-09-07T07:16:02.3665643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3666309Z outputs = self.roberta( 2025-09-07T07:16:02.3666749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3667184Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3667598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3668005Z layer_outputs = layer_module( 2025-09-07T07:16:02.3668365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3668737Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3669138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3669554Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3669939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3670321Z return func(*args, **kwargs) 2025-09-07T07:16:02.3670758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3671183Z self_outputs = self.self( 2025-09-07T07:16:02.3671582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3671990Z return func(*args, **kwargs) 2025-09-07T07:16:02.3672407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.3672902Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.3673103Z 2025-09-07T07:16:02.3673219Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3673630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3674022Z return mod(**inputs) 2025-09-07T07:16:02.3674427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3674848Z outputs = self.roberta( 2025-09-07T07:16:02.3675258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3675690Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3676119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3676546Z layer_outputs = layer_module( 2025-09-07T07:16:02.3676916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3677311Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3677748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3678209Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3678655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3679065Z return func(*args, **kwargs) 2025-09-07T07:16:02.3679515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.3680013Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.3680507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.3680942Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3681103Z 2025-09-07T07:16:02.3681219Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3681614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3681981Z return mod(**inputs) 2025-09-07T07:16:02.3682407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3682891Z outputs = self.roberta( 2025-09-07T07:16:02.3683301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3683731Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3684157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3684585Z layer_outputs = layer_module( 2025-09-07T07:16:02.3684957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3685350Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3685786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3686250Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3686675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3687102Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3687567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3688083Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3688558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.3688989Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3689165Z 2025-09-07T07:16:02.3689279Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3689672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3690031Z return mod(**inputs) 2025-09-07T07:16:02.3690433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3690860Z outputs = self.roberta( 2025-09-07T07:16:02.3691268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3691695Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3692116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3692537Z layer_outputs = layer_module( 2025-09-07T07:16:02.3692915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3693312Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3693767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3694215Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3694665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3695097Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3695559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3696078Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3696573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.3697032Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.3697429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.3697788Z return self.act(input) 2025-09-07T07:16:02.3697905Z 2025-09-07T07:16:02.3698019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3698384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3698724Z return mod(**inputs) 2025-09-07T07:16:02.3699113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3699513Z outputs = self.roberta( 2025-09-07T07:16:02.3699903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3700304Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3700707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3701111Z layer_outputs = layer_module( 2025-09-07T07:16:02.3701499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3701894Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3702341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3702785Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3703225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3703655Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3704115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.3704663Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.3705168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.3705623Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3705861Z 2025-09-07T07:16:02.3705989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3706383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3706768Z return mod(**inputs) 2025-09-07T07:16:02.3707154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3707560Z outputs = self.roberta( 2025-09-07T07:16:02.3707945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3708350Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3708752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3709181Z layer_outputs = layer_module( 2025-09-07T07:16:02.3709541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3709904Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3710327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3710745Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3711144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3711530Z return func(*args, **kwargs) 2025-09-07T07:16:02.3711920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3712327Z self_outputs = self.self( 2025-09-07T07:16:02.3712705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3713093Z return func(*args, **kwargs) 2025-09-07T07:16:02.3713490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.3714034Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.3714312Z 2025-09-07T07:16:02.3714419Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3714792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3715126Z return mod(**inputs) 2025-09-07T07:16:02.3715529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3715930Z outputs = self.roberta( 2025-09-07T07:16:02.3716319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3716745Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3717148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3717543Z layer_outputs = layer_module( 2025-09-07T07:16:02.3717909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3718270Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3718667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3719068Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3719461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3720040Z return func(*args, **kwargs) 2025-09-07T07:16:02.3720437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3720834Z self_outputs = self.self( 2025-09-07T07:16:02.3721200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3721563Z return func(*args, **kwargs) 2025-09-07T07:16:02.3721950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.3722346Z self.key(current_states) 2025-09-07T07:16:02.3722464Z 2025-09-07T07:16:02.3722576Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3722940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3723275Z return mod(**inputs) 2025-09-07T07:16:02.3723722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3724131Z outputs = self.roberta( 2025-09-07T07:16:02.3724541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3724956Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3725371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3725786Z layer_outputs = layer_module( 2025-09-07T07:16:02.3726152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3726526Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3726947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3727389Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3727792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3728203Z return func(*args, **kwargs) 2025-09-07T07:16:02.3728600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3729014Z self_outputs = self.self( 2025-09-07T07:16:02.3729391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3729778Z return func(*args, **kwargs) 2025-09-07T07:16:02.3730179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.3730589Z self.value(current_states) 2025-09-07T07:16:02.3730721Z 2025-09-07T07:16:02.3730806Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3731060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3731468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3731793Z return mod(**inputs) 2025-09-07T07:16:02.3732177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3732578Z outputs = self.roberta( 2025-09-07T07:16:02.3732972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3733364Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3733761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3734228Z layer_outputs = layer_module( 2025-09-07T07:16:02.3734602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3734995Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3735420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3735853Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3736233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3736604Z return func(*args, **kwargs) 2025-09-07T07:16:02.3736985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3737370Z self_outputs = self.self( 2025-09-07T07:16:02.3737738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3738128Z return func(*args, **kwargs) 2025-09-07T07:16:02.3738529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.3738989Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.3739174Z 2025-09-07T07:16:02.3739282Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3739670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3740008Z return mod(**inputs) 2025-09-07T07:16:02.3740391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3740788Z outputs = self.roberta( 2025-09-07T07:16:02.3741178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3741612Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3742033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3742443Z layer_outputs = layer_module( 2025-09-07T07:16:02.3742796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3743173Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3743602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3744053Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3744470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3744877Z return func(*args, **kwargs) 2025-09-07T07:16:02.3745291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.3745869Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.3746402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.3746838Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3746987Z 2025-09-07T07:16:02.3747095Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3747457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3747786Z return mod(**inputs) 2025-09-07T07:16:02.3748163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3748549Z outputs = self.roberta( 2025-09-07T07:16:02.3748935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3749347Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3749740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3750130Z layer_outputs = layer_module( 2025-09-07T07:16:02.3750470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3750834Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3751228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3751633Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3752030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3752427Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3752861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3753362Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3753812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.3754232Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3754380Z 2025-09-07T07:16:02.3754484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3754841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3755165Z return mod(**inputs) 2025-09-07T07:16:02.3755539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3755922Z outputs = self.roberta( 2025-09-07T07:16:02.3756312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3756715Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3757113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3757506Z layer_outputs = layer_module( 2025-09-07T07:16:02.3757872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3758237Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3758633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3759034Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3759425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3759822Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3760248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3760751Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3761205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.3761630Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.3762015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.3762355Z return self.act(input) 2025-09-07T07:16:02.3762467Z 2025-09-07T07:16:02.3762578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3762950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3763335Z return mod(**inputs) 2025-09-07T07:16:02.3763724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3764129Z outputs = self.roberta( 2025-09-07T07:16:02.3764511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3764910Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3765316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3765719Z layer_outputs = layer_module( 2025-09-07T07:16:02.3766080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3766454Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3766868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3767295Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3767731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3768142Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3768597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.3769088Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.3769552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.3769967Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3770108Z 2025-09-07T07:16:02.3770222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3770587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3770923Z return mod(**inputs) 2025-09-07T07:16:02.3771311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3771713Z outputs = self.roberta( 2025-09-07T07:16:02.3772099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3772499Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3772902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3773307Z layer_outputs = layer_module( 2025-09-07T07:16:02.3773662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3774035Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3774440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3774885Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3775282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3775663Z return func(*args, **kwargs) 2025-09-07T07:16:02.3776048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3776450Z self_outputs = self.self( 2025-09-07T07:16:02.3776822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3777204Z return func(*args, **kwargs) 2025-09-07T07:16:02.3777590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.3778141Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.3778419Z 2025-09-07T07:16:02.3778525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3778895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3779226Z return mod(**inputs) 2025-09-07T07:16:02.3779614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3780007Z outputs = self.roberta( 2025-09-07T07:16:02.3780394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3780800Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3781221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3781647Z layer_outputs = layer_module( 2025-09-07T07:16:02.3782047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3782444Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3782880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3783355Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3783763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3784171Z return func(*args, **kwargs) 2025-09-07T07:16:02.3784597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3785030Z self_outputs = self.self( 2025-09-07T07:16:02.3785431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3785926Z return func(*args, **kwargs) 2025-09-07T07:16:02.3786367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.3786811Z self.key(current_states) 2025-09-07T07:16:02.3786938Z 2025-09-07T07:16:02.3787062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3787467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3787820Z return mod(**inputs) 2025-09-07T07:16:02.3788243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3788680Z outputs = self.roberta( 2025-09-07T07:16:02.3789087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3789507Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3789927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3790390Z layer_outputs = layer_module( 2025-09-07T07:16:02.3790768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3791157Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3791589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3792029Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3792447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3792851Z return func(*args, **kwargs) 2025-09-07T07:16:02.3793258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3793707Z self_outputs = self.self( 2025-09-07T07:16:02.3794103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3794509Z return func(*args, **kwargs) 2025-09-07T07:16:02.3794922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.3795347Z self.value(current_states) 2025-09-07T07:16:02.3795482Z 2025-09-07T07:16:02.3795573Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3795835Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3796228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3796571Z return mod(**inputs) 2025-09-07T07:16:02.3796974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3797397Z outputs = self.roberta( 2025-09-07T07:16:02.3798012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3798450Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3798886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3799319Z layer_outputs = layer_module( 2025-09-07T07:16:02.3799773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3800176Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3800607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3801050Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3801483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3801875Z return func(*args, **kwargs) 2025-09-07T07:16:02.3802269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3802672Z self_outputs = self.self( 2025-09-07T07:16:02.3803044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3803437Z return func(*args, **kwargs) 2025-09-07T07:16:02.3803852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.3804348Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.3804546Z 2025-09-07T07:16:02.3804659Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3805058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3805418Z return mod(**inputs) 2025-09-07T07:16:02.3805836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3806226Z outputs = self.roberta( 2025-09-07T07:16:02.3806613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3807014Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3807409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3807811Z layer_outputs = layer_module( 2025-09-07T07:16:02.3808158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3808551Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3808962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3809383Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3809773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3810147Z return func(*args, **kwargs) 2025-09-07T07:16:02.3810543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.3811005Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.3811466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.3811869Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3812018Z 2025-09-07T07:16:02.3812127Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3812494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3812851Z return mod(**inputs) 2025-09-07T07:16:02.3813233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3813626Z outputs = self.roberta( 2025-09-07T07:16:02.3814029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3814437Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3814839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3815241Z layer_outputs = layer_module( 2025-09-07T07:16:02.3815592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3815967Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3816377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3816797Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3817202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3817634Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3818102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3818619Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3819092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.3819513Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3819828Z 2025-09-07T07:16:02.3819946Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3820342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3820754Z return mod(**inputs) 2025-09-07T07:16:02.3821160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3821584Z outputs = self.roberta( 2025-09-07T07:16:02.3822009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3822448Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3822882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3823317Z layer_outputs = layer_module( 2025-09-07T07:16:02.3823696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3824129Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3824580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3825026Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3825453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3825952Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3826423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3826962Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3827462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.3827928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.3828367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.3828746Z return self.act(input) 2025-09-07T07:16:02.3828867Z 2025-09-07T07:16:02.3828989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3829411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3829759Z return mod(**inputs) 2025-09-07T07:16:02.3830161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3830602Z outputs = self.roberta( 2025-09-07T07:16:02.3831016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3831448Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3831877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3832304Z layer_outputs = layer_module( 2025-09-07T07:16:02.3832682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3833075Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3833499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3833933Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3834339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3834737Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3835177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.3835668Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.3836166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.3836580Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3836726Z 2025-09-07T07:16:02.3836840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3837202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3837541Z return mod(**inputs) 2025-09-07T07:16:02.3837924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3838322Z outputs = self.roberta( 2025-09-07T07:16:02.3838708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3839130Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3839533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3839941Z layer_outputs = layer_module( 2025-09-07T07:16:02.3840303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3840678Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3841081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3841501Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3841896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3842284Z return func(*args, **kwargs) 2025-09-07T07:16:02.3842676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3843088Z self_outputs = self.self( 2025-09-07T07:16:02.3843487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3843873Z return func(*args, **kwargs) 2025-09-07T07:16:02.3844295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.3844832Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.3845107Z 2025-09-07T07:16:02.3845212Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3845584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3845918Z return mod(**inputs) 2025-09-07T07:16:02.3846302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3846701Z outputs = self.roberta( 2025-09-07T07:16:02.3847092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3847486Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3847873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3848256Z layer_outputs = layer_module( 2025-09-07T07:16:02.3848598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3848959Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3849357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3849763Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3850139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3850544Z return func(*args, **kwargs) 2025-09-07T07:16:02.3850933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3851334Z self_outputs = self.self( 2025-09-07T07:16:02.3851726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3852114Z return func(*args, **kwargs) 2025-09-07T07:16:02.3852505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.3852905Z self.key(current_states) 2025-09-07T07:16:02.3853022Z 2025-09-07T07:16:02.3853135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3853524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3853855Z return mod(**inputs) 2025-09-07T07:16:02.3854239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3854637Z outputs = self.roberta( 2025-09-07T07:16:02.3855024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3855420Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3855824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3856229Z layer_outputs = layer_module( 2025-09-07T07:16:02.3856590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3856950Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3857350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3857788Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3858188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3858605Z return func(*args, **kwargs) 2025-09-07T07:16:02.3859035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3859437Z self_outputs = self.self( 2025-09-07T07:16:02.3859807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3860187Z return func(*args, **kwargs) 2025-09-07T07:16:02.3860579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.3860974Z self.value(current_states) 2025-09-07T07:16:02.3861103Z 2025-09-07T07:16:02.3861189Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3861452Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3861845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3862198Z return mod(**inputs) 2025-09-07T07:16:02.3862615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3863042Z outputs = self.roberta( 2025-09-07T07:16:02.3863448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3863883Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3864299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3864736Z layer_outputs = layer_module( 2025-09-07T07:16:02.3865117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3865535Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3866053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3866530Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3866961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3867380Z return func(*args, **kwargs) 2025-09-07T07:16:02.3867828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3868258Z self_outputs = self.self( 2025-09-07T07:16:02.3868685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3869090Z return func(*args, **kwargs) 2025-09-07T07:16:02.3869514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.3870008Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.3870206Z 2025-09-07T07:16:02.3870318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3870711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3871071Z return mod(**inputs) 2025-09-07T07:16:02.3871480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3871905Z outputs = self.roberta( 2025-09-07T07:16:02.3872324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3872754Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3873196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3873645Z layer_outputs = layer_module( 2025-09-07T07:16:02.3874026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3874422Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3874852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3875289Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3875704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3876104Z return func(*args, **kwargs) 2025-09-07T07:16:02.3876523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.3877016Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.3877507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.3877938Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3878100Z 2025-09-07T07:16:02.3878215Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3878615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3878949Z return mod(**inputs) 2025-09-07T07:16:02.3879332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3879727Z outputs = self.roberta( 2025-09-07T07:16:02.3880111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3880511Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3880932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3881337Z layer_outputs = layer_module( 2025-09-07T07:16:02.3881687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3882060Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3882493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3882943Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3883351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3883777Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3884217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3884710Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3885189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.3885623Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3885780Z 2025-09-07T07:16:02.3885892Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3886282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3886633Z return mod(**inputs) 2025-09-07T07:16:02.3887032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3887446Z outputs = self.roberta( 2025-09-07T07:16:02.3887855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3888303Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3888725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3889163Z layer_outputs = layer_module( 2025-09-07T07:16:02.3889554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3889929Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3890335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3890744Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3891143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3891548Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3891990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3892476Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3892943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.3893407Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.3893832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.3894206Z return self.act(input) 2025-09-07T07:16:02.3894328Z 2025-09-07T07:16:02.3894450Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3894840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3895188Z return mod(**inputs) 2025-09-07T07:16:02.3895596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3896041Z outputs = self.roberta( 2025-09-07T07:16:02.3896453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3896887Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3897318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3897751Z layer_outputs = layer_module( 2025-09-07T07:16:02.3898139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3898537Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3898982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3899419Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3899857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3900284Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3900750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.3901268Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.3901781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.3902234Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3902386Z 2025-09-07T07:16:02.3902511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3902918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3903272Z return mod(**inputs) 2025-09-07T07:16:02.3903705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3904140Z outputs = self.roberta( 2025-09-07T07:16:02.3904572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3905008Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3905447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3905968Z layer_outputs = layer_module( 2025-09-07T07:16:02.3906354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3906776Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3907219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3907679Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3908106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3908517Z return func(*args, **kwargs) 2025-09-07T07:16:02.3908958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3909406Z self_outputs = self.self( 2025-09-07T07:16:02.3909810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3910234Z return func(*args, **kwargs) 2025-09-07T07:16:02.3910674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.3911273Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.3911606Z 2025-09-07T07:16:02.3911724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3912131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3912493Z return mod(**inputs) 2025-09-07T07:16:02.3912921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3913361Z outputs = self.roberta( 2025-09-07T07:16:02.3913793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3914248Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3914689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3915153Z layer_outputs = layer_module( 2025-09-07T07:16:02.3915543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3915947Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3916407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3916861Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3917281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3917699Z return func(*args, **kwargs) 2025-09-07T07:16:02.3918126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3918565Z self_outputs = self.self( 2025-09-07T07:16:02.3918968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3919377Z return func(*args, **kwargs) 2025-09-07T07:16:02.3919995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.3920451Z self.key(current_states) 2025-09-07T07:16:02.3920580Z 2025-09-07T07:16:02.3920735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3921134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3921514Z return mod(**inputs) 2025-09-07T07:16:02.3921934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3922375Z outputs = self.roberta( 2025-09-07T07:16:02.3922798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3923237Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3923678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3924123Z layer_outputs = layer_module( 2025-09-07T07:16:02.3924512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3924908Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3925366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3925800Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3926194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3926573Z return func(*args, **kwargs) 2025-09-07T07:16:02.3926958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3927392Z self_outputs = self.self( 2025-09-07T07:16:02.3927761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3928139Z return func(*args, **kwargs) 2025-09-07T07:16:02.3928525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.3928920Z self.value(current_states) 2025-09-07T07:16:02.3929047Z 2025-09-07T07:16:02.3929132Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3929375Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3929742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3930070Z return mod(**inputs) 2025-09-07T07:16:02.3930469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3930857Z outputs = self.roberta( 2025-09-07T07:16:02.3931239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3931646Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3932042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3932448Z layer_outputs = layer_module( 2025-09-07T07:16:02.3932794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3933156Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3933550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3933965Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3934373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3934750Z return func(*args, **kwargs) 2025-09-07T07:16:02.3935132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3935540Z self_outputs = self.self( 2025-09-07T07:16:02.3935912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3936294Z return func(*args, **kwargs) 2025-09-07T07:16:02.3936688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.3937164Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.3937371Z 2025-09-07T07:16:02.3937488Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3937879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3938220Z return mod(**inputs) 2025-09-07T07:16:02.3938608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3939018Z outputs = self.roberta( 2025-09-07T07:16:02.3939430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3939862Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3940288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3940714Z layer_outputs = layer_module( 2025-09-07T07:16:02.3941088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3941486Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3941921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3942378Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3942800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3943205Z return func(*args, **kwargs) 2025-09-07T07:16:02.3943624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.3944116Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.3944603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.3945043Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3945220Z 2025-09-07T07:16:02.3945335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3945796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3946161Z return mod(**inputs) 2025-09-07T07:16:02.3946566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3946982Z outputs = self.roberta( 2025-09-07T07:16:02.3947406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3947838Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3948272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3948711Z layer_outputs = layer_module( 2025-09-07T07:16:02.3949080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3949479Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3949958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3950416Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3950863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3951292Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3951757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3952276Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3952763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.3953207Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3953364Z 2025-09-07T07:16:02.3953480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3953889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3954257Z return mod(**inputs) 2025-09-07T07:16:02.3954680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3955187Z outputs = self.roberta( 2025-09-07T07:16:02.3955576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3955982Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3956380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3956785Z layer_outputs = layer_module( 2025-09-07T07:16:02.3957139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3957537Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3957946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3958360Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3958765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3959172Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3959606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.3960094Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.3960547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.3961015Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.3961410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.3961760Z return self.act(input) 2025-09-07T07:16:02.3961876Z 2025-09-07T07:16:02.3961990Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3962363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3962686Z return mod(**inputs) 2025-09-07T07:16:02.3963064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3963462Z outputs = self.roberta( 2025-09-07T07:16:02.3963848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3964243Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3964658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3965067Z layer_outputs = layer_module( 2025-09-07T07:16:02.3965423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3965808Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3966218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.3966624Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.3967022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.3967425Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.3967862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.3968356Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.3968826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.3969247Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.3969385Z 2025-09-07T07:16:02.3969498Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3969852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3970179Z return mod(**inputs) 2025-09-07T07:16:02.3970554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3970954Z outputs = self.roberta( 2025-09-07T07:16:02.3971336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3971735Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3972164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3972580Z layer_outputs = layer_module( 2025-09-07T07:16:02.3972946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3973336Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3973732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3974142Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3974537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3974934Z return func(*args, **kwargs) 2025-09-07T07:16:02.3975346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3975796Z self_outputs = self.self( 2025-09-07T07:16:02.3976194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3976611Z return func(*args, **kwargs) 2025-09-07T07:16:02.3977043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.3977626Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.3977901Z 2025-09-07T07:16:02.3978007Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3978379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3978716Z return mod(**inputs) 2025-09-07T07:16:02.3979102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3979532Z outputs = self.roberta( 2025-09-07T07:16:02.3979945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3980374Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3980815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3981240Z layer_outputs = layer_module( 2025-09-07T07:16:02.3981602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3981998Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3982434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3982878Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3983289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3983693Z return func(*args, **kwargs) 2025-09-07T07:16:02.3984106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3984533Z self_outputs = self.self( 2025-09-07T07:16:02.3984921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3985313Z return func(*args, **kwargs) 2025-09-07T07:16:02.3985795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.3986232Z self.key(current_states) 2025-09-07T07:16:02.3986359Z 2025-09-07T07:16:02.3986486Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3986884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3987274Z return mod(**inputs) 2025-09-07T07:16:02.3987692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3988131Z outputs = self.roberta( 2025-09-07T07:16:02.3988540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3988965Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3989406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3989845Z layer_outputs = layer_module( 2025-09-07T07:16:02.3990224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3990632Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3991082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.3991526Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.3991941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3992348Z return func(*args, **kwargs) 2025-09-07T07:16:02.3992774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.3993212Z self_outputs = self.self( 2025-09-07T07:16:02.3993613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.3994023Z return func(*args, **kwargs) 2025-09-07T07:16:02.3994451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.3994883Z self.value(current_states) 2025-09-07T07:16:02.3995019Z 2025-09-07T07:16:02.3995126Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.3995391Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.3995786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.3996162Z return mod(**inputs) 2025-09-07T07:16:02.3996584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.3997020Z outputs = self.roberta( 2025-09-07T07:16:02.3997436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.3997881Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.3998316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.3998757Z layer_outputs = layer_module( 2025-09-07T07:16:02.3999141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.3999538Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.3999973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4000414Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4000830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4001243Z return func(*args, **kwargs) 2025-09-07T07:16:02.4001661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4002098Z self_outputs = self.self( 2025-09-07T07:16:02.4002493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4002916Z return func(*args, **kwargs) 2025-09-07T07:16:02.4003334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.4003824Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.4004027Z 2025-09-07T07:16:02.4004141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4004539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4004894Z return mod(**inputs) 2025-09-07T07:16:02.4005302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4005719Z outputs = self.roberta( 2025-09-07T07:16:02.4006147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4006573Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4006999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4007426Z layer_outputs = layer_module( 2025-09-07T07:16:02.4007803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4008197Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4008632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4009074Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4009491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4009888Z return func(*args, **kwargs) 2025-09-07T07:16:02.4010323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.4010818Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.4011309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.4011771Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4011930Z 2025-09-07T07:16:02.4012043Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4012446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4012800Z return mod(**inputs) 2025-09-07T07:16:02.4013212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4013641Z outputs = self.roberta( 2025-09-07T07:16:02.4014054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4014488Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4014918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4015347Z layer_outputs = layer_module( 2025-09-07T07:16:02.4015715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4016115Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4016550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4016996Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4017427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4017864Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4018336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4018884Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4019363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.4019957Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4020118Z 2025-09-07T07:16:02.4020232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4020632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4020997Z return mod(**inputs) 2025-09-07T07:16:02.4021420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4021890Z outputs = self.roberta( 2025-09-07T07:16:02.4037304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4037953Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4038406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4038823Z layer_outputs = layer_module( 2025-09-07T07:16:02.4039191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4039566Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4039982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4040404Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4040833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4041247Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4041840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4042345Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4042844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.4043289Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.4043682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.4044048Z return self.act(input) 2025-09-07T07:16:02.4044177Z 2025-09-07T07:16:02.4044300Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4044716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4045079Z return mod(**inputs) 2025-09-07T07:16:02.4045495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4045924Z outputs = self.roberta( 2025-09-07T07:16:02.4046347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4046767Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4047170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4047571Z layer_outputs = layer_module( 2025-09-07T07:16:02.4047931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4048310Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4048721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4049191Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4049607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4050018Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4050459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.4050961Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.4051418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.4051839Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4052020Z 2025-09-07T07:16:02.4052131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4052517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4052590Z return mod(**inputs) 2025-09-07T07:16:02.4052866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4052937Z outputs = self.roberta( 2025-09-07T07:16:02.4053207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4053290Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4053557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4053638Z layer_outputs = layer_module( 2025-09-07T07:16:02.4053868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4053951Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4054239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4054329Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4054605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4054683Z return func(*args, **kwargs) 2025-09-07T07:16:02.4054959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4055034Z self_outputs = self.self( 2025-09-07T07:16:02.4055283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4055360Z return func(*args, **kwargs) 2025-09-07T07:16:02.4055632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.4055874Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.4055879Z 2025-09-07T07:16:02.4055995Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4056227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4056306Z return mod(**inputs) 2025-09-07T07:16:02.4056576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4056652Z outputs = self.roberta( 2025-09-07T07:16:02.4056923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4057005Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4057275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4057368Z layer_outputs = layer_module( 2025-09-07T07:16:02.4057605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4057685Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4057963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4058047Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4058294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4058373Z return func(*args, **kwargs) 2025-09-07T07:16:02.4058639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4058740Z self_outputs = self.self( 2025-09-07T07:16:02.4058987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4059062Z return func(*args, **kwargs) 2025-09-07T07:16:02.4059341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.4059415Z self.key(current_states) 2025-09-07T07:16:02.4059419Z 2025-09-07T07:16:02.4059534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4059738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4059816Z return mod(**inputs) 2025-09-07T07:16:02.4060091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4060162Z outputs = self.roberta( 2025-09-07T07:16:02.4060442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4060515Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4060808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4060882Z layer_outputs = layer_module( 2025-09-07T07:16:02.4061127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4061216Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4061482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4061572Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4061820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4061892Z return func(*args, **kwargs) 2025-09-07T07:16:02.4062168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4062241Z self_outputs = self.self( 2025-09-07T07:16:02.4062494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4062565Z return func(*args, **kwargs) 2025-09-07T07:16:02.4062838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.4062912Z self.value(current_states) 2025-09-07T07:16:02.4062915Z 2025-09-07T07:16:02.4063000Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.4063114Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4063317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4063392Z return mod(**inputs) 2025-09-07T07:16:02.4063659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4063749Z outputs = self.roberta( 2025-09-07T07:16:02.4064034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4064108Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4064387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4064460Z layer_outputs = layer_module( 2025-09-07T07:16:02.4064689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4064775Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4065061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4065174Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4065436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4065525Z return func(*args, **kwargs) 2025-09-07T07:16:02.4065910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4065995Z self_outputs = self.self( 2025-09-07T07:16:02.4066273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4066349Z return func(*args, **kwargs) 2025-09-07T07:16:02.4066649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.4066835Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.4066842Z 2025-09-07T07:16:02.4066955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4067200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4067274Z return mod(**inputs) 2025-09-07T07:16:02.4067565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4067669Z outputs = self.roberta( 2025-09-07T07:16:02.4067961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4068049Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4068318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4068401Z layer_outputs = layer_module( 2025-09-07T07:16:02.4068630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4068718Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4068989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4069073Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4069346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4069419Z return func(*args, **kwargs) 2025-09-07T07:16:02.4069718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.4069863Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.4070145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.4070247Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4070250Z 2025-09-07T07:16:02.4070361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4070600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4070672Z return mod(**inputs) 2025-09-07T07:16:02.4070964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4071037Z outputs = self.roberta( 2025-09-07T07:16:02.4071318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4071403Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4071683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4071765Z layer_outputs = layer_module( 2025-09-07T07:16:02.4072024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4072107Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4072402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4072493Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4072781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4072864Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4073192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4073325Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4073610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.4073709Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4073714Z 2025-09-07T07:16:02.4073842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4074066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4074139Z return mod(**inputs) 2025-09-07T07:16:02.4074477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4074561Z outputs = self.roberta( 2025-09-07T07:16:02.4074841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4074927Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4075215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4075301Z layer_outputs = layer_module( 2025-09-07T07:16:02.4075543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4075628Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4075917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4076010Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4076296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4076379Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4076698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4076836Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4077121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.4077271Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.4077503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.4077587Z return self.act(input) 2025-09-07T07:16:02.4077591Z 2025-09-07T07:16:02.4077705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4077920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4078001Z return mod(**inputs) 2025-09-07T07:16:02.4078282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4078373Z outputs = self.roberta( 2025-09-07T07:16:02.4078640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4078733Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4079007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4079083Z layer_outputs = layer_module( 2025-09-07T07:16:02.4079315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4079398Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4079664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4079755Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4080025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4080111Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4080418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.4080585Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.4080855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.4081149Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4081153Z 2025-09-07T07:16:02.4081274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4081478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4081555Z return mod(**inputs) 2025-09-07T07:16:02.4081823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4081891Z outputs = self.roberta( 2025-09-07T07:16:02.4082168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4082248Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4082522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4082594Z layer_outputs = layer_module( 2025-09-07T07:16:02.4082830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4082911Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4083178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4083273Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4083522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4083605Z return func(*args, **kwargs) 2025-09-07T07:16:02.4083875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4083967Z self_outputs = self.self( 2025-09-07T07:16:02.4084232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4084305Z return func(*args, **kwargs) 2025-09-07T07:16:02.4084583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.4084800Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.4084803Z 2025-09-07T07:16:02.4084916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4085118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4085204Z return mod(**inputs) 2025-09-07T07:16:02.4085478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4085547Z outputs = self.roberta( 2025-09-07T07:16:02.4085820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4085894Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4086160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4086240Z layer_outputs = layer_module( 2025-09-07T07:16:02.4086465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4086553Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4086819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4086901Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4087172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4087246Z return func(*args, **kwargs) 2025-09-07T07:16:02.4087533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4087606Z self_outputs = self.self( 2025-09-07T07:16:02.4087860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4087929Z return func(*args, **kwargs) 2025-09-07T07:16:02.4088198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.4088278Z self.key(current_states) 2025-09-07T07:16:02.4088283Z 2025-09-07T07:16:02.4088387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4088597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4088668Z return mod(**inputs) 2025-09-07T07:16:02.4088935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4089012Z outputs = self.roberta( 2025-09-07T07:16:02.4089276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4089356Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4089630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4089709Z layer_outputs = layer_module( 2025-09-07T07:16:02.4089937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4090018Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4090317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4090400Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4090655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4090725Z return func(*args, **kwargs) 2025-09-07T07:16:02.4090995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4091071Z self_outputs = self.self( 2025-09-07T07:16:02.4091318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4091397Z return func(*args, **kwargs) 2025-09-07T07:16:02.4091679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.4091757Z self.value(current_states) 2025-09-07T07:16:02.4091761Z 2025-09-07T07:16:02.4092029Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.4092136Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4092346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4092414Z return mod(**inputs) 2025-09-07T07:16:02.4092687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4092755Z outputs = self.roberta( 2025-09-07T07:16:02.4093019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4093100Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4093374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4093487Z layer_outputs = layer_module( 2025-09-07T07:16:02.4093718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4093802Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4094111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4094194Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4094459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4094532Z return func(*args, **kwargs) 2025-09-07T07:16:02.4094828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4094910Z self_outputs = self.self( 2025-09-07T07:16:02.4095172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4095255Z return func(*args, **kwargs) 2025-09-07T07:16:02.4095556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.4095711Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.4095715Z 2025-09-07T07:16:02.4095826Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4096051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4096129Z return mod(**inputs) 2025-09-07T07:16:02.4096427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4096509Z outputs = self.roberta( 2025-09-07T07:16:02.4096805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4096906Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4097193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4097263Z layer_outputs = layer_module( 2025-09-07T07:16:02.4097489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4097569Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4097843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4097924Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4098169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4098266Z return func(*args, **kwargs) 2025-09-07T07:16:02.4098541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.4098683Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.4098961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.4099048Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4099051Z 2025-09-07T07:16:02.4099163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4099368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4099443Z return mod(**inputs) 2025-09-07T07:16:02.4099717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4099793Z outputs = self.roberta( 2025-09-07T07:16:02.4100085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4100163Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4100438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4100528Z layer_outputs = layer_module( 2025-09-07T07:16:02.4100763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4100841Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4101110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4101203Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4101464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4101550Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4101853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4101979Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4102258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.4102342Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4102346Z 2025-09-07T07:16:02.4102457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4102662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4102731Z return mod(**inputs) 2025-09-07T07:16:02.4103014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4103082Z outputs = self.roberta( 2025-09-07T07:16:02.4103375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4103449Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4103715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4103794Z layer_outputs = layer_module( 2025-09-07T07:16:02.4104020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4104106Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4104377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4104492Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4104766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4104851Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4105173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4105302Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4105591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.4105792Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.4106040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.4106127Z return self.act(input) 2025-09-07T07:16:02.4106131Z 2025-09-07T07:16:02.4106249Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4106476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4106575Z return mod(**inputs) 2025-09-07T07:16:02.4106866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4106949Z outputs = self.roberta( 2025-09-07T07:16:02.4107254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4107338Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4107604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4107685Z layer_outputs = layer_module( 2025-09-07T07:16:02.4107913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4107995Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4108276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4108362Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4108635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4108713Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4109017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.4109164Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.4109435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.4109529Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4109532Z 2025-09-07T07:16:02.4109637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4109868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4109934Z return mod(**inputs) 2025-09-07T07:16:02.4110197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4110273Z outputs = self.roberta( 2025-09-07T07:16:02.4110539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4110619Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4110886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4110955Z layer_outputs = layer_module( 2025-09-07T07:16:02.4111207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4111286Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4111565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4111647Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4111912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4111989Z return func(*args, **kwargs) 2025-09-07T07:16:02.4112273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4112355Z self_outputs = self.self( 2025-09-07T07:16:02.4112620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4112703Z return func(*args, **kwargs) 2025-09-07T07:16:02.4112987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.4113232Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.4113237Z 2025-09-07T07:16:02.4113358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4113600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4113681Z return mod(**inputs) 2025-09-07T07:16:02.4113979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4114058Z outputs = self.roberta( 2025-09-07T07:16:02.4114339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4114420Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4114720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4114802Z layer_outputs = layer_module( 2025-09-07T07:16:02.4115049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4115133Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4115430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4115527Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4115786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4115870Z return func(*args, **kwargs) 2025-09-07T07:16:02.4116162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4116239Z self_outputs = self.self( 2025-09-07T07:16:02.4116505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4116599Z return func(*args, **kwargs) 2025-09-07T07:16:02.4116902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.4116974Z self.key(current_states) 2025-09-07T07:16:02.4116977Z 2025-09-07T07:16:02.4117090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4117296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4117363Z return mod(**inputs) 2025-09-07T07:16:02.4117638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4117733Z outputs = self.roberta( 2025-09-07T07:16:02.4118005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4118086Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4118385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4118467Z layer_outputs = layer_module( 2025-09-07T07:16:02.4118704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4118795Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4119091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4119180Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4119449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4119524Z return func(*args, **kwargs) 2025-09-07T07:16:02.4120068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4120149Z self_outputs = self.self( 2025-09-07T07:16:02.4120403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4120499Z return func(*args, **kwargs) 2025-09-07T07:16:02.4120801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.4120887Z self.value(current_states) 2025-09-07T07:16:02.4120891Z 2025-09-07T07:16:02.4120980Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.4121099Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4121326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4121401Z return mod(**inputs) 2025-09-07T07:16:02.4121711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4121787Z outputs = self.roberta( 2025-09-07T07:16:02.4122082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4122158Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4122428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4122507Z layer_outputs = layer_module( 2025-09-07T07:16:02.4122739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4122826Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4123098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4123189Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4123461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4123531Z return func(*args, **kwargs) 2025-09-07T07:16:02.4123807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4123877Z self_outputs = self.self( 2025-09-07T07:16:02.4124127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4124196Z return func(*args, **kwargs) 2025-09-07T07:16:02.4124460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.4124634Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.4124638Z 2025-09-07T07:16:02.4124741Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4124954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4125022Z return mod(**inputs) 2025-09-07T07:16:02.4125299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4125367Z outputs = self.roberta( 2025-09-07T07:16:02.4125633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4125716Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4125982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4126058Z layer_outputs = layer_module( 2025-09-07T07:16:02.4126284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4126380Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4126658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4126737Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4126988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4127057Z return func(*args, **kwargs) 2025-09-07T07:16:02.4127307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.4127441Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.4127692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.4127783Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4127786Z 2025-09-07T07:16:02.4127888Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4128084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4128147Z return mod(**inputs) 2025-09-07T07:16:02.4128399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4128473Z outputs = self.roberta( 2025-09-07T07:16:02.4128726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4128800Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4129050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4129118Z layer_outputs = layer_module( 2025-09-07T07:16:02.4129336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4129429Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4129686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4129769Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4130024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4130098Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4130380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4130505Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4130774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.4130860Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4130866Z 2025-09-07T07:16:02.4130966Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4131155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4131230Z return mod(**inputs) 2025-09-07T07:16:02.4131480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4131552Z outputs = self.roberta( 2025-09-07T07:16:02.4131800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4131876Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4132126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4132196Z layer_outputs = layer_module( 2025-09-07T07:16:02.4132435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4132514Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4132789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4132871Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4133120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4133202Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4133485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4133611Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4133862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.4133974Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.4134188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.4134260Z return self.act(input) 2025-09-07T07:16:02.4134263Z 2025-09-07T07:16:02.4134372Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4134568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4134639Z return mod(**inputs) 2025-09-07T07:16:02.4134898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4134965Z outputs = self.roberta( 2025-09-07T07:16:02.4135233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4135305Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4135591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4135659Z layer_outputs = layer_module( 2025-09-07T07:16:02.4135881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4135967Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4136228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4136319Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4136572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4136671Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4136963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.4137097Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.4137375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.4137455Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4137458Z 2025-09-07T07:16:02.4137563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4137755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4137821Z return mod(**inputs) 2025-09-07T07:16:02.4138088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4138158Z outputs = self.roberta( 2025-09-07T07:16:02.4138437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4138513Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4138778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4138862Z layer_outputs = layer_module( 2025-09-07T07:16:02.4139082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4139166Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4139423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4139511Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4139751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4139821Z return func(*args, **kwargs) 2025-09-07T07:16:02.4140087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4140156Z self_outputs = self.self( 2025-09-07T07:16:02.4140407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4140475Z return func(*args, **kwargs) 2025-09-07T07:16:02.4140733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.4140952Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.4140955Z 2025-09-07T07:16:02.4141058Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4141268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4141334Z return mod(**inputs) 2025-09-07T07:16:02.4141633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4141701Z outputs = self.roberta( 2025-09-07T07:16:02.4141966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4142050Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4142330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4142411Z layer_outputs = layer_module( 2025-09-07T07:16:02.4142646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4142728Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4143033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4143123Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4143390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4143463Z return func(*args, **kwargs) 2025-09-07T07:16:02.4143761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4143836Z self_outputs = self.self( 2025-09-07T07:16:02.4144101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4144184Z return func(*args, **kwargs) 2025-09-07T07:16:02.4144474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.4144568Z self.key(current_states) 2025-09-07T07:16:02.4144571Z 2025-09-07T07:16:02.4144681Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4144912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4144995Z return mod(**inputs) 2025-09-07T07:16:02.4145310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4145390Z outputs = self.roberta( 2025-09-07T07:16:02.4145678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4145823Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4146135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4146213Z layer_outputs = layer_module( 2025-09-07T07:16:02.4146477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4146567Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4146874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4146965Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4147240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4147326Z return func(*args, **kwargs) 2025-09-07T07:16:02.4147636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4147719Z self_outputs = self.self( 2025-09-07T07:16:02.4147984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4148061Z return func(*args, **kwargs) 2025-09-07T07:16:02.4148363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.4148465Z self.value(current_states) 2025-09-07T07:16:02.4148469Z 2025-09-07T07:16:02.4148569Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.4148681Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4148894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4148973Z return mod(**inputs) 2025-09-07T07:16:02.4149255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4149334Z outputs = self.roberta( 2025-09-07T07:16:02.4149614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4149718Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4150002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4150079Z layer_outputs = layer_module( 2025-09-07T07:16:02.4150326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4150412Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4150703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4150789Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4151047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4151129Z return func(*args, **kwargs) 2025-09-07T07:16:02.4151414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4151498Z self_outputs = self.self( 2025-09-07T07:16:02.4151779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4151855Z return func(*args, **kwargs) 2025-09-07T07:16:02.4152161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.4152308Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.4152312Z 2025-09-07T07:16:02.4152427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4152639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4152716Z return mod(**inputs) 2025-09-07T07:16:02.4152995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4153069Z outputs = self.roberta( 2025-09-07T07:16:02.4153360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4153440Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4153727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4153805Z layer_outputs = layer_module( 2025-09-07T07:16:02.4154042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4154133Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4154416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4154517Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4154766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4154842Z return func(*args, **kwargs) 2025-09-07T07:16:02.4155137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.4155270Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.4155555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.4155639Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4155642Z 2025-09-07T07:16:02.4155752Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4155951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4156017Z return mod(**inputs) 2025-09-07T07:16:02.4156294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4156382Z outputs = self.roberta( 2025-09-07T07:16:02.4156653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4156726Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4157001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4157074Z layer_outputs = layer_module( 2025-09-07T07:16:02.4157297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4157381Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4157646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4157736Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4158001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4158096Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4158403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4158543Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4158818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.4158902Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4158905Z 2025-09-07T07:16:02.4159013Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4159216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4159285Z return mod(**inputs) 2025-09-07T07:16:02.4159561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4159635Z outputs = self.roberta( 2025-09-07T07:16:02.4159908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4159982Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4160250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4160330Z layer_outputs = layer_module( 2025-09-07T07:16:02.4160552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4160638Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4160904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4160991Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4161263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4161360Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4161670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4161791Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4162071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.4162195Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.4162430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.4162537Z return self.act(input) 2025-09-07T07:16:02.4162541Z 2025-09-07T07:16:02.4162666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4162892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4162967Z return mod(**inputs) 2025-09-07T07:16:02.4163255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4163342Z outputs = self.roberta( 2025-09-07T07:16:02.4163626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4163710Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4163997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4164081Z layer_outputs = layer_module( 2025-09-07T07:16:02.4164321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4164404Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4164696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4164782Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4165073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4165151Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4165453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.4165607Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.4165876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.4165967Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4165971Z 2025-09-07T07:16:02.4166075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4166291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4166359Z return mod(**inputs) 2025-09-07T07:16:02.4166629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4166707Z outputs = self.roberta( 2025-09-07T07:16:02.4166977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4167058Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4167325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4167399Z layer_outputs = layer_module( 2025-09-07T07:16:02.4167636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4167739Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4168010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4168093Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4168343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4168421Z return func(*args, **kwargs) 2025-09-07T07:16:02.4168688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4168767Z self_outputs = self.self( 2025-09-07T07:16:02.4169013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4169108Z return func(*args, **kwargs) 2025-09-07T07:16:02.4169382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.4169602Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.4169606Z 2025-09-07T07:16:02.4169722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4169929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4170001Z return mod(**inputs) 2025-09-07T07:16:02.4170284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4170353Z outputs = self.roberta( 2025-09-07T07:16:02.4170628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4170702Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4170993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4171066Z layer_outputs = layer_module( 2025-09-07T07:16:02.4171292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4171388Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4171648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4171738Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4171980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4172056Z return func(*args, **kwargs) 2025-09-07T07:16:02.4172318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4172386Z self_outputs = self.self( 2025-09-07T07:16:02.4172634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4172701Z return func(*args, **kwargs) 2025-09-07T07:16:02.4172966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.4173035Z self.key(current_states) 2025-09-07T07:16:02.4173038Z 2025-09-07T07:16:02.4173141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4173350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4173417Z return mod(**inputs) 2025-09-07T07:16:02.4173690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4173766Z outputs = self.roberta( 2025-09-07T07:16:02.4174033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4174121Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4174379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4174459Z layer_outputs = layer_module( 2025-09-07T07:16:02.4174677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4174762Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4175021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4175103Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4175366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4175434Z return func(*args, **kwargs) 2025-09-07T07:16:02.4175709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4175776Z self_outputs = self.self( 2025-09-07T07:16:02.4176009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4176084Z return func(*args, **kwargs) 2025-09-07T07:16:02.4176345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.4176427Z self.value(current_states) 2025-09-07T07:16:02.4176431Z 2025-09-07T07:16:02.4176512Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.4176621Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4176820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4176886Z return mod(**inputs) 2025-09-07T07:16:02.4177209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4177279Z outputs = self.roberta( 2025-09-07T07:16:02.4177570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4177646Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4177915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4178004Z layer_outputs = layer_module( 2025-09-07T07:16:02.4178221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4178306Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4178573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4178657Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4178907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4178976Z return func(*args, **kwargs) 2025-09-07T07:16:02.4179243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4179310Z self_outputs = self.self( 2025-09-07T07:16:02.4179556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4179625Z return func(*args, **kwargs) 2025-09-07T07:16:02.4179887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.4180031Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.4180034Z 2025-09-07T07:16:02.4180157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4180361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4180429Z return mod(**inputs) 2025-09-07T07:16:02.4180699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4180777Z outputs = self.roberta( 2025-09-07T07:16:02.4181043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4181120Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4181379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4181472Z layer_outputs = layer_module( 2025-09-07T07:16:02.4181695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4181776Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4182049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4182131Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4182383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4182451Z return func(*args, **kwargs) 2025-09-07T07:16:02.4182716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.4182855Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.4183123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.4183215Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4183236Z 2025-09-07T07:16:02.4183341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4183547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4183612Z return mod(**inputs) 2025-09-07T07:16:02.4183896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4183974Z outputs = self.roberta( 2025-09-07T07:16:02.4184242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4184319Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4184586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4184659Z layer_outputs = layer_module( 2025-09-07T07:16:02.4184893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4184975Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4185252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4185338Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4185601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4185689Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4186069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4186208Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4186479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.4186598Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4186602Z 2025-09-07T07:16:02.4186713Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4186933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4187015Z return mod(**inputs) 2025-09-07T07:16:02.4187298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4187379Z outputs = self.roberta( 2025-09-07T07:16:02.4187659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4187735Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4188058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4188130Z layer_outputs = layer_module( 2025-09-07T07:16:02.4188370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4188449Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4188727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4188811Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4189075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4189159Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4189460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4189591Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4189887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.4190008Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.4190248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.4190321Z return self.act(input) 2025-09-07T07:16:02.4190324Z 2025-09-07T07:16:02.4190436Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4190643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4190719Z return mod(**inputs) 2025-09-07T07:16:02.4190992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4191062Z outputs = self.roberta( 2025-09-07T07:16:02.4191341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4191418Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4191693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4191766Z layer_outputs = layer_module( 2025-09-07T07:16:02.4191996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4192086Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4192354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4192445Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4192714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4192793Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4193103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.4193262Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.4193542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.4193629Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4193633Z 2025-09-07T07:16:02.4193747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4193956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4194029Z return mod(**inputs) 2025-09-07T07:16:02.4194305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4194391Z outputs = self.roberta( 2025-09-07T07:16:02.4194664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4194738Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4195003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4195084Z layer_outputs = layer_module( 2025-09-07T07:16:02.4195310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4195396Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4195661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4195750Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4195998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4196073Z return func(*args, **kwargs) 2025-09-07T07:16:02.4196366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4196439Z self_outputs = self.self( 2025-09-07T07:16:02.4196711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4196784Z return func(*args, **kwargs) 2025-09-07T07:16:02.4197050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:02.4197269Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:02.4197273Z 2025-09-07T07:16:02.4197376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4197590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4197658Z return mod(**inputs) 2025-09-07T07:16:02.4197939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4198008Z outputs = self.roberta( 2025-09-07T07:16:02.4198269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4198351Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4198613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4198690Z layer_outputs = layer_module( 2025-09-07T07:16:02.4198908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4198987Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4199277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4199383Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4199653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4199730Z return func(*args, **kwargs) 2025-09-07T07:16:02.4200011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4200093Z self_outputs = self.self( 2025-09-07T07:16:02.4200351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4200434Z return func(*args, **kwargs) 2025-09-07T07:16:02.4200715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:02.4200822Z self.key(current_states) 2025-09-07T07:16:02.4200826Z 2025-09-07T07:16:02.4200941Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4201158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4201237Z return mod(**inputs) 2025-09-07T07:16:02.4201521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4201599Z outputs = self.roberta( 2025-09-07T07:16:02.4201878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4201955Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4202244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4202322Z layer_outputs = layer_module( 2025-09-07T07:16:02.4202564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4202665Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4202951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4203063Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4203326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4203407Z return func(*args, **kwargs) 2025-09-07T07:16:02.4203687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4203769Z self_outputs = self.self( 2025-09-07T07:16:02.4204032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4204107Z return func(*args, **kwargs) 2025-09-07T07:16:02.4204398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:02.4204476Z self.value(current_states) 2025-09-07T07:16:02.4204480Z 2025-09-07T07:16:02.4204576Z cudagraph partition due to non gpu ops 2025-09-07T07:16:02.4204688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4204902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4204981Z return mod(**inputs) 2025-09-07T07:16:02.4205265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4205344Z outputs = self.roberta( 2025-09-07T07:16:02.4205624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4205703Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4205994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4206089Z layer_outputs = layer_module( 2025-09-07T07:16:02.4206337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4206421Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4206708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4206795Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4207054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4207135Z return func(*args, **kwargs) 2025-09-07T07:16:02.4207438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:02.4207523Z self_outputs = self.self( 2025-09-07T07:16:02.4207784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4207859Z return func(*args, **kwargs) 2025-09-07T07:16:02.4208156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:02.4208303Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:02.4208307Z 2025-09-07T07:16:02.4208425Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4208642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4208721Z return mod(**inputs) 2025-09-07T07:16:02.4209005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4209079Z outputs = self.roberta( 2025-09-07T07:16:02.4209392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4209472Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4209777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4209856Z layer_outputs = layer_module( 2025-09-07T07:16:02.4210093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4210184Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4210465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:02.4210563Z self_attention_outputs = self.attention( 2025-09-07T07:16:02.4210824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:02.4210903Z return func(*args, **kwargs) 2025-09-07T07:16:02.4211192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:02.4211335Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:02.4211623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:02.4211711Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4211715Z 2025-09-07T07:16:02.4211831Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4212044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4212115Z return mod(**inputs) 2025-09-07T07:16:02.4212409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4212507Z outputs = self.roberta( 2025-09-07T07:16:02.4212794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4212871Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4213154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4213238Z layer_outputs = layer_module( 2025-09-07T07:16:02.4213473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4213565Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4213850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4213978Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4214258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4214342Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4214668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4214799Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4215086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:02.4215174Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4215178Z 2025-09-07T07:16:02.4215288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4215509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4215581Z return mod(**inputs) 2025-09-07T07:16:02.4215883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4215960Z outputs = self.roberta( 2025-09-07T07:16:02.4216247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4216339Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4216624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4216710Z layer_outputs = layer_module( 2025-09-07T07:16:02.4216949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4217039Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4217322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4217413Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4217704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4217787Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4218117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:02.4218246Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:02.4218553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:02.4218677Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:02.4218909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:02.4218995Z return self.act(input) 2025-09-07T07:16:02.4218998Z 2025-09-07T07:16:02.4219110Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4219357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4219426Z return mod(**inputs) 2025-09-07T07:16:02.4219866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-09-07T07:16:02.4219954Z outputs = self.roberta( 2025-09-07T07:16:02.4220255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:02.4220342Z encoder_outputs = self.encoder( 2025-09-07T07:16:02.4220642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:02.4220720Z layer_outputs = layer_module( 2025-09-07T07:16:02.4221015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:02.4221104Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:02.4221402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:02.4221491Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:02.4221774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:02.4221858Z return forward_fn(*input_tensors) 2025-09-07T07:16:02.4222174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:02.4222327Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:02.4222626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:02.4222723Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:02.4222728Z 2025-09-07T07:16:02.4222872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4223088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4223168Z return mod(**inputs) 2025-09-07T07:16:02.4223483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-09-07T07:16:02.4223600Z prediction_scores = self.lm_head(sequence_output) 2025-09-07T07:16:02.4223891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1149, in forward 2025-09-07T07:16:02.4223974Z x = self.dense(features) 2025-09-07T07:16:02.4223978Z 2025-09-07T07:16:02.4224088Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4224304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4224383Z return mod(**inputs) 2025-09-07T07:16:02.4224672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-09-07T07:16:02.4224782Z prediction_scores = self.lm_head(sequence_output) 2025-09-07T07:16:02.4225086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1154, in forward 2025-09-07T07:16:02.4225159Z x = self.decoder(x) 2025-09-07T07:16:02.4225171Z 2025-09-07T07:16:02.4225279Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:02.4225489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:02.4225568Z return mod(**inputs) 2025-09-07T07:16:02.4226088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1022, in forward 2025-09-07T07:16:02.4226185Z lm_loss = self.loss_function( 2025-09-07T07:16:02.4226465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-09-07T07:16:02.4226703Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-09-07T07:16:02.4227001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-09-07T07:16:02.4227231Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-09-07T07:16:02.4227235Z 2025-09-07T07:16:13.6114520Z Compilation time (from dynamo_timed): 17.633372589 2025-09-07T07:16:13.6243900Z pass 2025-09-07T07:16:13.6244579Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:13.6245574Z TIMING: _recursive_pre_grad_passes:0.00783 _recursive_joint_graph_passes:0.37253 _recursive_post_grad_passes:0.07785 async_compile.wait:0.77797 code_gen:10.37663 inductor_compile:11.66122 backend_compile:14.88171 gc:0.00136 entire_frame_compile:17.63337 total_wall_time:17.63337 2025-09-07T07:16:13.6247052Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12458 | FakeTensor.__torch_dispatch__:4402 | ProxyTorchDispatchMode.__torch_dispatch__:4539 2025-09-07T07:16:13.6247611Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-09-07T07:16:16.2500879Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:16:16.2501787Z import pynvml # type: ignore[import] 2025-09-07T07:16:19.0868392Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:16:19.0870802Z from pkg_resources import resource_filename 2025-09-07T07:16:19.7698770Z 2025-09-07T07:16:20.7716992Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-09-07T07:16:20.7718106Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-09-07T07:16:20.7719117Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-09-07T07:16:20.7720393Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-09-07T07:16:20.9212597Z 2025-09-07T07:16:20.9213391Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:16:20.9223092Z cpu eval RobertaForQuestionAnswering 2025-09-07T07:16:21.3436549Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:21.5525644Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:21.7570462Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:29.6528502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6529052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6529505Z return mod(**inputs) 2025-09-07T07:16:29.6530109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6531002Z outputs = self.roberta( 2025-09-07T07:16:29.6531436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-09-07T07:16:29.6531881Z embedding_output = self.embeddings( 2025-09-07T07:16:29.6532279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-09-07T07:16:29.6532815Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:16:29.6533421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-09-07T07:16:29.6533965Z mask = input_ids.ne(padding_idx).int() 2025-09-07T07:16:29.6534118Z 2025-09-07T07:16:29.6534205Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6534427Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6534644Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6534862Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6535062Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6535266Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6535496Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6535723Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6535946Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6536190Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6536413Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6536635Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6536884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6537271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6537615Z return mod(**inputs) 2025-09-07T07:16:29.6538078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6538483Z outputs = self.roberta( 2025-09-07T07:16:29.6538919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-09-07T07:16:29.6539389Z embedding_output = self.embeddings( 2025-09-07T07:16:29.6539806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-09-07T07:16:29.6540358Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:16:29.6540977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-09-07T07:16:29.6541586Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:16:29.6541845Z 2025-09-07T07:16:29.6541961Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6542350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6542704Z return mod(**inputs) 2025-09-07T07:16:29.6543132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6543577Z outputs = self.roberta( 2025-09-07T07:16:29.6543988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-09-07T07:16:29.6544437Z embedding_output = self.embeddings( 2025-09-07T07:16:29.6544885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-09-07T07:16:29.6545465Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-09-07T07:16:29.6546318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-09-07T07:16:29.6546972Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-09-07T07:16:29.6547236Z 2025-09-07T07:16:29.6547362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6547758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6548108Z return mod(**inputs) 2025-09-07T07:16:29.6548496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6548907Z outputs = self.roberta( 2025-09-07T07:16:29.6549396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6549826Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6550254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6550680Z layer_outputs = layer_module( 2025-09-07T07:16:29.6551073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6551489Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6551926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6552371Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6552796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6553208Z return func(*args, **kwargs) 2025-09-07T07:16:29.6553629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6554072Z self_outputs = self.self( 2025-09-07T07:16:29.6554477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6554890Z return func(*args, **kwargs) 2025-09-07T07:16:29.6555336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6555926Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6556215Z 2025-09-07T07:16:29.6556328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6556728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6557086Z return mod(**inputs) 2025-09-07T07:16:29.6557498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6557931Z outputs = self.roberta( 2025-09-07T07:16:29.6558343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6558778Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6559212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6559648Z layer_outputs = layer_module( 2025-09-07T07:16:29.6560022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6560424Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6560860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6561303Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6561724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6562148Z return func(*args, **kwargs) 2025-09-07T07:16:29.6562564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6562991Z self_outputs = self.self( 2025-09-07T07:16:29.6563380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6563774Z return func(*args, **kwargs) 2025-09-07T07:16:29.6564192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6564614Z self.key(current_states) 2025-09-07T07:16:29.6564760Z 2025-09-07T07:16:29.6564881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6565274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6565628Z return mod(**inputs) 2025-09-07T07:16:29.6566034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6566460Z outputs = self.roberta( 2025-09-07T07:16:29.6566871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6567303Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6567725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6568134Z layer_outputs = layer_module( 2025-09-07T07:16:29.6568495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6568897Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6569321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6569736Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6570150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6570536Z return func(*args, **kwargs) 2025-09-07T07:16:29.6570921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6571326Z self_outputs = self.self( 2025-09-07T07:16:29.6571699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6572079Z return func(*args, **kwargs) 2025-09-07T07:16:29.6572468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6572865Z self.value(current_states) 2025-09-07T07:16:29.6572996Z 2025-09-07T07:16:29.6573081Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6573334Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6573708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6574034Z return mod(**inputs) 2025-09-07T07:16:29.6574427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6574834Z outputs = self.roberta( 2025-09-07T07:16:29.6575240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6575666Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6576084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6576513Z layer_outputs = layer_module( 2025-09-07T07:16:29.6576927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6577304Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6577722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6578148Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6578548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6578937Z return func(*args, **kwargs) 2025-09-07T07:16:29.6579334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6579745Z self_outputs = self.self( 2025-09-07T07:16:29.6580133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6580544Z return func(*args, **kwargs) 2025-09-07T07:16:29.6580957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6581447Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6581648Z 2025-09-07T07:16:29.6581764Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6582156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6582506Z return mod(**inputs) 2025-09-07T07:16:29.6582912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6583343Z outputs = self.roberta( 2025-09-07T07:16:29.6583751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6584216Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6584673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6585100Z layer_outputs = layer_module( 2025-09-07T07:16:29.6585500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6586023Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6586490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6586954Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6587374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6587778Z return func(*args, **kwargs) 2025-09-07T07:16:29.6588213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6588717Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6589205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6589647Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6589802Z 2025-09-07T07:16:29.6589917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6590306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6590659Z return mod(**inputs) 2025-09-07T07:16:29.6591072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6591496Z outputs = self.roberta( 2025-09-07T07:16:29.6591921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6592376Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6592799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6593226Z layer_outputs = layer_module( 2025-09-07T07:16:29.6593598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6593993Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6594439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6594882Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6595322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6595764Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6596231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6596748Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6597201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6597613Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6597757Z 2025-09-07T07:16:29.6597861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6598216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6598540Z return mod(**inputs) 2025-09-07T07:16:29.6598916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6599301Z outputs = self.roberta( 2025-09-07T07:16:29.6599699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6600098Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6600514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6600906Z layer_outputs = layer_module( 2025-09-07T07:16:29.6601243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6601604Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6602004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6602404Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6602808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6603212Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6603647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6604137Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6604577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6605002Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6605384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6605732Z return self.act(input) 2025-09-07T07:16:29.6605845Z 2025-09-07T07:16:29.6605959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6606333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6606697Z return mod(**inputs) 2025-09-07T07:16:29.6607074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6607470Z outputs = self.roberta( 2025-09-07T07:16:29.6607849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6608237Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6608630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6609025Z layer_outputs = layer_module( 2025-09-07T07:16:29.6609377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6609764Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6610156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6610567Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6610965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6611355Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6611780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6612253Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6612703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6613105Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6613242Z 2025-09-07T07:16:29.6613352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6613736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6614058Z return mod(**inputs) 2025-09-07T07:16:29.6614429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6614840Z outputs = self.roberta( 2025-09-07T07:16:29.6615212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6615595Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6615991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6616399Z layer_outputs = layer_module( 2025-09-07T07:16:29.6616745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6617116Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6617532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6617933Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6618326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6618708Z return func(*args, **kwargs) 2025-09-07T07:16:29.6619093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6619496Z self_outputs = self.self( 2025-09-07T07:16:29.6620212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6620600Z return func(*args, **kwargs) 2025-09-07T07:16:29.6621002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6621629Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6621909Z 2025-09-07T07:16:29.6622017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6622393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6622731Z return mod(**inputs) 2025-09-07T07:16:29.6623121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6623523Z outputs = self.roberta( 2025-09-07T07:16:29.6623912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6624318Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6624754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6625162Z layer_outputs = layer_module( 2025-09-07T07:16:29.6625532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6625996Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6626452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6626913Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6627322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6627725Z return func(*args, **kwargs) 2025-09-07T07:16:29.6628146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6628556Z self_outputs = self.self( 2025-09-07T07:16:29.6628966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6629345Z return func(*args, **kwargs) 2025-09-07T07:16:29.6629738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6630170Z self.key(current_states) 2025-09-07T07:16:29.6630290Z 2025-09-07T07:16:29.6630403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6630768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6631104Z return mod(**inputs) 2025-09-07T07:16:29.6631488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6631893Z outputs = self.roberta( 2025-09-07T07:16:29.6632280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6632680Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6633078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6633483Z layer_outputs = layer_module( 2025-09-07T07:16:29.6633840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6634219Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6634621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6635038Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6635431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6635873Z return func(*args, **kwargs) 2025-09-07T07:16:29.6636273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6636714Z self_outputs = self.self( 2025-09-07T07:16:29.6637095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6637526Z return func(*args, **kwargs) 2025-09-07T07:16:29.6637958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6638376Z self.value(current_states) 2025-09-07T07:16:29.6638513Z 2025-09-07T07:16:29.6638603Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6638869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6639267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6639622Z return mod(**inputs) 2025-09-07T07:16:29.6640006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6640420Z outputs = self.roberta( 2025-09-07T07:16:29.6640828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6641253Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6641667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6642123Z layer_outputs = layer_module( 2025-09-07T07:16:29.6642502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6642895Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6643327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6643760Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6644199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6644603Z return func(*args, **kwargs) 2025-09-07T07:16:29.6645042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6645462Z self_outputs = self.self( 2025-09-07T07:16:29.6645854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6646266Z return func(*args, **kwargs) 2025-09-07T07:16:29.6646702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6647286Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6647492Z 2025-09-07T07:16:29.6647605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6648001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6648360Z return mod(**inputs) 2025-09-07T07:16:29.6648751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6649154Z outputs = self.roberta( 2025-09-07T07:16:29.6649530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6649940Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6650341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6650821Z layer_outputs = layer_module( 2025-09-07T07:16:29.6651171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6651540Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6651972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6652384Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6652780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6653160Z return func(*args, **kwargs) 2025-09-07T07:16:29.6653552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6654015Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6654477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6654912Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6655056Z 2025-09-07T07:16:29.6655165Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6655536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6655870Z return mod(**inputs) 2025-09-07T07:16:29.6656259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6656687Z outputs = self.roberta( 2025-09-07T07:16:29.6657098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6657498Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6657896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6658295Z layer_outputs = layer_module( 2025-09-07T07:16:29.6658642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6659029Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6659445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6659875Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6660332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6660741Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6661190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6661689Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6662152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6662573Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6662726Z 2025-09-07T07:16:29.6662833Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6663210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6663630Z return mod(**inputs) 2025-09-07T07:16:29.6664050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6664475Z outputs = self.roberta( 2025-09-07T07:16:29.6664891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6665331Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6665837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6666280Z layer_outputs = layer_module( 2025-09-07T07:16:29.6666652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6667075Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6667526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6668048Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6668469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6668900Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6669364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6669879Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6670389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6670857Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6671278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6671653Z return self.act(input) 2025-09-07T07:16:29.6671777Z 2025-09-07T07:16:29.6671899Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6672294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6672640Z return mod(**inputs) 2025-09-07T07:16:29.6673047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6673478Z outputs = self.roberta( 2025-09-07T07:16:29.6673886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6674312Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6674767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6675196Z layer_outputs = layer_module( 2025-09-07T07:16:29.6675594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6675994Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6676423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6676869Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6677284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6677685Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6678130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6678623Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6679090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6679526Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6679665Z 2025-09-07T07:16:29.6680064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6680431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6680750Z return mod(**inputs) 2025-09-07T07:16:29.6681129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6681527Z outputs = self.roberta( 2025-09-07T07:16:29.6681904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6682317Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6682719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6683122Z layer_outputs = layer_module( 2025-09-07T07:16:29.6683483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6683860Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6684268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6684672Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6685054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6685455Z return func(*args, **kwargs) 2025-09-07T07:16:29.6685840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6686233Z self_outputs = self.self( 2025-09-07T07:16:29.6686609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6686987Z return func(*args, **kwargs) 2025-09-07T07:16:29.6687378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6687926Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6688193Z 2025-09-07T07:16:29.6688297Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6688655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6688978Z return mod(**inputs) 2025-09-07T07:16:29.6689372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6689764Z outputs = self.roberta( 2025-09-07T07:16:29.6690154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6690546Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6690935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6691329Z layer_outputs = layer_module( 2025-09-07T07:16:29.6691672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6692036Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6692438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6692845Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6693227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6693604Z return func(*args, **kwargs) 2025-09-07T07:16:29.6693988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6694381Z self_outputs = self.self( 2025-09-07T07:16:29.6694748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6695116Z return func(*args, **kwargs) 2025-09-07T07:16:29.6695509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6695917Z self.key(current_states) 2025-09-07T07:16:29.6696039Z 2025-09-07T07:16:29.6696154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6696550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6696880Z return mod(**inputs) 2025-09-07T07:16:29.6697274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6697685Z outputs = self.roberta( 2025-09-07T07:16:29.6698080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6698489Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6698889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6699290Z layer_outputs = layer_module( 2025-09-07T07:16:29.6699669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6700039Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6700443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6700863Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6701258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6701641Z return func(*args, **kwargs) 2025-09-07T07:16:29.6702035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6702437Z self_outputs = self.self( 2025-09-07T07:16:29.6702825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6703225Z return func(*args, **kwargs) 2025-09-07T07:16:29.6703637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6704079Z self.value(current_states) 2025-09-07T07:16:29.6704219Z 2025-09-07T07:16:29.6704309Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6704569Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6704988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6705347Z return mod(**inputs) 2025-09-07T07:16:29.6705833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6706292Z outputs = self.roberta( 2025-09-07T07:16:29.6706715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6707164Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6707595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6708017Z layer_outputs = layer_module( 2025-09-07T07:16:29.6708375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6708748Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6709162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6709574Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6709967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6710352Z return func(*args, **kwargs) 2025-09-07T07:16:29.6710744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6711151Z self_outputs = self.self( 2025-09-07T07:16:29.6711520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6711932Z return func(*args, **kwargs) 2025-09-07T07:16:29.6712333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6712812Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6713014Z 2025-09-07T07:16:29.6713122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6713488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6713820Z return mod(**inputs) 2025-09-07T07:16:29.6714204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6714623Z outputs = self.roberta( 2025-09-07T07:16:29.6714999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6715412Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6715811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6716227Z layer_outputs = layer_module( 2025-09-07T07:16:29.6716572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6716935Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6717331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6717736Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6718121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6718487Z return func(*args, **kwargs) 2025-09-07T07:16:29.6718913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6719362Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6720014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6720436Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6720582Z 2025-09-07T07:16:29.6720688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6721052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6721381Z return mod(**inputs) 2025-09-07T07:16:29.6721761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6722159Z outputs = self.roberta( 2025-09-07T07:16:29.6722530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6722931Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6723324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6723716Z layer_outputs = layer_module( 2025-09-07T07:16:29.6724057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6724423Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6724821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6725229Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6725634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6726061Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6726489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6726963Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6727407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6727815Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6727954Z 2025-09-07T07:16:29.6728059Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6728427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6728753Z return mod(**inputs) 2025-09-07T07:16:29.6729175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6729558Z outputs = self.roberta( 2025-09-07T07:16:29.6729933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6730335Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6730740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6731129Z layer_outputs = layer_module( 2025-09-07T07:16:29.6731473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6731841Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6732242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6732655Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6733086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6733483Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6733920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6734449Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6734925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6735378Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6735771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6736130Z return self.act(input) 2025-09-07T07:16:29.6736250Z 2025-09-07T07:16:29.6736370Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6736749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6737089Z return mod(**inputs) 2025-09-07T07:16:29.6737483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6737901Z outputs = self.roberta( 2025-09-07T07:16:29.6738282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6738682Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6739080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6739495Z layer_outputs = layer_module( 2025-09-07T07:16:29.6739859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6740235Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6740646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6741094Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6741508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6741907Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6742376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6742894Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6743394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6743862Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6744011Z 2025-09-07T07:16:29.6744134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6744539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6744896Z return mod(**inputs) 2025-09-07T07:16:29.6745304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6745792Z outputs = self.roberta( 2025-09-07T07:16:29.6746227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6746670Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6747099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6747536Z layer_outputs = layer_module( 2025-09-07T07:16:29.6747923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6748364Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6748802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6749257Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6749711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6750129Z return func(*args, **kwargs) 2025-09-07T07:16:29.6750556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6750990Z self_outputs = self.self( 2025-09-07T07:16:29.6751395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6751811Z return func(*args, **kwargs) 2025-09-07T07:16:29.6752236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6752819Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6753111Z 2025-09-07T07:16:29.6753225Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6753618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6753982Z return mod(**inputs) 2025-09-07T07:16:29.6754385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6754762Z outputs = self.roberta( 2025-09-07T07:16:29.6755123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6755509Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6755897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6756300Z layer_outputs = layer_module( 2025-09-07T07:16:29.6756628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6756984Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6757371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6757762Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6758131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6758484Z return func(*args, **kwargs) 2025-09-07T07:16:29.6758853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6759259Z self_outputs = self.self( 2025-09-07T07:16:29.6759615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6759968Z return func(*args, **kwargs) 2025-09-07T07:16:29.6760347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6760732Z self.key(current_states) 2025-09-07T07:16:29.6760843Z 2025-09-07T07:16:29.6760952Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6761306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6761614Z return mod(**inputs) 2025-09-07T07:16:29.6761980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6762370Z outputs = self.roberta( 2025-09-07T07:16:29.6762759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6763138Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6763523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6764576Z layer_outputs = layer_module( 2025-09-07T07:16:29.6764936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6765299Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6765700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6766091Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6766464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6766828Z return func(*args, **kwargs) 2025-09-07T07:16:29.6767213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6767600Z self_outputs = self.self( 2025-09-07T07:16:29.6767966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6768347Z return func(*args, **kwargs) 2025-09-07T07:16:29.6768748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6769120Z self.value(current_states) 2025-09-07T07:16:29.6769242Z 2025-09-07T07:16:29.6769323Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6769563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6769917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6770231Z return mod(**inputs) 2025-09-07T07:16:29.6770626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6771019Z outputs = self.roberta( 2025-09-07T07:16:29.6771395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6771791Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6772174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6772570Z layer_outputs = layer_module( 2025-09-07T07:16:29.6772919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6773284Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6773721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6774122Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6774503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6774871Z return func(*args, **kwargs) 2025-09-07T07:16:29.6775257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6775646Z self_outputs = self.self( 2025-09-07T07:16:29.6775999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6776367Z return func(*args, **kwargs) 2025-09-07T07:16:29.6776750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6777203Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6777389Z 2025-09-07T07:16:29.6777517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6777879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6778206Z return mod(**inputs) 2025-09-07T07:16:29.6778597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6778992Z outputs = self.roberta( 2025-09-07T07:16:29.6779362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6779763Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6780155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6780552Z layer_outputs = layer_module( 2025-09-07T07:16:29.6780903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6781273Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6781680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6782110Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6782526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6782921Z return func(*args, **kwargs) 2025-09-07T07:16:29.6783344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6783807Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6784288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6784734Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6784913Z 2025-09-07T07:16:29.6785030Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6785424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6785857Z return mod(**inputs) 2025-09-07T07:16:29.6786275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6786703Z outputs = self.roberta( 2025-09-07T07:16:29.6787111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6787524Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6787929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6788355Z layer_outputs = layer_module( 2025-09-07T07:16:29.6788710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6789091Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6789490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6789900Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6790301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6790693Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6791122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6791595Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6792040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6792476Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6792613Z 2025-09-07T07:16:29.6792714Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6793085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6793405Z return mod(**inputs) 2025-09-07T07:16:29.6793771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6794169Z outputs = self.roberta( 2025-09-07T07:16:29.6794552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6794956Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6795386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6795815Z layer_outputs = layer_module( 2025-09-07T07:16:29.6796157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6796519Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6796923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6797338Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6797724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6798098Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6798520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6798992Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6799431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6799891Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6800268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6800617Z return self.act(input) 2025-09-07T07:16:29.6800729Z 2025-09-07T07:16:29.6800840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6801222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6801561Z return mod(**inputs) 2025-09-07T07:16:29.6801937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6802398Z outputs = self.roberta( 2025-09-07T07:16:29.6802778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6803177Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6803574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6803999Z layer_outputs = layer_module( 2025-09-07T07:16:29.6804375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6804768Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6805195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6805654Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6806110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6806555Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6807058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6807597Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6808130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6808571Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6808722Z 2025-09-07T07:16:29.6808842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6809238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6809605Z return mod(**inputs) 2025-09-07T07:16:29.6810015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6810446Z outputs = self.roberta( 2025-09-07T07:16:29.6810854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6811282Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6811697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6812119Z layer_outputs = layer_module( 2025-09-07T07:16:29.6812496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6812889Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6813308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6813747Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6814166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6814570Z return func(*args, **kwargs) 2025-09-07T07:16:29.6815017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6815439Z self_outputs = self.self( 2025-09-07T07:16:29.6815845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6816265Z return func(*args, **kwargs) 2025-09-07T07:16:29.6816693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6817292Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6817575Z 2025-09-07T07:16:29.6817686Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6818106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6818461Z return mod(**inputs) 2025-09-07T07:16:29.6818872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6819296Z outputs = self.roberta( 2025-09-07T07:16:29.6819847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6820289Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6820719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6821149Z layer_outputs = layer_module( 2025-09-07T07:16:29.6821522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6821921Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6822425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6822867Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6823284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6823710Z return func(*args, **kwargs) 2025-09-07T07:16:29.6824138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6824568Z self_outputs = self.self( 2025-09-07T07:16:29.6824963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6825362Z return func(*args, **kwargs) 2025-09-07T07:16:29.6825841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6826277Z self.key(current_states) 2025-09-07T07:16:29.6826403Z 2025-09-07T07:16:29.6826529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6826927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6827274Z return mod(**inputs) 2025-09-07T07:16:29.6827684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6828118Z outputs = self.roberta( 2025-09-07T07:16:29.6828529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6828958Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6829344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6829733Z layer_outputs = layer_module( 2025-09-07T07:16:29.6830078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6830477Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6830861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6831265Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6831648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6832019Z return func(*args, **kwargs) 2025-09-07T07:16:29.6832401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6832789Z self_outputs = self.self( 2025-09-07T07:16:29.6833146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6833535Z return func(*args, **kwargs) 2025-09-07T07:16:29.6833908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6834286Z self.value(current_states) 2025-09-07T07:16:29.6834408Z 2025-09-07T07:16:29.6834489Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6834727Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6835078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6835390Z return mod(**inputs) 2025-09-07T07:16:29.6835758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6836148Z outputs = self.roberta( 2025-09-07T07:16:29.6836518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6836912Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6837317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6837717Z layer_outputs = layer_module( 2025-09-07T07:16:29.6838056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6838428Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6838818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6839207Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6839581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6839946Z return func(*args, **kwargs) 2025-09-07T07:16:29.6840317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6840699Z self_outputs = self.self( 2025-09-07T07:16:29.6841045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6841408Z return func(*args, **kwargs) 2025-09-07T07:16:29.6841780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6842228Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6842407Z 2025-09-07T07:16:29.6842509Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6842863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6843176Z return mod(**inputs) 2025-09-07T07:16:29.6843541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6843925Z outputs = self.roberta( 2025-09-07T07:16:29.6844310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6844704Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6845098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6845491Z layer_outputs = layer_module( 2025-09-07T07:16:29.6845838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6846193Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6846588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6846993Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6847394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6847764Z return func(*args, **kwargs) 2025-09-07T07:16:29.6848147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6848597Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6849045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6849448Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6849586Z 2025-09-07T07:16:29.6849689Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6850047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6850367Z return mod(**inputs) 2025-09-07T07:16:29.6850745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6851136Z outputs = self.roberta( 2025-09-07T07:16:29.6851520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6851913Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6852321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6852716Z layer_outputs = layer_module( 2025-09-07T07:16:29.6853055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6853418Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6853822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6854231Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6854634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6855024Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6855455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6855947Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6856407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6856826Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6856967Z 2025-09-07T07:16:29.6857075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6857451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6857782Z return mod(**inputs) 2025-09-07T07:16:29.6858162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6858581Z outputs = self.roberta( 2025-09-07T07:16:29.6858967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6859372Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6859769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6860184Z layer_outputs = layer_module( 2025-09-07T07:16:29.6860534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6860895Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6861295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6861725Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6862138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6862574Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6863038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6863548Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6864027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6864506Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6864911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6865283Z return self.act(input) 2025-09-07T07:16:29.6865404Z 2025-09-07T07:16:29.6865524Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6866087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6866462Z return mod(**inputs) 2025-09-07T07:16:29.6866905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6867353Z outputs = self.roberta( 2025-09-07T07:16:29.6867758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6868194Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6868611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6869038Z layer_outputs = layer_module( 2025-09-07T07:16:29.6869420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6869819Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6870245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6870688Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6871123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6871547Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6872012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6872529Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6873024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6873465Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6873657Z 2025-09-07T07:16:29.6873780Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6874163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6874481Z return mod(**inputs) 2025-09-07T07:16:29.6874856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6875248Z outputs = self.roberta( 2025-09-07T07:16:29.6875620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6876014Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6876396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6876813Z layer_outputs = layer_module( 2025-09-07T07:16:29.6877170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6877542Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6877947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6878366Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6878761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6879144Z return func(*args, **kwargs) 2025-09-07T07:16:29.6879535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6879927Z self_outputs = self.self( 2025-09-07T07:16:29.6880299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6880682Z return func(*args, **kwargs) 2025-09-07T07:16:29.6881096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6881631Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6881894Z 2025-09-07T07:16:29.6882027Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6882391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6882717Z return mod(**inputs) 2025-09-07T07:16:29.6883094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6883488Z outputs = self.roberta( 2025-09-07T07:16:29.6883870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6884265Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6884665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6885065Z layer_outputs = layer_module( 2025-09-07T07:16:29.6885418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6885792Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6886210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6886616Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6887000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6887430Z return func(*args, **kwargs) 2025-09-07T07:16:29.6887829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6888251Z self_outputs = self.self( 2025-09-07T07:16:29.6888619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6888991Z return func(*args, **kwargs) 2025-09-07T07:16:29.6889381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6889791Z self.key(current_states) 2025-09-07T07:16:29.6889905Z 2025-09-07T07:16:29.6890016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6890378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6890699Z return mod(**inputs) 2025-09-07T07:16:29.6891081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6891506Z outputs = self.roberta( 2025-09-07T07:16:29.6891894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6892292Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6892693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6893096Z layer_outputs = layer_module( 2025-09-07T07:16:29.6893457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6893832Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6894232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6894654Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6895044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6895445Z return func(*args, **kwargs) 2025-09-07T07:16:29.6895841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6896239Z self_outputs = self.self( 2025-09-07T07:16:29.6896635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6897021Z return func(*args, **kwargs) 2025-09-07T07:16:29.6897423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6897826Z self.value(current_states) 2025-09-07T07:16:29.6897955Z 2025-09-07T07:16:29.6898039Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6898291Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6898667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6899008Z return mod(**inputs) 2025-09-07T07:16:29.6899393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6899834Z outputs = self.roberta( 2025-09-07T07:16:29.6900264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6900713Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6901149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6901594Z layer_outputs = layer_module( 2025-09-07T07:16:29.6901976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6902377Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6902830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6903299Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6903712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6904120Z return func(*args, **kwargs) 2025-09-07T07:16:29.6904544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6904977Z self_outputs = self.self( 2025-09-07T07:16:29.6905356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6905850Z return func(*args, **kwargs) 2025-09-07T07:16:29.6906275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6906822Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6907041Z 2025-09-07T07:16:29.6907164Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6907552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6907912Z return mod(**inputs) 2025-09-07T07:16:29.6908336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6908780Z outputs = self.roberta( 2025-09-07T07:16:29.6909211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6909652Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6910091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6910531Z layer_outputs = layer_module( 2025-09-07T07:16:29.6910944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6911350Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6911820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6912278Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6912712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6913126Z return func(*args, **kwargs) 2025-09-07T07:16:29.6913558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6914061Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6914567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6915027Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6915182Z 2025-09-07T07:16:29.6915300Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6915705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6916062Z return mod(**inputs) 2025-09-07T07:16:29.6916484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6916922Z outputs = self.roberta( 2025-09-07T07:16:29.6917333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6917772Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6918210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6918649Z layer_outputs = layer_module( 2025-09-07T07:16:29.6919063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6919459Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6920066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6920515Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6920952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6921372Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6921837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6922414Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6922897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6923338Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6923488Z 2025-09-07T07:16:29.6923603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6924002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6924357Z return mod(**inputs) 2025-09-07T07:16:29.6924766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6925196Z outputs = self.roberta( 2025-09-07T07:16:29.6925599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6926028Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6926454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6926914Z layer_outputs = layer_module( 2025-09-07T07:16:29.6927293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6927700Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6928129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6928539Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6928938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6929323Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6929756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6930236Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6930681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6931115Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6931491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6931840Z return self.act(input) 2025-09-07T07:16:29.6931960Z 2025-09-07T07:16:29.6932068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6932432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6932754Z return mod(**inputs) 2025-09-07T07:16:29.6933131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6933530Z outputs = self.roberta( 2025-09-07T07:16:29.6933905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6934334Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6934734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6935128Z layer_outputs = layer_module( 2025-09-07T07:16:29.6935475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6935838Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6936230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6936639Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6937061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6937450Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6937876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6938350Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6938796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6939208Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6939342Z 2025-09-07T07:16:29.6939451Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6939804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6939870Z return mod(**inputs) 2025-09-07T07:16:29.6940135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6940210Z outputs = self.roberta( 2025-09-07T07:16:29.6940504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6940585Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6940869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6940951Z layer_outputs = layer_module( 2025-09-07T07:16:29.6941171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6941250Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6941514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6941600Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6941856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6941929Z return func(*args, **kwargs) 2025-09-07T07:16:29.6942197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6942279Z self_outputs = self.self( 2025-09-07T07:16:29.6942526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6942605Z return func(*args, **kwargs) 2025-09-07T07:16:29.6942873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6943089Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6943103Z 2025-09-07T07:16:29.6943212Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6943418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6943518Z return mod(**inputs) 2025-09-07T07:16:29.6943788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6943864Z outputs = self.roberta( 2025-09-07T07:16:29.6944136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6944210Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6944483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6944556Z layer_outputs = layer_module( 2025-09-07T07:16:29.6944789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6944890Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6945162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6945260Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6945522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6945603Z return func(*args, **kwargs) 2025-09-07T07:16:29.6945972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6946064Z self_outputs = self.self( 2025-09-07T07:16:29.6946333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6946409Z return func(*args, **kwargs) 2025-09-07T07:16:29.6946721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6946801Z self.key(current_states) 2025-09-07T07:16:29.6946832Z 2025-09-07T07:16:29.6946956Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6947172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6947263Z return mod(**inputs) 2025-09-07T07:16:29.6947566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6947633Z outputs = self.roberta( 2025-09-07T07:16:29.6947894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6947965Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6948216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6948296Z layer_outputs = layer_module( 2025-09-07T07:16:29.6948511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6948598Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6948865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6948952Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6949184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6949251Z return func(*args, **kwargs) 2025-09-07T07:16:29.6949536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6949612Z self_outputs = self.self( 2025-09-07T07:16:29.6949880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6949953Z return func(*args, **kwargs) 2025-09-07T07:16:29.6950295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6950382Z self.value(current_states) 2025-09-07T07:16:29.6950385Z 2025-09-07T07:16:29.6950474Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6950591Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6950817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6950888Z return mod(**inputs) 2025-09-07T07:16:29.6951188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6951253Z outputs = self.roberta( 2025-09-07T07:16:29.6951535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6951605Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6951876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6951945Z layer_outputs = layer_module( 2025-09-07T07:16:29.6952160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6952243Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6952499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6952585Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6952820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6952888Z return func(*args, **kwargs) 2025-09-07T07:16:29.6953195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6953266Z self_outputs = self.self( 2025-09-07T07:16:29.6953509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6953575Z return func(*args, **kwargs) 2025-09-07T07:16:29.6953846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6953986Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6953989Z 2025-09-07T07:16:29.6954092Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6954298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6954365Z return mod(**inputs) 2025-09-07T07:16:29.6954635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6954705Z outputs = self.roberta( 2025-09-07T07:16:29.6954971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6955048Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6955303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6955382Z layer_outputs = layer_module( 2025-09-07T07:16:29.6955599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6955676Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6955941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6956024Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6956271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6956359Z return func(*args, **kwargs) 2025-09-07T07:16:29.6956638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6956764Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6957021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6957107Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6957110Z 2025-09-07T07:16:29.6957211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6957412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6957494Z return mod(**inputs) 2025-09-07T07:16:29.6957756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6957832Z outputs = self.roberta( 2025-09-07T07:16:29.6958099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6958178Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6958435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6958505Z layer_outputs = layer_module( 2025-09-07T07:16:29.6958733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6958810Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6959078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6959162Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6959441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6959521Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6959824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6959956Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6960217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6960305Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6960308Z 2025-09-07T07:16:29.6960408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6960617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6960682Z return mod(**inputs) 2025-09-07T07:16:29.6960948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6961024Z outputs = self.roberta( 2025-09-07T07:16:29.6961285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6961363Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6961625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6961695Z layer_outputs = layer_module( 2025-09-07T07:16:29.6961919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6961997Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6962263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6962364Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6962622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6962708Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6963001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6963128Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6963392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6963513Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6963749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6963821Z return self.act(input) 2025-09-07T07:16:29.6963826Z 2025-09-07T07:16:29.6963938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6964139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6964212Z return mod(**inputs) 2025-09-07T07:16:29.6964482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6964550Z outputs = self.roberta( 2025-09-07T07:16:29.6964816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6964888Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6965157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6965229Z layer_outputs = layer_module( 2025-09-07T07:16:29.6965475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6965572Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6965872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6965990Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6966268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6966352Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6966651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6966785Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6967064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6967151Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6967154Z 2025-09-07T07:16:29.6967268Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6967478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6967550Z return mod(**inputs) 2025-09-07T07:16:29.6967815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6967882Z outputs = self.roberta( 2025-09-07T07:16:29.6968150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6968220Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6968490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6968561Z layer_outputs = layer_module( 2025-09-07T07:16:29.6968798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6968883Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6969149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6969238Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6969477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6969548Z return func(*args, **kwargs) 2025-09-07T07:16:29.6969825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6969911Z self_outputs = self.self( 2025-09-07T07:16:29.6970152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6970225Z return func(*args, **kwargs) 2025-09-07T07:16:29.6970482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6970687Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6970691Z 2025-09-07T07:16:29.6970791Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6970994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6971058Z return mod(**inputs) 2025-09-07T07:16:29.6971319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6971387Z outputs = self.roberta( 2025-09-07T07:16:29.6971639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6971733Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6971986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6972061Z layer_outputs = layer_module( 2025-09-07T07:16:29.6972289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6972372Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6972622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6972700Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6972937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6973005Z return func(*args, **kwargs) 2025-09-07T07:16:29.6973259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6973329Z self_outputs = self.self( 2025-09-07T07:16:29.6973560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6973635Z return func(*args, **kwargs) 2025-09-07T07:16:29.6973882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.6973956Z self.key(current_states) 2025-09-07T07:16:29.6973959Z 2025-09-07T07:16:29.6974059Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6974249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6974321Z return mod(**inputs) 2025-09-07T07:16:29.6974575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6974667Z outputs = self.roberta( 2025-09-07T07:16:29.6974921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6974998Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6975267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6975340Z layer_outputs = layer_module( 2025-09-07T07:16:29.6975576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6975656Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6975929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6976030Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6976283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6976361Z return func(*args, **kwargs) 2025-09-07T07:16:29.6976623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6976701Z self_outputs = self.self( 2025-09-07T07:16:29.6976943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6977013Z return func(*args, **kwargs) 2025-09-07T07:16:29.6977279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.6977349Z self.value(current_states) 2025-09-07T07:16:29.6977352Z 2025-09-07T07:16:29.6977444Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.6977548Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6977770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6977840Z return mod(**inputs) 2025-09-07T07:16:29.6978099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6978191Z outputs = self.roberta( 2025-09-07T07:16:29.6978450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6978533Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6978795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6978866Z layer_outputs = layer_module( 2025-09-07T07:16:29.6979094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6979175Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6979444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6979524Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6979766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6979842Z return func(*args, **kwargs) 2025-09-07T07:16:29.6980101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6980176Z self_outputs = self.self( 2025-09-07T07:16:29.6980410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6980486Z return func(*args, **kwargs) 2025-09-07T07:16:29.6980747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.6980901Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.6980904Z 2025-09-07T07:16:29.6981015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6981215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6981287Z return mod(**inputs) 2025-09-07T07:16:29.6981552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6981620Z outputs = self.roberta( 2025-09-07T07:16:29.6981894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6981966Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6982249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6982319Z layer_outputs = layer_module( 2025-09-07T07:16:29.6982556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6982635Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6982900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6982991Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6983228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6983305Z return func(*args, **kwargs) 2025-09-07T07:16:29.6983566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.6983696Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.6983984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.6984069Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6984073Z 2025-09-07T07:16:29.6984182Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6984399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6984468Z return mod(**inputs) 2025-09-07T07:16:29.6984745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6984816Z outputs = self.roberta( 2025-09-07T07:16:29.6985089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6985163Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6985438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6985513Z layer_outputs = layer_module( 2025-09-07T07:16:29.6985810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6985910Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6986183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6986278Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6986559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6986642Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6986970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6987102Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6987441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.6987531Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6987535Z 2025-09-07T07:16:29.6987655Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6987870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6987938Z return mod(**inputs) 2025-09-07T07:16:29.6988217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6988287Z outputs = self.roberta( 2025-09-07T07:16:29.6988561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6988658Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6988927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6989009Z layer_outputs = layer_module( 2025-09-07T07:16:29.6989234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6989322Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6989599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6989688Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6989945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6990021Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6990320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.6990457Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.6990730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.6990859Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.6991069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.6991146Z return self.act(input) 2025-09-07T07:16:29.6991150Z 2025-09-07T07:16:29.6991251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6991458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6991522Z return mod(**inputs) 2025-09-07T07:16:29.6991794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6991862Z outputs = self.roberta( 2025-09-07T07:16:29.6992129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6992210Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6992476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6992551Z layer_outputs = layer_module( 2025-09-07T07:16:29.6992776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6992854Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6993120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.6993202Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.6993471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.6993566Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.6993856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.6993998Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.6994260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.6994349Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.6994353Z 2025-09-07T07:16:29.6994458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6994669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6994756Z return mod(**inputs) 2025-09-07T07:16:29.6995027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6995106Z outputs = self.roberta( 2025-09-07T07:16:29.6995371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6995457Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6995721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6995792Z layer_outputs = layer_module( 2025-09-07T07:16:29.6996032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6996112Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.6996394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.6996481Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.6996756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6996830Z return func(*args, **kwargs) 2025-09-07T07:16:29.6997117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.6997197Z self_outputs = self.self( 2025-09-07T07:16:29.6997443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.6997533Z return func(*args, **kwargs) 2025-09-07T07:16:29.6997799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.6998009Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.6998014Z 2025-09-07T07:16:29.6998124Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.6998325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.6998399Z return mod(**inputs) 2025-09-07T07:16:29.6998663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.6998740Z outputs = self.roberta( 2025-09-07T07:16:29.6999000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.6999075Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.6999344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.6999415Z layer_outputs = layer_module( 2025-09-07T07:16:29.6999652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.6999730Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7000029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7000119Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7000364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7000440Z return func(*args, **kwargs) 2025-09-07T07:16:29.7000697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7000766Z self_outputs = self.self( 2025-09-07T07:16:29.7001010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7001104Z return func(*args, **kwargs) 2025-09-07T07:16:29.7001375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.7001445Z self.key(current_states) 2025-09-07T07:16:29.7001449Z 2025-09-07T07:16:29.7001557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7001754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7001818Z return mod(**inputs) 2025-09-07T07:16:29.7002087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7002153Z outputs = self.roberta( 2025-09-07T07:16:29.7002419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7002490Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7002747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7002825Z layer_outputs = layer_module( 2025-09-07T07:16:29.7003073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7003161Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7003442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7003528Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7003777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7003847Z return func(*args, **kwargs) 2025-09-07T07:16:29.7004118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7004194Z self_outputs = self.self( 2025-09-07T07:16:29.7004446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7004518Z return func(*args, **kwargs) 2025-09-07T07:16:29.7004784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.7004867Z self.value(current_states) 2025-09-07T07:16:29.7004871Z 2025-09-07T07:16:29.7004954Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.7005068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7005267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7005333Z return mod(**inputs) 2025-09-07T07:16:29.7005611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7005681Z outputs = self.roberta( 2025-09-07T07:16:29.7005954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7006047Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7006314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7006393Z layer_outputs = layer_module( 2025-09-07T07:16:29.7006618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7006705Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7006973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7007062Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7007307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7007396Z return func(*args, **kwargs) 2025-09-07T07:16:29.7007674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7007746Z self_outputs = self.self( 2025-09-07T07:16:29.7008006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7008077Z return func(*args, **kwargs) 2025-09-07T07:16:29.7008347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.7008491Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.7008494Z 2025-09-07T07:16:29.7008601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7008811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7008880Z return mod(**inputs) 2025-09-07T07:16:29.7009174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7009247Z outputs = self.roberta( 2025-09-07T07:16:29.7009519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7009626Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7009895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7009973Z layer_outputs = layer_module( 2025-09-07T07:16:29.7010203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7010283Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7010562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7010647Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7010902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7010973Z return func(*args, **kwargs) 2025-09-07T07:16:29.7011243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.7011383Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.7011650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.7011742Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7011746Z 2025-09-07T07:16:29.7011849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7012060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7012126Z return mod(**inputs) 2025-09-07T07:16:29.7012417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7012496Z outputs = self.roberta( 2025-09-07T07:16:29.7012763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7012845Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7013114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7013185Z layer_outputs = layer_module( 2025-09-07T07:16:29.7013415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7013494Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7013788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7013876Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7014151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7014231Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7014536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7014667Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7014938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.7015030Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7015033Z 2025-09-07T07:16:29.7015141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7015356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7015455Z return mod(**inputs) 2025-09-07T07:16:29.7015740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7015826Z outputs = self.roberta( 2025-09-07T07:16:29.7016103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7016186Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7016459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7016532Z layer_outputs = layer_module( 2025-09-07T07:16:29.7016769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7016850Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7017127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7017213Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7017474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7017558Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7017858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7017988Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7018254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.7018374Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.7018594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.7018689Z return self.act(input) 2025-09-07T07:16:29.7018692Z 2025-09-07T07:16:29.7018804Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7019005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7019080Z return mod(**inputs) 2025-09-07T07:16:29.7019351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7019425Z outputs = self.roberta( 2025-09-07T07:16:29.7019906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7019991Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7020281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7020411Z layer_outputs = layer_module( 2025-09-07T07:16:29.7020656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7020755Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7021048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7021149Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7021443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7021531Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7021850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.7021997Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.7022320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.7022413Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7022417Z 2025-09-07T07:16:29.7022539Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7022780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7022854Z return mod(**inputs) 2025-09-07T07:16:29.7023145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7023218Z outputs = self.roberta( 2025-09-07T07:16:29.7023502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7023582Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7023867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7023946Z layer_outputs = layer_module( 2025-09-07T07:16:29.7024184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7024276Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7024581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7024678Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7024939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7025013Z return func(*args, **kwargs) 2025-09-07T07:16:29.7025320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7025398Z self_outputs = self.self( 2025-09-07T07:16:29.7025669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7025830Z return func(*args, **kwargs) 2025-09-07T07:16:29.7026176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.7026411Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.7026415Z 2025-09-07T07:16:29.7026527Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7026750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7026820Z return mod(**inputs) 2025-09-07T07:16:29.7027112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7027211Z outputs = self.roberta( 2025-09-07T07:16:29.7027514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7027605Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7027912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7028000Z layer_outputs = layer_module( 2025-09-07T07:16:29.7028243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7028328Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7028639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7028728Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7029006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7029081Z return func(*args, **kwargs) 2025-09-07T07:16:29.7029395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7029471Z self_outputs = self.self( 2025-09-07T07:16:29.7029752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7029835Z return func(*args, **kwargs) 2025-09-07T07:16:29.7030135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.7030216Z self.key(current_states) 2025-09-07T07:16:29.7030219Z 2025-09-07T07:16:29.7030331Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7030557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7030636Z return mod(**inputs) 2025-09-07T07:16:29.7030939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7031020Z outputs = self.roberta( 2025-09-07T07:16:29.7031321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7031399Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7031707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7031780Z layer_outputs = layer_module( 2025-09-07T07:16:29.7032028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7032113Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7032409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7032491Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7032747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7032822Z return func(*args, **kwargs) 2025-09-07T07:16:29.7033085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7033161Z self_outputs = self.self( 2025-09-07T07:16:29.7033400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7033468Z return func(*args, **kwargs) 2025-09-07T07:16:29.7033738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.7033825Z self.value(current_states) 2025-09-07T07:16:29.7033829Z 2025-09-07T07:16:29.7033917Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.7034018Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7034224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7034288Z return mod(**inputs) 2025-09-07T07:16:29.7034549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7034621Z outputs = self.roberta( 2025-09-07T07:16:29.7034881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7034959Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7035217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7035285Z layer_outputs = layer_module( 2025-09-07T07:16:29.7035507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7035610Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7035875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7035955Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7036211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7036289Z return func(*args, **kwargs) 2025-09-07T07:16:29.7036548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7036624Z self_outputs = self.self( 2025-09-07T07:16:29.7036863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7036934Z return func(*args, **kwargs) 2025-09-07T07:16:29.7037208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.7037341Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.7037345Z 2025-09-07T07:16:29.7037457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7037655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7037728Z return mod(**inputs) 2025-09-07T07:16:29.7037990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7038058Z outputs = self.roberta( 2025-09-07T07:16:29.7038325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7038399Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7038669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7038757Z layer_outputs = layer_module( 2025-09-07T07:16:29.7038976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7039063Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7039325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7039416Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7039659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7039735Z return func(*args, **kwargs) 2025-09-07T07:16:29.7040002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.7040150Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.7040422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.7040505Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7040509Z 2025-09-07T07:16:29.7040620Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7040820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7040886Z return mod(**inputs) 2025-09-07T07:16:29.7041157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7041224Z outputs = self.roberta( 2025-09-07T07:16:29.7041495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7041570Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7041854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7041925Z layer_outputs = layer_module( 2025-09-07T07:16:29.7042146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7042247Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7042509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7042597Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7042852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7042927Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7043230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7043354Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7043619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.7043702Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7043706Z 2025-09-07T07:16:29.7043815Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7044010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7044076Z return mod(**inputs) 2025-09-07T07:16:29.7044346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7044412Z outputs = self.roberta( 2025-09-07T07:16:29.7044677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7044746Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7045023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7045101Z layer_outputs = layer_module( 2025-09-07T07:16:29.7045323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7045407Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7045665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7045747Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7046011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7046106Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7046409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7046532Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7046811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.7046927Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.7047145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.7047224Z return self.act(input) 2025-09-07T07:16:29.7047228Z 2025-09-07T07:16:29.7047331Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7047543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7047610Z return mod(**inputs) 2025-09-07T07:16:29.7047889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7047983Z outputs = self.roberta( 2025-09-07T07:16:29.7048245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7048323Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7048602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7048680Z layer_outputs = layer_module( 2025-09-07T07:16:29.7048900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7048978Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7049243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7049327Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7049590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7049667Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7049965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.7050104Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.7050364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.7050451Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7050454Z 2025-09-07T07:16:29.7050554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7050758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7050822Z return mod(**inputs) 2025-09-07T07:16:29.7051085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7051175Z outputs = self.roberta( 2025-09-07T07:16:29.7051435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7051514Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7051776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7051846Z layer_outputs = layer_module( 2025-09-07T07:16:29.7052070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7052147Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7052467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7052553Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7052793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7052871Z return func(*args, **kwargs) 2025-09-07T07:16:29.7053137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7053213Z self_outputs = self.self( 2025-09-07T07:16:29.7053447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7053520Z return func(*args, **kwargs) 2025-09-07T07:16:29.7053770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.7053973Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.7053976Z 2025-09-07T07:16:29.7054101Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7054298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7054374Z return mod(**inputs) 2025-09-07T07:16:29.7054657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7054727Z outputs = self.roberta( 2025-09-07T07:16:29.7054992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7055065Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7055331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7055401Z layer_outputs = layer_module( 2025-09-07T07:16:29.7055639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7055725Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7056008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7056106Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7056368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7056451Z return func(*args, **kwargs) 2025-09-07T07:16:29.7056732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7056809Z self_outputs = self.self( 2025-09-07T07:16:29.7057075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7057151Z return func(*args, **kwargs) 2025-09-07T07:16:29.7057448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.7057534Z self.key(current_states) 2025-09-07T07:16:29.7057538Z 2025-09-07T07:16:29.7057647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7057845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7057918Z return mod(**inputs) 2025-09-07T07:16:29.7058187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7058253Z outputs = self.roberta( 2025-09-07T07:16:29.7058523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7058616Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7058880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7058960Z layer_outputs = layer_module( 2025-09-07T07:16:29.7059179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7059271Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7059552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7059640Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7059916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7059985Z return func(*args, **kwargs) 2025-09-07T07:16:29.7060262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7060336Z self_outputs = self.self( 2025-09-07T07:16:29.7060607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7060677Z return func(*args, **kwargs) 2025-09-07T07:16:29.7060958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.7061040Z self.value(current_states) 2025-09-07T07:16:29.7061043Z 2025-09-07T07:16:29.7061132Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.7061253Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7061468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7061538Z return mod(**inputs) 2025-09-07T07:16:29.7061828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7061903Z outputs = self.roberta( 2025-09-07T07:16:29.7062193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7062271Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7062553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7062636Z layer_outputs = layer_module( 2025-09-07T07:16:29.7062876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7062969Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7063252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7063345Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7063607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7063701Z return func(*args, **kwargs) 2025-09-07T07:16:29.7063991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7064063Z self_outputs = self.self( 2025-09-07T07:16:29.7064332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7064405Z return func(*args, **kwargs) 2025-09-07T07:16:29.7064683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.7064834Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.7064837Z 2025-09-07T07:16:29.7064945Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7065181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7065251Z return mod(**inputs) 2025-09-07T07:16:29.7065539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7065619Z outputs = self.roberta( 2025-09-07T07:16:29.7065994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7066087Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7066385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7066469Z layer_outputs = layer_module( 2025-09-07T07:16:29.7066708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7066792Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7067086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7067197Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7067470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7067540Z return func(*args, **kwargs) 2025-09-07T07:16:29.7067822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.7067962Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.7068229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.7068319Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7068323Z 2025-09-07T07:16:29.7068429Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7068634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7068702Z return mod(**inputs) 2025-09-07T07:16:29.7068963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7069039Z outputs = self.roberta( 2025-09-07T07:16:29.7069305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7069385Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7069654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7069724Z layer_outputs = layer_module( 2025-09-07T07:16:29.7069963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7070042Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7070315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7070415Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7070667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7070751Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7071043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7071171Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7071431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.7071521Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7071542Z 2025-09-07T07:16:29.7071644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7071843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7071920Z return mod(**inputs) 2025-09-07T07:16:29.7072183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7072258Z outputs = self.roberta( 2025-09-07T07:16:29.7072525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7072596Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7072868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7072936Z layer_outputs = layer_module( 2025-09-07T07:16:29.7073165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7073243Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7073528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7073613Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7073881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7073967Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7074267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7074394Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7074664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.7074782Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.7075009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.7075082Z return self.act(input) 2025-09-07T07:16:29.7075086Z 2025-09-07T07:16:29.7075197Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7075402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7075477Z return mod(**inputs) 2025-09-07T07:16:29.7075750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7075818Z outputs = self.roberta( 2025-09-07T07:16:29.7076097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7076171Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7076448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7076541Z layer_outputs = layer_module( 2025-09-07T07:16:29.7076770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7076857Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7077136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7077227Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7077492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7077577Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7077883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.7078035Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.7078303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.7078385Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7078389Z 2025-09-07T07:16:29.7078500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7078697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7078764Z return mod(**inputs) 2025-09-07T07:16:29.7079036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7079101Z outputs = self.roberta( 2025-09-07T07:16:29.7079364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7079436Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7079712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7079792Z layer_outputs = layer_module( 2025-09-07T07:16:29.7080013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7080119Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7080379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7080468Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7080709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7080777Z return func(*args, **kwargs) 2025-09-07T07:16:29.7081045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7081113Z self_outputs = self.self( 2025-09-07T07:16:29.7081361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7081429Z return func(*args, **kwargs) 2025-09-07T07:16:29.7081689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-09-07T07:16:29.7081902Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-09-07T07:16:29.7081906Z 2025-09-07T07:16:29.7082008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7082212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7082277Z return mod(**inputs) 2025-09-07T07:16:29.7082547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7082614Z outputs = self.roberta( 2025-09-07T07:16:29.7082899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7082979Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7083238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7083314Z layer_outputs = layer_module( 2025-09-07T07:16:29.7083531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7083609Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7083875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7083974Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7084219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7084291Z return func(*args, **kwargs) 2025-09-07T07:16:29.7084561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7084630Z self_outputs = self.self( 2025-09-07T07:16:29.7084871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7084948Z return func(*args, **kwargs) 2025-09-07T07:16:29.7085208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-09-07T07:16:29.7085283Z self.key(current_states) 2025-09-07T07:16:29.7085286Z 2025-09-07T07:16:29.7085390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7085592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7085667Z return mod(**inputs) 2025-09-07T07:16:29.7085957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7086034Z outputs = self.roberta( 2025-09-07T07:16:29.7086334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7086410Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7086685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7086755Z layer_outputs = layer_module( 2025-09-07T07:16:29.7086989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7087072Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7087356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7087440Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7087681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7087757Z return func(*args, **kwargs) 2025-09-07T07:16:29.7088032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7088111Z self_outputs = self.self( 2025-09-07T07:16:29.7088357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7088426Z return func(*args, **kwargs) 2025-09-07T07:16:29.7088701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-09-07T07:16:29.7088774Z self.value(current_states) 2025-09-07T07:16:29.7088777Z 2025-09-07T07:16:29.7088869Z cudagraph partition due to non gpu ops 2025-09-07T07:16:29.7088994Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7089193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7089268Z return mod(**inputs) 2025-09-07T07:16:29.7089542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7089618Z outputs = self.roberta( 2025-09-07T07:16:29.7089885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7089966Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7090231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7090322Z layer_outputs = layer_module( 2025-09-07T07:16:29.7090555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7090637Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7090913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7090999Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7091242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7091321Z return func(*args, **kwargs) 2025-09-07T07:16:29.7091593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-09-07T07:16:29.7091671Z self_outputs = self.self( 2025-09-07T07:16:29.7091914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7091983Z return func(*args, **kwargs) 2025-09-07T07:16:29.7092273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-09-07T07:16:29.7092410Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-09-07T07:16:29.7092414Z 2025-09-07T07:16:29.7092541Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7092743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7092817Z return mod(**inputs) 2025-09-07T07:16:29.7093087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7093155Z outputs = self.roberta( 2025-09-07T07:16:29.7093430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7093504Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7093780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7093854Z layer_outputs = layer_module( 2025-09-07T07:16:29.7094077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7094163Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7094430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-09-07T07:16:29.7094519Z self_attention_outputs = self.attention( 2025-09-07T07:16:29.7094762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-09-07T07:16:29.7094840Z return func(*args, **kwargs) 2025-09-07T07:16:29.7095107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-09-07T07:16:29.7095262Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:16:29.7095534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-09-07T07:16:29.7095618Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7095623Z 2025-09-07T07:16:29.7095734Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7095939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7096005Z return mod(**inputs) 2025-09-07T07:16:29.7096283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7096350Z outputs = self.roberta( 2025-09-07T07:16:29.7096640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7096714Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7096987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7097065Z layer_outputs = layer_module( 2025-09-07T07:16:29.7097292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7097379Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7097645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7097739Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7098004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7098084Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7098413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7098538Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7098830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-09-07T07:16:29.7098917Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7098920Z 2025-09-07T07:16:29.7099032Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7099236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7099302Z return mod(**inputs) 2025-09-07T07:16:29.7099580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7099649Z outputs = self.roberta( 2025-09-07T07:16:29.7099927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7100003Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7100271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7100354Z layer_outputs = layer_module( 2025-09-07T07:16:29.7100582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7100669Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7100936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7101020Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7101299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7101378Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7101706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-09-07T07:16:29.7101827Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:16:29.7102104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-09-07T07:16:29.7102219Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:16:29.7102435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:16:29.7102517Z return self.act(input) 2025-09-07T07:16:29.7102521Z 2025-09-07T07:16:29.7102625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7102836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7102923Z return mod(**inputs) 2025-09-07T07:16:29.7103201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-09-07T07:16:29.7103282Z outputs = self.roberta( 2025-09-07T07:16:29.7103567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-09-07T07:16:29.7103652Z encoder_outputs = self.encoder( 2025-09-07T07:16:29.7103939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-09-07T07:16:29.7104015Z layer_outputs = layer_module( 2025-09-07T07:16:29.7104270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:29.7104355Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:29.7104662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-09-07T07:16:29.7104775Z layer_output = apply_chunking_to_forward( 2025-09-07T07:16:29.7105073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:16:29.7105157Z return forward_fn(*input_tensors) 2025-09-07T07:16:29.7105510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-09-07T07:16:29.7105669Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:16:29.7106181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-09-07T07:16:29.7106289Z hidden_states = self.dense(hidden_states) 2025-09-07T07:16:29.7106293Z 2025-09-07T07:16:29.7106409Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7106641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7106717Z return mod(**inputs) 2025-09-07T07:16:29.7107014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1530, in forward 2025-09-07T07:16:29.7107116Z logits = self.qa_outputs(sequence_output) 2025-09-07T07:16:29.7107119Z 2025-09-07T07:16:29.7107235Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7107461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7107533Z return mod(**inputs) 2025-09-07T07:16:29.7107827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in forward 2025-09-07T07:16:29.7107953Z start_loss = loss_fct(start_logits, start_positions) 2025-09-07T07:16:29.7107959Z 2025-09-07T07:16:29.7108072Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:29.7108299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:29.7108399Z return mod(**inputs) 2025-09-07T07:16:29.7108698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in forward 2025-09-07T07:16:29.7108817Z end_loss = loss_fct(end_logits, end_positions) 2025-09-07T07:16:29.7108822Z 2025-09-07T07:16:39.9265567Z Compilation time (from dynamo_timed): 16.849679398 2025-09-07T07:16:39.9266201Z pass 2025-09-07T07:16:39.9266560Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:39.9267472Z TIMING: _recursive_pre_grad_passes:0.0077 _recursive_joint_graph_passes:0.37077 _recursive_post_grad_passes:0.08187 async_compile.wait:0.00279 code_gen:9.5623 inductor_compile:10.84509 backend_compile:14.07578 gc:0.00164 entire_frame_compile:16.84968 total_wall_time:16.84968 2025-09-07T07:16:39.9268830Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12459 | FakeTensor.__torch_dispatch__:4435 | ProxyTorchDispatchMode.__torch_dispatch__:4566 2025-09-07T07:16:39.9269442Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-09-07T07:16:42.6840287Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:16:42.6841147Z import pynvml # type: ignore[import] 2025-09-07T07:16:45.4512849Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:16:45.4513925Z from pkg_resources import resource_filename 2025-09-07T07:16:46.1507905Z 2025-09-07T07:16:47.0964425Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:16:47.0964866Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:16:47.0979147Z cpu eval T5ForConditionalGeneration 2025-09-07T07:16:48.1810337Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:48.5638753Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:48.9696375Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:16:58.7629405Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.7632502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7633085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7633542Z return mod(**inputs) 2025-09-07T07:16:58.7633982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7634430Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7634846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7635245Z layer_outputs = layer_module( 2025-09-07T07:16:58.7635782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7636190Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7636593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7636998Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7637416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7637855Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7638283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward 2025-09-07T07:16:58.7639038Z position_bias = position_bias + causal_mask 2025-09-07T07:16:58.7639201Z 2025-09-07T07:16:58.7639321Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7639727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7640149Z return mod(**inputs) 2025-09-07T07:16:58.7640533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7640936Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7641337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7641801Z layer_outputs = layer_module( 2025-09-07T07:16:58.7642174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7642573Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7642965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7643374Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7643766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.7644188Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.7644613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.7644989Z return self.weight * hidden_states 2025-09-07T07:16:58.7645144Z 2025-09-07T07:16:58.7645275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7645660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7646032Z return mod(**inputs) 2025-09-07T07:16:58.7646549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7646970Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7647442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7647844Z layer_outputs = layer_module( 2025-09-07T07:16:58.7648204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7648563Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7648941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7649323Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7649724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7650147Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7650552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.7650960Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.7651115Z 2025-09-07T07:16:58.7651232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7651625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7651974Z return mod(**inputs) 2025-09-07T07:16:58.7652351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7652739Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7653140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7653550Z layer_outputs = layer_module( 2025-09-07T07:16:58.7653978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7654399Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7654822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7655244Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7655660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7656188Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7656592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.7657011Z key_states = self.k(current_states) 2025-09-07T07:16:58.7657148Z 2025-09-07T07:16:58.7657262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7657644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7657997Z return mod(**inputs) 2025-09-07T07:16:58.7658373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7658795Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7659208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7659604Z layer_outputs = layer_module( 2025-09-07T07:16:58.7659978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7660382Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7660796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7661215Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7661665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7662092Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7662534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.7663013Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.7663220Z 2025-09-07T07:16:58.7663339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7663753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7664129Z return mod(**inputs) 2025-09-07T07:16:58.7664523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7664948Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7665356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7666045Z layer_outputs = layer_module( 2025-09-07T07:16:58.7666455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7666883Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7667307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7667733Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7668146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7668602Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7669016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7669504Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7669773Z 2025-09-07T07:16:58.7669886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7670288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7670651Z return mod(**inputs) 2025-09-07T07:16:58.7671032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7671440Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7671840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7672248Z layer_outputs = layer_module( 2025-09-07T07:16:58.7672635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7673038Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7673441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7673849Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7674263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7674670Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7675079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.7675490Z value_states = self.v(current_states) 2025-09-07T07:16:58.7675649Z 2025-09-07T07:16:58.7675763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7676167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7676552Z return mod(**inputs) 2025-09-07T07:16:58.7676977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7677404Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7677822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7678253Z layer_outputs = layer_module( 2025-09-07T07:16:58.7678625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7679023Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7679427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7679835Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7680246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7680649Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7681062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7681515Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7681690Z 2025-09-07T07:16:58.7681811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7682200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7682546Z return mod(**inputs) 2025-09-07T07:16:58.7683075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7683487Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7683931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7684330Z layer_outputs = layer_module( 2025-09-07T07:16:58.7684719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7685171Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7685588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7686010Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7686417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7686871Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7687298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7687762Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7687943Z 2025-09-07T07:16:58.7688087Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7688488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7688857Z return mod(**inputs) 2025-09-07T07:16:58.7689270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7689707Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7690140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7690561Z layer_outputs = layer_module( 2025-09-07T07:16:58.7691024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7703985Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7704611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7705095Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7705634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7706189Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7706617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.7707120Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.7707322Z 2025-09-07T07:16:58.7707450Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7707877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7708245Z return mod(**inputs) 2025-09-07T07:16:58.7708632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7709062Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7709476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7709894Z layer_outputs = layer_module( 2025-09-07T07:16:58.7710281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7710703Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7711115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7711535Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7711953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7712365Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7712779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.7713190Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.7713334Z 2025-09-07T07:16:58.7713459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7713894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7714254Z return mod(**inputs) 2025-09-07T07:16:58.7714643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.7715051Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.7715450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7715842Z layer_outputs = layer_module( 2025-09-07T07:16:58.7716229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7716633Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7717081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.7717495Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.7717900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.7718315Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.7718728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.7719134Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.7719283Z 2025-09-07T07:16:58.7719407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7720097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7720549Z return mod(**inputs) 2025-09-07T07:16:58.7720933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7721343Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7721831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7722215Z layer_outputs = layer_module( 2025-09-07T07:16:58.7722606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7722985Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7723367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7723746Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7724137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7724530Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7724920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.7725300Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.7725448Z 2025-09-07T07:16:58.7725559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7725936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7726276Z return mod(**inputs) 2025-09-07T07:16:58.7726633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7727008Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7727383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7727757Z layer_outputs = layer_module( 2025-09-07T07:16:58.7728113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7728484Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7728859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7729304Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7729691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7730076Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7730449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.7730828Z key_states = self.k(current_states) 2025-09-07T07:16:58.7730972Z 2025-09-07T07:16:58.7731081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7731453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7731821Z return mod(**inputs) 2025-09-07T07:16:58.7732170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7732556Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7732933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7733314Z layer_outputs = layer_module( 2025-09-07T07:16:58.7733665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7734040Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7734420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7734805Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7735187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7735566Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7735965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.7736402Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.7736591Z 2025-09-07T07:16:58.7736707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7737101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7737435Z return mod(**inputs) 2025-09-07T07:16:58.7737797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7738189Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7738575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7738961Z layer_outputs = layer_module( 2025-09-07T07:16:58.7739342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7739745Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7740150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7740561Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7740961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7741372Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7741781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7742275Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7742508Z 2025-09-07T07:16:58.7742631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7743017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7743395Z return mod(**inputs) 2025-09-07T07:16:58.7743777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7744181Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7744571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7744972Z layer_outputs = layer_module( 2025-09-07T07:16:58.7745350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7745815Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7746237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7746692Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7747099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7747507Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7747884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7748339Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7748549Z 2025-09-07T07:16:58.7748654Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7749019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7749346Z return mod(**inputs) 2025-09-07T07:16:58.7749693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7750058Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7750427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7750816Z layer_outputs = layer_module( 2025-09-07T07:16:58.7751169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7751546Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7751912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7752285Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7752663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7753026Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7753377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7753813Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7754021Z 2025-09-07T07:16:58.7754123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7754473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7754789Z return mod(**inputs) 2025-09-07T07:16:58.7755117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7755471Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7756362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7756731Z layer_outputs = layer_module( 2025-09-07T07:16:58.7757077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7757430Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7757799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7758200Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7758573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7758950Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7759315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.7759683Z value_states = self.v(current_states) 2025-09-07T07:16:58.7759818Z 2025-09-07T07:16:58.7759927Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7760292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7760634Z return mod(**inputs) 2025-09-07T07:16:58.7760989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7761399Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7761766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7762135Z layer_outputs = layer_module( 2025-09-07T07:16:58.7762480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7762835Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7763207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7763582Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7763954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7764339Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7764714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7765110Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7765276Z 2025-09-07T07:16:58.7765380Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7765761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7766082Z return mod(**inputs) 2025-09-07T07:16:58.7766434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7766816Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7767197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7767584Z layer_outputs = layer_module( 2025-09-07T07:16:58.7767926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7768294Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7768662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7769033Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7769406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7769786Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7770147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7770546Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7770703Z 2025-09-07T07:16:58.7770814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7771174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7771498Z return mod(**inputs) 2025-09-07T07:16:58.7771860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7772228Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7772593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7772952Z layer_outputs = layer_module( 2025-09-07T07:16:58.7773290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7773641Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7774004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7774367Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7774750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7775135Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7775522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.7775934Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.7776102Z 2025-09-07T07:16:58.7776210Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7776581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7776920Z return mod(**inputs) 2025-09-07T07:16:58.7777275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7777658Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7778030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7778409Z layer_outputs = layer_module( 2025-09-07T07:16:58.7778788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7779152Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7779545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7779925Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7780299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7780688Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7781082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.7781458Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.7781601Z 2025-09-07T07:16:58.7781687Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.7781938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7782312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7782647Z return mod(**inputs) 2025-09-07T07:16:58.7782997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7783379Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7783753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7784130Z layer_outputs = layer_module( 2025-09-07T07:16:58.7784477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7784845Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7785228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7785646Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7786147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.7786586Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.7787021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.7787436Z return self.weight * hidden_states 2025-09-07T07:16:58.7787587Z 2025-09-07T07:16:58.7787715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7788098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7788451Z return mod(**inputs) 2025-09-07T07:16:58.7788828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7789207Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7789582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7789952Z layer_outputs = layer_module( 2025-09-07T07:16:58.7790306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7790679Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7791060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7791468Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7791854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7792280Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7792698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.7793099Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.7793238Z 2025-09-07T07:16:58.7793340Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7793725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7794063Z return mod(**inputs) 2025-09-07T07:16:58.7794417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7794798Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7795192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7795600Z layer_outputs = layer_module( 2025-09-07T07:16:58.7795972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7796340Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7796719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7797109Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7797503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7797931Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7798349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.7798726Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.7798875Z 2025-09-07T07:16:58.7798982Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7799358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7799697Z return mod(**inputs) 2025-09-07T07:16:58.7800054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7800446Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7800821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7801195Z layer_outputs = layer_module( 2025-09-07T07:16:58.7801547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7801905Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7802278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7802664Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7803074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7803490Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7803900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.7804280Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.7804423Z 2025-09-07T07:16:58.7804509Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.7804754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7805120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7805446Z return mod(**inputs) 2025-09-07T07:16:58.7805801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7806182Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7806559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7806950Z layer_outputs = layer_module( 2025-09-07T07:16:58.7807346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7807736Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7808153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7808552Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7808941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.7809383Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.7809817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.7810228Z return self.weight * hidden_states 2025-09-07T07:16:58.7810374Z 2025-09-07T07:16:58.7810494Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7810887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7811247Z return mod(**inputs) 2025-09-07T07:16:58.7811623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7812033Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7812434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7812845Z layer_outputs = layer_module( 2025-09-07T07:16:58.7813222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7813619Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7814024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7814457Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7814863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7815278Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7815687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.7816085Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.7816239Z 2025-09-07T07:16:58.7816352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7816745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7817110Z return mod(**inputs) 2025-09-07T07:16:58.7817486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7817910Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7818305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7818707Z layer_outputs = layer_module( 2025-09-07T07:16:58.7819091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7819490Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7820065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7820474Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7820879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7821282Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7821687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.7822092Z key_states = self.k(current_states) 2025-09-07T07:16:58.7822301Z 2025-09-07T07:16:58.7822416Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7822809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7823184Z return mod(**inputs) 2025-09-07T07:16:58.7823558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7823958Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7824358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7824769Z layer_outputs = layer_module( 2025-09-07T07:16:58.7825139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7825539Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7826008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7826431Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7826852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7827257Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7827649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.7828095Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.7828288Z 2025-09-07T07:16:58.7828406Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7828780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7829109Z return mod(**inputs) 2025-09-07T07:16:58.7829470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7829882Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7830269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7830643Z layer_outputs = layer_module( 2025-09-07T07:16:58.7831004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7831374Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7831756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7832137Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7832511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7832929Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7833318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7833792Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7834009Z 2025-09-07T07:16:58.7834123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7834486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7834825Z return mod(**inputs) 2025-09-07T07:16:58.7835185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7835570Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7835942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7836324Z layer_outputs = layer_module( 2025-09-07T07:16:58.7836711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7837088Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7837467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7837862Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7838242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7838631Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7838999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7839435Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7839649Z 2025-09-07T07:16:58.7839750Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7840109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7840440Z return mod(**inputs) 2025-09-07T07:16:58.7840797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7841173Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7841541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7841910Z layer_outputs = layer_module( 2025-09-07T07:16:58.7842257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7842617Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7842978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7843354Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7843727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7844130Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7844500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7844954Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7845168Z 2025-09-07T07:16:58.7845271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7845636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7845963Z return mod(**inputs) 2025-09-07T07:16:58.7846304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7846691Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7847061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7847429Z layer_outputs = layer_module( 2025-09-07T07:16:58.7847774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7848136Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7848515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7848891Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7849257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7849624Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7849997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.7850368Z value_states = self.v(current_states) 2025-09-07T07:16:58.7850505Z 2025-09-07T07:16:58.7850647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7851011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7851332Z return mod(**inputs) 2025-09-07T07:16:58.7851694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7852065Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7852430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7852792Z layer_outputs = layer_module( 2025-09-07T07:16:58.7853138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7853501Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7853875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7854254Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7854623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7855004Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7855379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7855781Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7855939Z 2025-09-07T07:16:58.7856048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7856400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7856727Z return mod(**inputs) 2025-09-07T07:16:58.7857071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7857504Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7857869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7858249Z layer_outputs = layer_module( 2025-09-07T07:16:58.7858607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7858982Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7859353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7859721Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7860099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7860509Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7860892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7861312Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7861477Z 2025-09-07T07:16:58.7861587Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7861959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7862296Z return mod(**inputs) 2025-09-07T07:16:58.7862651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7863025Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7863410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7863791Z layer_outputs = layer_module( 2025-09-07T07:16:58.7864155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7864555Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7864932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7865320Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7865791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7866226Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7866636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.7867089Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.7867276Z 2025-09-07T07:16:58.7867390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7867786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7868140Z return mod(**inputs) 2025-09-07T07:16:58.7868497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7868888Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7869269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7869652Z layer_outputs = layer_module( 2025-09-07T07:16:58.7870010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7870379Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7870762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7871147Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7871529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7871936Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7872327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.7872708Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.7872850Z 2025-09-07T07:16:58.7872971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7873360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7873708Z return mod(**inputs) 2025-09-07T07:16:58.7874094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7874478Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7874871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7875252Z layer_outputs = layer_module( 2025-09-07T07:16:58.7875602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7875974Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7876365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7876758Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7877149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.7877574Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.7877993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.7878383Z return self.weight * hidden_states 2025-09-07T07:16:58.7878521Z 2025-09-07T07:16:58.7878635Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7879017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7879359Z return mod(**inputs) 2025-09-07T07:16:58.7879723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7880126Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7880493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7880879Z layer_outputs = layer_module( 2025-09-07T07:16:58.7881224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7881588Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7881958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7882342Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7882733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7883150Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7883563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.7883938Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.7884072Z 2025-09-07T07:16:58.7884174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7884535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7884857Z return mod(**inputs) 2025-09-07T07:16:58.7885198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7885566Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7885938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7886354Z layer_outputs = layer_module( 2025-09-07T07:16:58.7886727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7887118Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7887507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7887901Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7888295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7888721Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7889162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.7889532Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.7889675Z 2025-09-07T07:16:58.7889777Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7890136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7890465Z return mod(**inputs) 2025-09-07T07:16:58.7890803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7891177Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7891543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7891916Z layer_outputs = layer_module( 2025-09-07T07:16:58.7892262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7892621Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7893018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7893410Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7893813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7894233Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7894643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.7895027Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.7895166Z 2025-09-07T07:16:58.7895259Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.7895511Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7895878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7896219Z return mod(**inputs) 2025-09-07T07:16:58.7896581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7896968Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7897348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7897726Z layer_outputs = layer_module( 2025-09-07T07:16:58.7898095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7898467Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7898854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7899231Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7899616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.7900060Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.7900472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.7900861Z return self.weight * hidden_states 2025-09-07T07:16:58.7900998Z 2025-09-07T07:16:58.7901107Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7901479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7901814Z return mod(**inputs) 2025-09-07T07:16:58.7902169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7902554Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7902922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7903321Z layer_outputs = layer_module( 2025-09-07T07:16:58.7903680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7904052Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7904425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7904806Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7905184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7905572Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7906046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.7906449Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.7906607Z 2025-09-07T07:16:58.7906722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7907139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7907502Z return mod(**inputs) 2025-09-07T07:16:58.7907862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7908263Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7908640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7909020Z layer_outputs = layer_module( 2025-09-07T07:16:58.7909373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7909735Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7910116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7910501Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7910888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7911274Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7911650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.7912036Z key_states = self.k(current_states) 2025-09-07T07:16:58.7912171Z 2025-09-07T07:16:58.7912296Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7912645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7912952Z return mod(**inputs) 2025-09-07T07:16:58.7913293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7913653Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7914010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7914391Z layer_outputs = layer_module( 2025-09-07T07:16:58.7914724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7915080Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7915442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7915810Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7916177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7916556Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7916930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.7917374Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.7917560Z 2025-09-07T07:16:58.7917675Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7918029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7918360Z return mod(**inputs) 2025-09-07T07:16:58.7918700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7919064Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7919422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7920036Z layer_outputs = layer_module( 2025-09-07T07:16:58.7920421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7920780Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7921195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7921557Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7921921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7922311Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7922680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7923116Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7923319Z 2025-09-07T07:16:58.7923422Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7923779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7924097Z return mod(**inputs) 2025-09-07T07:16:58.7924436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7924812Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7925163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7925523Z layer_outputs = layer_module( 2025-09-07T07:16:58.7925865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7926224Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7926589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7926965Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7927341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7927790Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7928154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7928612Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7928821Z 2025-09-07T07:16:58.7928922Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7929278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7929598Z return mod(**inputs) 2025-09-07T07:16:58.7929941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7930294Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7930652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7931037Z layer_outputs = layer_module( 2025-09-07T07:16:58.7931384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7931745Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7932125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7932492Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7932859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7933227Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7933589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.7934020Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.7934226Z 2025-09-07T07:16:58.7934329Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7934679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7935016Z return mod(**inputs) 2025-09-07T07:16:58.7935350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7935711Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7936088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7936457Z layer_outputs = layer_module( 2025-09-07T07:16:58.7936802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7937168Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7937536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7937919Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7938282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7938652Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7939021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.7939402Z value_states = self.v(current_states) 2025-09-07T07:16:58.7939544Z 2025-09-07T07:16:58.7939674Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7940078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7940404Z return mod(**inputs) 2025-09-07T07:16:58.7940750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7941166Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7941563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7942009Z layer_outputs = layer_module( 2025-09-07T07:16:58.7942405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7942817Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7943235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7943653Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7944061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7944559Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7944963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7945429Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7945606Z 2025-09-07T07:16:58.7945786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7946196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7946572Z return mod(**inputs) 2025-09-07T07:16:58.7946966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7947338Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7947700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7948074Z layer_outputs = layer_module( 2025-09-07T07:16:58.7948425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7948789Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7949159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7949563Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7949938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7950316Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7950707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.7951102Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.7951272Z 2025-09-07T07:16:58.7951376Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7951733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7952059Z return mod(**inputs) 2025-09-07T07:16:58.7952409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7952776Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7953146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7953517Z layer_outputs = layer_module( 2025-09-07T07:16:58.7953864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7954226Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7954598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7954970Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7955341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7955719Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7956085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.7956514Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.7956682Z 2025-09-07T07:16:58.7956787Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7957154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7957493Z return mod(**inputs) 2025-09-07T07:16:58.7957831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7958196Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7958558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7958923Z layer_outputs = layer_module( 2025-09-07T07:16:58.7959290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7959661Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7960041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7960432Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7960801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7961166Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7961680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.7962056Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.7962189Z 2025-09-07T07:16:58.7962281Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.7962518Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7962891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7963231Z return mod(**inputs) 2025-09-07T07:16:58.7963613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7963989Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7964365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7964744Z layer_outputs = layer_module( 2025-09-07T07:16:58.7965099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7965454Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7965831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7966212Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7966600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.7966996Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.7967391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.7967766Z return self.weight * hidden_states 2025-09-07T07:16:58.7967910Z 2025-09-07T07:16:58.7968016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7968389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7968727Z return mod(**inputs) 2025-09-07T07:16:58.7969081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7969471Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7969869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7970272Z layer_outputs = layer_module( 2025-09-07T07:16:58.7970674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7971066Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7971472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7971895Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7972316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7972765Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7973202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.7973637Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.7973789Z 2025-09-07T07:16:58.7973902Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7974295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7974648Z return mod(**inputs) 2025-09-07T07:16:58.7975017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7975419Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7975814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7976215Z layer_outputs = layer_module( 2025-09-07T07:16:58.7976590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7976989Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7977399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7977819Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7978260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7978700Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7979160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.7979566Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.7979714Z 2025-09-07T07:16:58.7979844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7980213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7980588Z return mod(**inputs) 2025-09-07T07:16:58.7980963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7981376Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7981775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7982169Z layer_outputs = layer_module( 2025-09-07T07:16:58.7982547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7982917Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7983297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.7983688Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.7984072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.7984490Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.7984921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.7985365Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.7985503Z 2025-09-07T07:16:58.7985593Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.7985895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7986288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7986655Z return mod(**inputs) 2025-09-07T07:16:58.7987030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7987424Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7987806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7988185Z layer_outputs = layer_module( 2025-09-07T07:16:58.7988566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7988945Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7989322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7989706Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7990091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.7990498Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.7990892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.7991281Z return self.weight * hidden_states 2025-09-07T07:16:58.7991422Z 2025-09-07T07:16:58.7991530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7991904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7992241Z return mod(**inputs) 2025-09-07T07:16:58.7992612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7993002Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7993412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7993790Z layer_outputs = layer_module( 2025-09-07T07:16:58.7994140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7994512Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.7994894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.7995285Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.7995657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.7996031Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.7996406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.7996780Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.7996915Z 2025-09-07T07:16:58.7997030Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.7997393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.7997711Z return mod(**inputs) 2025-09-07T07:16:58.7998059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.7998434Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.7998800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.7999164Z layer_outputs = layer_module( 2025-09-07T07:16:58.7999540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.7999902Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8000273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8000648Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8001012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8001389Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8001758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8002130Z key_states = self.k(current_states) 2025-09-07T07:16:58.8002283Z 2025-09-07T07:16:58.8002395Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8002747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8003075Z return mod(**inputs) 2025-09-07T07:16:58.8003421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8003791Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8004147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8004516Z layer_outputs = layer_module( 2025-09-07T07:16:58.8004873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8005233Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8005601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8005973Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8006364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8006745Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8007133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8007554Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8007738Z 2025-09-07T07:16:58.8007840Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8008186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8008502Z return mod(**inputs) 2025-09-07T07:16:58.8008838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8009195Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8009558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8009921Z layer_outputs = layer_module( 2025-09-07T07:16:58.8010261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8010614Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8010972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8011337Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8011706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8012076Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8012438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8012883Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8013114Z 2025-09-07T07:16:58.8013216Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8013566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8013884Z return mod(**inputs) 2025-09-07T07:16:58.8014214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8014585Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8014936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8015295Z layer_outputs = layer_module( 2025-09-07T07:16:58.8015632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8016003Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8016381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8016761Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8017145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8017532Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8017920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8018369Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8018575Z 2025-09-07T07:16:58.8018688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8019060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8019377Z return mod(**inputs) 2025-09-07T07:16:58.8019897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8020270Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8020629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8021019Z layer_outputs = layer_module( 2025-09-07T07:16:58.8021362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8021725Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8022096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8022474Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8022837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8023216Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8023589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8024046Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8024256Z 2025-09-07T07:16:58.8024371Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8024731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8025072Z return mod(**inputs) 2025-09-07T07:16:58.8025414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8025831Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8026214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8026615Z layer_outputs = layer_module( 2025-09-07T07:16:58.8026991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8027431Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8027848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8028250Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8028634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8029018Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8029400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8029787Z value_states = self.v(current_states) 2025-09-07T07:16:58.8029926Z 2025-09-07T07:16:58.8030062Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8030426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8030763Z return mod(**inputs) 2025-09-07T07:16:58.8031112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8031480Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8031849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8032220Z layer_outputs = layer_module( 2025-09-07T07:16:58.8032570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8032935Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8033301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8033680Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8034089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8034472Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8034843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8034971Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8034975Z 2025-09-07T07:16:58.8035079Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8035286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8035352Z return mod(**inputs) 2025-09-07T07:16:58.8035595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8035669Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8035904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8035987Z layer_outputs = layer_module( 2025-09-07T07:16:58.8036210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8036296Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8036537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8036618Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8036858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8036938Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8037178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8037288Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8037291Z 2025-09-07T07:16:58.8037422Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8037624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8037690Z return mod(**inputs) 2025-09-07T07:16:58.8037936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8038008Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8038250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8038320Z layer_outputs = layer_module( 2025-09-07T07:16:58.8038539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8038669Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8038910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8038998Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8039233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8039312Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8039556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8039664Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8039667Z 2025-09-07T07:16:58.8039776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8039980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8040053Z return mod(**inputs) 2025-09-07T07:16:58.8040300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8040369Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8040640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8040711Z layer_outputs = layer_module( 2025-09-07T07:16:58.8040948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8041027Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8041255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8041339Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8041568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8041654Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8041878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8041964Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8041967Z 2025-09-07T07:16:58.8042067Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8042261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8042334Z return mod(**inputs) 2025-09-07T07:16:58.8042561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8042637Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8042863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8042930Z layer_outputs = layer_module( 2025-09-07T07:16:58.8043153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8043229Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8043478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8043556Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8043784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-09-07T07:16:58.8043920Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:16:58.8043924Z 2025-09-07T07:16:58.8044003Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8044108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8044301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8044372Z return mod(**inputs) 2025-09-07T07:16:58.8044622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8044693Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8044932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8045001Z layer_outputs = layer_module( 2025-09-07T07:16:58.8045222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8045298Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8045524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8045622Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8045849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8045953Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8046181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8046277Z return self.weight * hidden_states 2025-09-07T07:16:58.8046292Z 2025-09-07T07:16:58.8046392Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8046601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8046674Z return mod(**inputs) 2025-09-07T07:16:58.8046901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8046977Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8047203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8047273Z layer_outputs = layer_module( 2025-09-07T07:16:58.8047495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8047569Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8047809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8047900Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8048133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8048268Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8048495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8048578Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8048581Z 2025-09-07T07:16:58.8048678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8048880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8048945Z return mod(**inputs) 2025-09-07T07:16:58.8049199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8049278Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8049521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8049598Z layer_outputs = layer_module( 2025-09-07T07:16:58.8049811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8049889Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8050122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8050208Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8050457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8050575Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8050813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8050903Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8050909Z 2025-09-07T07:16:58.8051012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8051235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8051302Z return mod(**inputs) 2025-09-07T07:16:58.8051543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8051616Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8051851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8051931Z layer_outputs = layer_module( 2025-09-07T07:16:58.8052182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8052266Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8052549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8052639Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8052879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8052993Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8053232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8053313Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8053316Z 2025-09-07T07:16:58.8053397Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8053506Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8053714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8053782Z return mod(**inputs) 2025-09-07T07:16:58.8054010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8054086Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8054313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8054380Z layer_outputs = layer_module( 2025-09-07T07:16:58.8054598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8054674Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8054906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8055004Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8055226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.8055338Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8055563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8055648Z return self.weight * hidden_states 2025-09-07T07:16:58.8055651Z 2025-09-07T07:16:58.8055749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8055939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8056009Z return mod(**inputs) 2025-09-07T07:16:58.8056254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8056332Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8056565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8056643Z layer_outputs = layer_module( 2025-09-07T07:16:58.8056864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8056941Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8057179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8057258Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8057498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8057580Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8057823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8057927Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8057931Z 2025-09-07T07:16:58.8058031Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8058230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8058308Z return mod(**inputs) 2025-09-07T07:16:58.8058539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8058616Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8058844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8058923Z layer_outputs = layer_module( 2025-09-07T07:16:58.8059139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8059224Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8059454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8059532Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8059768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8059849Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8060087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8060163Z key_states = self.k(current_states) 2025-09-07T07:16:58.8060166Z 2025-09-07T07:16:58.8060265Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8060465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8060529Z return mod(**inputs) 2025-09-07T07:16:58.8060767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8060855Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8061091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8061161Z layer_outputs = layer_module( 2025-09-07T07:16:58.8061382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8061467Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8061700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8061787Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8062019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8062137Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8062374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8062499Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8062503Z 2025-09-07T07:16:58.8062609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8062801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8062865Z return mod(**inputs) 2025-09-07T07:16:58.8063097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8063165Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8063398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8063469Z layer_outputs = layer_module( 2025-09-07T07:16:58.8063703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8063783Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8064006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8064108Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8064335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8064422Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8064653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8064807Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8064812Z 2025-09-07T07:16:58.8064921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8065119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8065194Z return mod(**inputs) 2025-09-07T07:16:58.8065429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8065502Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8065805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8065885Z layer_outputs = layer_module( 2025-09-07T07:16:58.8066114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8066196Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8066447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8066530Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8066783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8066938Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8067173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8067331Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8067335Z 2025-09-07T07:16:58.8067435Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8067632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8067706Z return mod(**inputs) 2025-09-07T07:16:58.8067939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8068036Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8068277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8068358Z layer_outputs = layer_module( 2025-09-07T07:16:58.8068583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8068663Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8068913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8068991Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8069228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8069307Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8069537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8069701Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8069726Z 2025-09-07T07:16:58.8069829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8070031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8070112Z return mod(**inputs) 2025-09-07T07:16:58.8070350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8070428Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8070665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8070744Z layer_outputs = layer_module( 2025-09-07T07:16:58.8070965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8071050Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8071287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8071366Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8071611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8071693Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8071931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8072009Z value_states = self.v(current_states) 2025-09-07T07:16:58.8072013Z 2025-09-07T07:16:58.8072113Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8072317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8072384Z return mod(**inputs) 2025-09-07T07:16:58.8072625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8072716Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8072956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8073025Z layer_outputs = layer_module( 2025-09-07T07:16:58.8073247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8073333Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8073565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8073651Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8073884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8073982Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8074223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8074335Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8074338Z 2025-09-07T07:16:58.8074446Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8074643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8074708Z return mod(**inputs) 2025-09-07T07:16:58.8074948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8075019Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8075258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8075329Z layer_outputs = layer_module( 2025-09-07T07:16:58.8075554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8075651Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8075888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8075974Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8076227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8076315Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8076547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8076655Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8076658Z 2025-09-07T07:16:58.8076768Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8076966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8077039Z return mod(**inputs) 2025-09-07T07:16:58.8077281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8077355Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8077605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8077676Z layer_outputs = layer_module( 2025-09-07T07:16:58.8077910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8077988Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8078232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8078312Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8078549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8078659Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8078895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8079013Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8079016Z 2025-09-07T07:16:58.8079120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8079321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8079394Z return mod(**inputs) 2025-09-07T07:16:58.8079632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8079714Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8079970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8080052Z layer_outputs = layer_module( 2025-09-07T07:16:58.8080283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8080361Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8080601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8080679Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8080914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8080992Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8081224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8081309Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8081314Z 2025-09-07T07:16:58.8081394Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8081522Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8081723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8081789Z return mod(**inputs) 2025-09-07T07:16:58.8082052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8082130Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8082379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8082451Z layer_outputs = layer_module( 2025-09-07T07:16:58.8082677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8082766Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8083003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8083107Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8083353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8083462Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8083701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8083781Z return self.weight * hidden_states 2025-09-07T07:16:58.8083784Z 2025-09-07T07:16:58.8083897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8084101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8084176Z return mod(**inputs) 2025-09-07T07:16:58.8084419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8084494Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8084771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8084844Z layer_outputs = layer_module( 2025-09-07T07:16:58.8085080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8085161Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8085410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8085504Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8085782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8085975Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8086331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8086448Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8086453Z 2025-09-07T07:16:58.8086596Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8086882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8086960Z return mod(**inputs) 2025-09-07T07:16:58.8087202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8087286Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8087537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8087615Z layer_outputs = layer_module( 2025-09-07T07:16:58.8087861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8087947Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8088232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8088331Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8088607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8088734Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8088992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8089091Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8089095Z 2025-09-07T07:16:58.8089209Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8089445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8089530Z return mod(**inputs) 2025-09-07T07:16:58.8089792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8089876Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8090133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8090217Z layer_outputs = layer_module( 2025-09-07T07:16:58.8090463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8090560Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8090827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8090926Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8091197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8093992Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8094280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8094378Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8094384Z 2025-09-07T07:16:58.8094478Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8094603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8094854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8094926Z return mod(**inputs) 2025-09-07T07:16:58.8095207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8095315Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8095590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8095709Z layer_outputs = layer_module( 2025-09-07T07:16:58.8095957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8096051Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8096311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8096407Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8096665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.8096783Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8097049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8097138Z return self.weight * hidden_states 2025-09-07T07:16:58.8097142Z 2025-09-07T07:16:58.8097282Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8097507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8097587Z return mod(**inputs) 2025-09-07T07:16:58.8097867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8097948Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8098221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8098300Z layer_outputs = layer_module( 2025-09-07T07:16:58.8098563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8098648Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8098916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8099014Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8099277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8099376Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8099641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8099726Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8099736Z 2025-09-07T07:16:58.8099850Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8100071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8100150Z return mod(**inputs) 2025-09-07T07:16:58.8100428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8100513Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8100858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8100937Z layer_outputs = layer_module( 2025-09-07T07:16:58.8101193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8101277Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8101568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8101681Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8102053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8102179Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8103754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8103853Z key_states = self.k(current_states) 2025-09-07T07:16:58.8103857Z 2025-09-07T07:16:58.8103971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8104201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8104277Z return mod(**inputs) 2025-09-07T07:16:58.8104539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8104627Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8104888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8104975Z layer_outputs = layer_module( 2025-09-07T07:16:58.8105220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8105309Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8105597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8105689Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8106111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8106210Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8106472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8106627Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8106631Z 2025-09-07T07:16:58.8106746Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8106975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8107050Z return mod(**inputs) 2025-09-07T07:16:58.8107330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8107412Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8107670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8107758Z layer_outputs = layer_module( 2025-09-07T07:16:58.8107997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8108090Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8108356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8108442Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8108704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8108791Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8109095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8109263Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8109267Z 2025-09-07T07:16:58.8109386Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8109600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8109669Z return mod(**inputs) 2025-09-07T07:16:58.8109931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8110009Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8110271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8110368Z layer_outputs = layer_module( 2025-09-07T07:16:58.8110605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8110698Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8110968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8111058Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8111311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8111395Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8111654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8111818Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8111824Z 2025-09-07T07:16:58.8111944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8112186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8112267Z return mod(**inputs) 2025-09-07T07:16:58.8112522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8112616Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8112877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8112953Z layer_outputs = layer_module( 2025-09-07T07:16:58.8113198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8113283Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8113533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8113625Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8113876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8113970Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8114220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8114384Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8114394Z 2025-09-07T07:16:58.8114505Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8114719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8114797Z return mod(**inputs) 2025-09-07T07:16:58.8115051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8115137Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8115392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8115491Z layer_outputs = layer_module( 2025-09-07T07:16:58.8115736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8115821Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8116080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8116163Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8116411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8116505Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8116773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8116861Z value_states = self.v(current_states) 2025-09-07T07:16:58.8116868Z 2025-09-07T07:16:58.8116977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8117198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8117267Z return mod(**inputs) 2025-09-07T07:16:58.8117520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8117608Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8117861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8117942Z layer_outputs = layer_module( 2025-09-07T07:16:58.8118179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8118265Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8118541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8118627Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8118891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8118994Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8119246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8119371Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8119375Z 2025-09-07T07:16:58.8119483Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8119938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8120051Z return mod(**inputs) 2025-09-07T07:16:58.8120328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8120413Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8120667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8120753Z layer_outputs = layer_module( 2025-09-07T07:16:58.8120993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8121084Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8121336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8121420Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8121681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8121769Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8122030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8122208Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8122212Z 2025-09-07T07:16:58.8122323Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8122548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8122622Z return mod(**inputs) 2025-09-07T07:16:58.8122882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8122961Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8123224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8123332Z layer_outputs = layer_module( 2025-09-07T07:16:58.8123556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8123651Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8123902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8123998Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8124257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8124347Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8124614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8124734Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8124738Z 2025-09-07T07:16:58.8124861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8125081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8125186Z return mod(**inputs) 2025-09-07T07:16:58.8125454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8125533Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8125828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8125900Z layer_outputs = layer_module( 2025-09-07T07:16:58.8126132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8126212Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8126457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8126550Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8126791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8126883Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8127122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8127203Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8127206Z 2025-09-07T07:16:58.8127317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8127518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8127592Z return mod(**inputs) 2025-09-07T07:16:58.8127832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8127912Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8128152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8128223Z layer_outputs = layer_module( 2025-09-07T07:16:58.8128482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8128561Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8128808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8128886Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8129127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-09-07T07:16:58.8129272Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:16:58.8129276Z 2025-09-07T07:16:58.8129359Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8129489Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8129691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8129762Z return mod(**inputs) 2025-09-07T07:16:58.8130011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8130083Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8130333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8130405Z layer_outputs = layer_module( 2025-09-07T07:16:58.8130638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8130717Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8130956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8131058Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8131315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8131424Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8131661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8131756Z return self.weight * hidden_states 2025-09-07T07:16:58.8131760Z 2025-09-07T07:16:58.8131872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8132077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8132150Z return mod(**inputs) 2025-09-07T07:16:58.8132390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8132463Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8132716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8132784Z layer_outputs = layer_module( 2025-09-07T07:16:58.8133005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8133080Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8133314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8133402Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8133629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8133750Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8133986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8134072Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8134075Z 2025-09-07T07:16:58.8134175Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8134398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8134470Z return mod(**inputs) 2025-09-07T07:16:58.8134708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8134785Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8135017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8135088Z layer_outputs = layer_module( 2025-09-07T07:16:58.8135315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8135391Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8135661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8135750Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8135988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8136103Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8136343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8136429Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8136432Z 2025-09-07T07:16:58.8136534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8136738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8136803Z return mod(**inputs) 2025-09-07T07:16:58.8137039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8137118Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8137375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8137455Z layer_outputs = layer_module( 2025-09-07T07:16:58.8137695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8137784Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8138015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8138104Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8138344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8138459Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8138699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8138780Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8138784Z 2025-09-07T07:16:58.8138866Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8138977Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8139178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8139252Z return mod(**inputs) 2025-09-07T07:16:58.8139485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-09-07T07:16:58.8139556Z encoder_outputs = self.encoder( 2025-09-07T07:16:58.8139794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward 2025-09-07T07:16:58.8139902Z hidden_states = self.final_layer_norm(hidden_states) 2025-09-07T07:16:58.8140142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8140242Z return self.weight * hidden_states 2025-09-07T07:16:58.8140245Z 2025-09-07T07:16:58.8140352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8140562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8140624Z return mod(**inputs) 2025-09-07T07:16:58.8140858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8140928Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8141168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8141239Z layer_outputs = layer_module( 2025-09-07T07:16:58.8141477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8141561Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8141796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8141884Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8142115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8142200Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8142437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8142511Z key_states = self.k(current_states) 2025-09-07T07:16:58.8142515Z 2025-09-07T07:16:58.8142622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8142817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8142891Z return mod(**inputs) 2025-09-07T07:16:58.8143140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8143214Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8143471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8143544Z layer_outputs = layer_module( 2025-09-07T07:16:58.8143778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8143855Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8144093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8144181Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8144418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8144512Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8144752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8144890Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8144893Z 2025-09-07T07:16:58.8144999Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8145201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8145275Z return mod(**inputs) 2025-09-07T07:16:58.8145514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8145594Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8145898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8145979Z layer_outputs = layer_module( 2025-09-07T07:16:58.8146222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8146325Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8146570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8146651Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8146887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8146975Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8147198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8147353Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8147376Z 2025-09-07T07:16:58.8147478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8147693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8147762Z return mod(**inputs) 2025-09-07T07:16:58.8148010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8148104Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8148341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8148416Z layer_outputs = layer_module( 2025-09-07T07:16:58.8148643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8148718Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8148962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8149040Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8149303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8149397Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8149637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8149720Z value_states = self.v(current_states) 2025-09-07T07:16:58.8149724Z 2025-09-07T07:16:58.8149821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8150027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8150091Z return mod(**inputs) 2025-09-07T07:16:58.8150330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8150403Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8150638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8150720Z layer_outputs = layer_module( 2025-09-07T07:16:58.8150936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8151023Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8151255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8151333Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8151568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8151652Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8151891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8152000Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8152028Z 2025-09-07T07:16:58.8152140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8152339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8152404Z return mod(**inputs) 2025-09-07T07:16:58.8152647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8152719Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8152958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8153030Z layer_outputs = layer_module( 2025-09-07T07:16:58.8153248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8153382Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8153613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8153700Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8153931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8154015Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8154260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8154368Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8154372Z 2025-09-07T07:16:58.8154479Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8154675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8154750Z return mod(**inputs) 2025-09-07T07:16:58.8154983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8155072Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8155318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8155390Z layer_outputs = layer_module( 2025-09-07T07:16:58.8155631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8155708Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8155941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8156029Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8156261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8156350Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8156585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8156692Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8156702Z 2025-09-07T07:16:58.8156803Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8157006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8157078Z return mod(**inputs) 2025-09-07T07:16:58.8157313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8157391Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8157632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8157705Z layer_outputs = layer_module( 2025-09-07T07:16:58.8157930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8158034Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8158275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8158353Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8158585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8158673Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8158903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8158988Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8158992Z 2025-09-07T07:16:58.8159071Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8159201Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8159407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8159473Z return mod(**inputs) 2025-09-07T07:16:58.8159714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8159788Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8160028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8160097Z layer_outputs = layer_module( 2025-09-07T07:16:58.8160316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8160399Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8160634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8160733Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8160982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8161081Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8161343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8161423Z return self.weight * hidden_states 2025-09-07T07:16:58.8161427Z 2025-09-07T07:16:58.8161532Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8161736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8161808Z return mod(**inputs) 2025-09-07T07:16:58.8162043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8162116Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8162359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8162432Z layer_outputs = layer_module( 2025-09-07T07:16:58.8162657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8162731Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8162970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8163068Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8163300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8163421Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8163656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8163736Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8163747Z 2025-09-07T07:16:58.8163870Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8164066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8164139Z return mod(**inputs) 2025-09-07T07:16:58.8164375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8164454Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8164700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8164768Z layer_outputs = layer_module( 2025-09-07T07:16:58.8164991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8165083Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8165325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8165415Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8165661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8165786Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8166026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8166112Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8166115Z 2025-09-07T07:16:58.8166217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8166425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8166493Z return mod(**inputs) 2025-09-07T07:16:58.8166734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8166831Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8167068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8167145Z layer_outputs = layer_module( 2025-09-07T07:16:58.8167382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8167459Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8167696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8167785Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8168022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8168137Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8168366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8168454Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8168457Z 2025-09-07T07:16:58.8168555Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8168759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8168823Z return mod(**inputs) 2025-09-07T07:16:58.8169064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8169136Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8169375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8169454Z layer_outputs = layer_module( 2025-09-07T07:16:58.8169664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8169766Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8169991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8170068Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8170304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.8170408Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8170642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8170717Z return self.weight * hidden_states 2025-09-07T07:16:58.8170721Z 2025-09-07T07:16:58.8170825Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8171044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8171110Z return mod(**inputs) 2025-09-07T07:16:58.8171348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8171419Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8171657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8171726Z layer_outputs = layer_module( 2025-09-07T07:16:58.8171939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8172022Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8172250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8172337Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8172568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8172668Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8172911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8172989Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8173011Z 2025-09-07T07:16:58.8173121Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8173321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8173385Z return mod(**inputs) 2025-09-07T07:16:58.8173629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8173702Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8173947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8174017Z layer_outputs = layer_module( 2025-09-07T07:16:58.8174254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8174333Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8174574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8174661Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8174899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8174989Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8175228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8175307Z key_states = self.k(current_states) 2025-09-07T07:16:58.8175311Z 2025-09-07T07:16:58.8175422Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8175628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8175725Z return mod(**inputs) 2025-09-07T07:16:58.8175967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8176048Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8176291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8176361Z layer_outputs = layer_module( 2025-09-07T07:16:58.8176592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8176670Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8176918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8177016Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8177260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8177349Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8177579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8177715Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8177719Z 2025-09-07T07:16:58.8177820Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8178017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8178089Z return mod(**inputs) 2025-09-07T07:16:58.8178325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8178405Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8178653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8178733Z layer_outputs = layer_module( 2025-09-07T07:16:58.8178955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8179057Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8179302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8179383Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8179627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8179708Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8179945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8180111Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8180116Z 2025-09-07T07:16:58.8180221Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8180431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8180499Z return mod(**inputs) 2025-09-07T07:16:58.8180738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8180820Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8181057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8181135Z layer_outputs = layer_module( 2025-09-07T07:16:58.8181358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8181446Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8181684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8181781Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8182029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8182111Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8182354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8182432Z value_states = self.v(current_states) 2025-09-07T07:16:58.8182436Z 2025-09-07T07:16:58.8182541Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8182750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8182835Z return mod(**inputs) 2025-09-07T07:16:58.8183086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8183162Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8183416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8183491Z layer_outputs = layer_module( 2025-09-07T07:16:58.8183718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8183805Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8184044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8184130Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8184369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8184452Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8184719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8184835Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8184839Z 2025-09-07T07:16:58.8184947Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8185163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8185232Z return mod(**inputs) 2025-09-07T07:16:58.8185491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8185569Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8185922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8186009Z layer_outputs = layer_module( 2025-09-07T07:16:58.8186260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8186347Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8186605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8186703Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8186968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8187066Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8187330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8187442Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8187446Z 2025-09-07T07:16:58.8187561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8187763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8187860Z return mod(**inputs) 2025-09-07T07:16:58.8188100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8188177Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8188424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8188495Z layer_outputs = layer_module( 2025-09-07T07:16:58.8188726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8188804Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8189047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8189146Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8189385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8189475Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8189711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8189828Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8189831Z 2025-09-07T07:16:58.8189933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8190133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8190206Z return mod(**inputs) 2025-09-07T07:16:58.8190447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8190525Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8190766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8190862Z layer_outputs = layer_module( 2025-09-07T07:16:58.8191088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8191169Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8191428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8191509Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8191756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8191838Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8192075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8192165Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8192168Z 2025-09-07T07:16:58.8192251Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8192366Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8192569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8192635Z return mod(**inputs) 2025-09-07T07:16:58.8192885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8192962Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8193209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8193281Z layer_outputs = layer_module( 2025-09-07T07:16:58.8193504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8193592Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8193828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8193939Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8194175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-09-07T07:16:58.8194291Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8194530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8194610Z return self.weight * hidden_states 2025-09-07T07:16:58.8194614Z 2025-09-07T07:16:58.8194725Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8194929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8195004Z return mod(**inputs) 2025-09-07T07:16:58.8195266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8195342Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8195597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8195670Z layer_outputs = layer_module( 2025-09-07T07:16:58.8195908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8195987Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8196240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8196332Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8196566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8196661Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8196911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8196998Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8197002Z 2025-09-07T07:16:58.8197102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8197318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8197391Z return mod(**inputs) 2025-09-07T07:16:58.8197634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8197714Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8197952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8198024Z layer_outputs = layer_module( 2025-09-07T07:16:58.8198256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8198333Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8198585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8198667Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8198913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8198999Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8199237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8199322Z key_states = self.k(current_states) 2025-09-07T07:16:58.8199326Z 2025-09-07T07:16:58.8199430Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8199646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8199713Z return mod(**inputs) 2025-09-07T07:16:58.8199950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8200080Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8200316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8200393Z layer_outputs = layer_module( 2025-09-07T07:16:58.8200612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8200689Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8200927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8201006Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8201261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8201343Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8201585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8201716Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8201720Z 2025-09-07T07:16:58.8201823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8202028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8202093Z return mod(**inputs) 2025-09-07T07:16:58.8202336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8202408Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8202640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8202725Z layer_outputs = layer_module( 2025-09-07T07:16:58.8202972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8203062Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8203312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8203394Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8203633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8203714Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8203950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8204100Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8204105Z 2025-09-07T07:16:58.8204213Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8204415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8204482Z return mod(**inputs) 2025-09-07T07:16:58.8204724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8204795Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8205033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8205104Z layer_outputs = layer_module( 2025-09-07T07:16:58.8205321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8205403Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8205633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8205716Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8205973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8206062Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8206300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8206379Z value_states = self.v(current_states) 2025-09-07T07:16:58.8206383Z 2025-09-07T07:16:58.8206503Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8206698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8206771Z return mod(**inputs) 2025-09-07T07:16:58.8207006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8207098Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8207342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8207413Z layer_outputs = layer_module( 2025-09-07T07:16:58.8207642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8207720Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8207954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8208039Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8208270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8208358Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8208592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8208707Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8208728Z 2025-09-07T07:16:58.8208833Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8209030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8209120Z return mod(**inputs) 2025-09-07T07:16:58.8209357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8209437Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8209671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8209741Z layer_outputs = layer_module( 2025-09-07T07:16:58.8209970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8210047Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8210286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8210365Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8210604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8210699Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8210932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8211046Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8211049Z 2025-09-07T07:16:58.8211151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8211355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8211422Z return mod(**inputs) 2025-09-07T07:16:58.8211661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8211760Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8211996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8212075Z layer_outputs = layer_module( 2025-09-07T07:16:58.8212296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8212372Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8212612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8212692Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8212935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8213033Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8213273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8213379Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8213383Z 2025-09-07T07:16:58.8213485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8213694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8213759Z return mod(**inputs) 2025-09-07T07:16:58.8214000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8214071Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8214314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8214395Z layer_outputs = layer_module( 2025-09-07T07:16:58.8214622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8214737Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8214970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8215065Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8215307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8215387Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8215623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8215700Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8215703Z 2025-09-07T07:16:58.8215790Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8215893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8216099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8216177Z return mod(**inputs) 2025-09-07T07:16:58.8216416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8216498Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8216738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8216810Z layer_outputs = layer_module( 2025-09-07T07:16:58.8217042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8217120Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8217363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8217457Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8217696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8217834Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8218071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8218156Z return self.weight * hidden_states 2025-09-07T07:16:58.8218159Z 2025-09-07T07:16:58.8218260Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8218469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8218537Z return mod(**inputs) 2025-09-07T07:16:58.8218778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8218880Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8219123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8219204Z layer_outputs = layer_module( 2025-09-07T07:16:58.8219427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8219507Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8220089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8220219Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8220482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8220611Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8220871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8220962Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8220967Z 2025-09-07T07:16:58.8221127Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8221353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8221425Z return mod(**inputs) 2025-09-07T07:16:58.8221712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8221792Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8222050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8222135Z layer_outputs = layer_module( 2025-09-07T07:16:58.8222374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8222467Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8222721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8222819Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8223083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8223214Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8223478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8223568Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8223572Z 2025-09-07T07:16:58.8223693Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8223909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8223981Z return mod(**inputs) 2025-09-07T07:16:58.8224246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8224357Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8224620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8224699Z layer_outputs = layer_module( 2025-09-07T07:16:58.8224937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8225029Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8225280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8225382Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8225632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8225841Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8226109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8226198Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8226202Z 2025-09-07T07:16:58.8226298Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8226409Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8226629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8226699Z return mod(**inputs) 2025-09-07T07:16:58.8226952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8227036Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8227286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8227367Z layer_outputs = layer_module( 2025-09-07T07:16:58.8227612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8227696Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8227946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8228044Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8228294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.8228403Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8228645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8228732Z return self.weight * hidden_states 2025-09-07T07:16:58.8228737Z 2025-09-07T07:16:58.8228842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8229054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8229124Z return mod(**inputs) 2025-09-07T07:16:58.8229374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8229447Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8229692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8229771Z layer_outputs = layer_module( 2025-09-07T07:16:58.8229996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8230081Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8230323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8230407Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8230659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8230763Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8231007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8231089Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8231092Z 2025-09-07T07:16:58.8231202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8231402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8231468Z return mod(**inputs) 2025-09-07T07:16:58.8231714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8231807Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8232054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8232128Z layer_outputs = layer_module( 2025-09-07T07:16:58.8232352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8232435Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8232673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8232760Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8233004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8233083Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8233317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8233393Z key_states = self.k(current_states) 2025-09-07T07:16:58.8233396Z 2025-09-07T07:16:58.8233517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8233715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8233783Z return mod(**inputs) 2025-09-07T07:16:58.8234034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8234106Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8234340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8234409Z layer_outputs = layer_module( 2025-09-07T07:16:58.8234628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8234703Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8234932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8235015Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8235241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8235326Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8235553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8235679Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8235690Z 2025-09-07T07:16:58.8235789Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8235979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8236050Z return mod(**inputs) 2025-09-07T07:16:58.8236282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8236358Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8236608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8236676Z layer_outputs = layer_module( 2025-09-07T07:16:58.8236897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8236974Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8237211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8237286Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8237509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8237612Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8237838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8238001Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8238004Z 2025-09-07T07:16:58.8238103Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8238305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8238369Z return mod(**inputs) 2025-09-07T07:16:58.8238598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8238676Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8238906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8238981Z layer_outputs = layer_module( 2025-09-07T07:16:58.8239195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8239270Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8239524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8239602Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8239877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8239957Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8240185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8240267Z value_states = self.v(current_states) 2025-09-07T07:16:58.8240270Z 2025-09-07T07:16:58.8240369Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8240571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8240635Z return mod(**inputs) 2025-09-07T07:16:58.8240874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8240947Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8241178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8241257Z layer_outputs = layer_module( 2025-09-07T07:16:58.8241473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8241555Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8241788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8241865Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8242104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8242183Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8242437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8242545Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8242548Z 2025-09-07T07:16:58.8242655Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8242849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8242914Z return mod(**inputs) 2025-09-07T07:16:58.8243155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8243224Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8243460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8243547Z layer_outputs = layer_module( 2025-09-07T07:16:58.8243774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8243861Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8244105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8244188Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8244429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8244506Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8244752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8244859Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8244864Z 2025-09-07T07:16:58.8244971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8245192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8245267Z return mod(**inputs) 2025-09-07T07:16:58.8245504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8245590Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8245837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8245907Z layer_outputs = layer_module( 2025-09-07T07:16:58.8246135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8246212Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8246449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8246547Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8246778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8246865Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8247092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8247195Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8247205Z 2025-09-07T07:16:58.8247303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8247498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8247569Z return mod(**inputs) 2025-09-07T07:16:58.8247798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8247875Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8248107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8248229Z layer_outputs = layer_module( 2025-09-07T07:16:58.8248452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8248528Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8248763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8248839Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8249064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8249147Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8249370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8249469Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8249472Z 2025-09-07T07:16:58.8249575Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8249777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8249841Z return mod(**inputs) 2025-09-07T07:16:58.8250075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8250154Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8250388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8250464Z layer_outputs = layer_module( 2025-09-07T07:16:58.8250682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8250758Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8251016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8251097Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8251331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-09-07T07:16:58.8251477Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:16:58.8251481Z 2025-09-07T07:16:58.8251560Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8251669Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8251863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8251932Z return mod(**inputs) 2025-09-07T07:16:58.8252161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8252234Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8252471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8252542Z layer_outputs = layer_module( 2025-09-07T07:16:58.8252771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8252846Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8253082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8253158Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8253384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-09-07T07:16:58.8253492Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8253721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8253805Z return self.weight * hidden_states 2025-09-07T07:16:58.8253826Z 2025-09-07T07:16:58.8253929Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8254128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8254201Z return mod(**inputs) 2025-09-07T07:16:58.8254438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8254516Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8254751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8254826Z layer_outputs = layer_module( 2025-09-07T07:16:58.8255046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8255138Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8255388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8255467Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8255700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8255783Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8256013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8256097Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8256101Z 2025-09-07T07:16:58.8256202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8256404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8256471Z return mod(**inputs) 2025-09-07T07:16:58.8256703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8256807Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8257043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8257120Z layer_outputs = layer_module( 2025-09-07T07:16:58.8257355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8257439Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8257671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8257751Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8257990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8258075Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8258317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8258394Z key_states = self.k(current_states) 2025-09-07T07:16:58.8258398Z 2025-09-07T07:16:58.8258500Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8258709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8258775Z return mod(**inputs) 2025-09-07T07:16:58.8259019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8259091Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8259329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8259406Z layer_outputs = layer_module( 2025-09-07T07:16:58.8259625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8259709Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8259960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8260047Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8260280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8260362Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8260599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8260725Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8260729Z 2025-09-07T07:16:58.8260836Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8261054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8261120Z return mod(**inputs) 2025-09-07T07:16:58.8261369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8261441Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8261683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8261752Z layer_outputs = layer_module( 2025-09-07T07:16:58.8261972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8262054Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8262287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8262373Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8262609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8262716Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8262954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8263121Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8263126Z 2025-09-07T07:16:58.8263236Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8263434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8263507Z return mod(**inputs) 2025-09-07T07:16:58.8263743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8263815Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8264065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8264138Z layer_outputs = layer_module( 2025-09-07T07:16:58.8264375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8264455Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8264704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8264786Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8265025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8265117Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8265357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8265444Z value_states = self.v(current_states) 2025-09-07T07:16:58.8265448Z 2025-09-07T07:16:58.8265553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8265974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8266055Z return mod(**inputs) 2025-09-07T07:16:58.8266307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8266392Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8266699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8266776Z layer_outputs = layer_module( 2025-09-07T07:16:58.8267028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8267112Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8267432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8267512Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8267758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8267856Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8268088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8268204Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8268207Z 2025-09-07T07:16:58.8268308Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8268510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8268574Z return mod(**inputs) 2025-09-07T07:16:58.8268809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8268891Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8269154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8269235Z layer_outputs = layer_module( 2025-09-07T07:16:58.8269479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8269558Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8269797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8269875Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8270115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8270196Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8270435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8270543Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8270547Z 2025-09-07T07:16:58.8270650Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8270858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8270925Z return mod(**inputs) 2025-09-07T07:16:58.8271170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8271244Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8271486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8271565Z layer_outputs = layer_module( 2025-09-07T07:16:58.8271779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8271862Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8272089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8272190Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8272416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8272496Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8272728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8272829Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8272832Z 2025-09-07T07:16:58.8272937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8273129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8273211Z return mod(**inputs) 2025-09-07T07:16:58.8273448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8273520Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8273752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8273823Z layer_outputs = layer_module( 2025-09-07T07:16:58.8274042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8274125Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8274356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8274440Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8274671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8274759Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8275003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8275083Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8275087Z 2025-09-07T07:16:58.8275176Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8275294Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8275499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8275564Z return mod(**inputs) 2025-09-07T07:16:58.8275800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8275879Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8276106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8276182Z layer_outputs = layer_module( 2025-09-07T07:16:58.8276395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8276470Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8276702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8276791Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8277023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8277117Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8277350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8277427Z return self.weight * hidden_states 2025-09-07T07:16:58.8277431Z 2025-09-07T07:16:58.8277531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8277732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8277814Z return mod(**inputs) 2025-09-07T07:16:58.8278057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8278128Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8278363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8278438Z layer_outputs = layer_module( 2025-09-07T07:16:58.8278658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8278740Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8278973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8279082Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8279316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8279433Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8279671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8279751Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8279755Z 2025-09-07T07:16:58.8279863Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8280058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8280123Z return mod(**inputs) 2025-09-07T07:16:58.8280364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8280438Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8280692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8280766Z layer_outputs = layer_module( 2025-09-07T07:16:58.8280989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8281092Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8281330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8281428Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8281676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8281807Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8282059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8282146Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8282153Z 2025-09-07T07:16:58.8282271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8282484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8282563Z return mod(**inputs) 2025-09-07T07:16:58.8282815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8282892Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8283151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8283226Z layer_outputs = layer_module( 2025-09-07T07:16:58.8283463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8283542Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8283773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8283884Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8284117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8284259Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8284490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8284576Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8284580Z 2025-09-07T07:16:58.8284660Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8284762Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8284984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8285049Z return mod(**inputs) 2025-09-07T07:16:58.8285292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8285365Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8285597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8285677Z layer_outputs = layer_module( 2025-09-07T07:16:58.8285893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8285976Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8286209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8286289Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8286529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.8286649Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8286888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8286966Z return self.weight * hidden_states 2025-09-07T07:16:58.8286969Z 2025-09-07T07:16:58.8287089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8287289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8287353Z return mod(**inputs) 2025-09-07T07:16:58.8287591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8287660Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8287898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8287969Z layer_outputs = layer_module( 2025-09-07T07:16:58.8288188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8288274Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8288504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8288588Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8288815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8288903Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8289131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8289206Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8289211Z 2025-09-07T07:16:58.8289319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8289517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8289608Z return mod(**inputs) 2025-09-07T07:16:58.8289848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8289920Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8290164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8290234Z layer_outputs = layer_module( 2025-09-07T07:16:58.8290458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8290534Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8290765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8290868Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8291098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8291185Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8291414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8291497Z key_states = self.k(current_states) 2025-09-07T07:16:58.8291500Z 2025-09-07T07:16:58.8291601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8291795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8291866Z return mod(**inputs) 2025-09-07T07:16:58.8292107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8292187Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8292420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8292515Z layer_outputs = layer_module( 2025-09-07T07:16:58.8292745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8292821Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8293072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8293151Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8293389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8293475Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8293706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8293843Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8293847Z 2025-09-07T07:16:58.8293954Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8294159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8294225Z return mod(**inputs) 2025-09-07T07:16:58.8294460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8294540Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8294772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8294851Z layer_outputs = layer_module( 2025-09-07T07:16:58.8295076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8295155Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8295395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8295501Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8295741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8295821Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8296052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8296215Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8296218Z 2025-09-07T07:16:58.8296319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8296524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8296605Z return mod(**inputs) 2025-09-07T07:16:58.8296847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8296922Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8297157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8297234Z layer_outputs = layer_module( 2025-09-07T07:16:58.8297455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8297540Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8297771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8297850Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8298087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8298170Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8298426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8298506Z value_states = self.v(current_states) 2025-09-07T07:16:58.8298510Z 2025-09-07T07:16:58.8298618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8298830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8298896Z return mod(**inputs) 2025-09-07T07:16:58.8299135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8299208Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8299449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8299518Z layer_outputs = layer_module( 2025-09-07T07:16:58.8299741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8299826Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8300061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8300145Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8300376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8300455Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8300691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8300800Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8300804Z 2025-09-07T07:16:58.8300914Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8301113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8301184Z return mod(**inputs) 2025-09-07T07:16:58.8301437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8301509Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8301753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8301821Z layer_outputs = layer_module( 2025-09-07T07:16:58.8302045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8302121Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8302349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8302434Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8302684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8302769Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8302999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8303104Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8303118Z 2025-09-07T07:16:58.8303217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8303411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8303483Z return mod(**inputs) 2025-09-07T07:16:58.8303722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8303800Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8304034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8304103Z layer_outputs = layer_module( 2025-09-07T07:16:58.8304347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8304427Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8304684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8304767Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8305007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8305098Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8305338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8305456Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8305459Z 2025-09-07T07:16:58.8305564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8305861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8305938Z return mod(**inputs) 2025-09-07T07:16:58.8306182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8306267Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8306511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8306592Z layer_outputs = layer_module( 2025-09-07T07:16:58.8306819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8306901Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8307164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8307252Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8307535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8307622Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8307876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8307968Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8307972Z 2025-09-07T07:16:58.8308059Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8308181Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8308397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8308476Z return mod(**inputs) 2025-09-07T07:16:58.8308729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8308826Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8309089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8309167Z layer_outputs = layer_module( 2025-09-07T07:16:58.8309415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8309498Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8309749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8309843Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8310092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-09-07T07:16:58.8310213Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8310469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8310574Z return self.weight * hidden_states 2025-09-07T07:16:58.8310585Z 2025-09-07T07:16:58.8310697Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8310907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8311003Z return mod(**inputs) 2025-09-07T07:16:58.8311257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8311344Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8311596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8311672Z layer_outputs = layer_module( 2025-09-07T07:16:58.8311917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8312003Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8312276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8312365Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8312619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8312717Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8312971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8313061Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8313065Z 2025-09-07T07:16:58.8313174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8313395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8313467Z return mod(**inputs) 2025-09-07T07:16:58.8313723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8313829Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8314084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8314168Z layer_outputs = layer_module( 2025-09-07T07:16:58.8314407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8314490Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8314747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8314831Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8315089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8315207Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8315467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8315562Z key_states = self.k(current_states) 2025-09-07T07:16:58.8315566Z 2025-09-07T07:16:58.8315682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8315908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8315980Z return mod(**inputs) 2025-09-07T07:16:58.8316235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8316322Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8316582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8316669Z layer_outputs = layer_module( 2025-09-07T07:16:58.8316931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8317024Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8317275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8317375Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8317638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8317729Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8317986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8318122Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8318128Z 2025-09-07T07:16:58.8318239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8318465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8318540Z return mod(**inputs) 2025-09-07T07:16:58.8318810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8318887Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8319149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8319224Z layer_outputs = layer_module( 2025-09-07T07:16:58.8319463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8319740Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8320154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8320278Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8320541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8320686Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8320947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8321116Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8321124Z 2025-09-07T07:16:58.8321243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8321461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8321540Z return mod(**inputs) 2025-09-07T07:16:58.8321791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8321895Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8322149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8322224Z layer_outputs = layer_module( 2025-09-07T07:16:58.8322461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8322542Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8322783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8322875Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8323114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8323205Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8323443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8323523Z value_states = self.v(current_states) 2025-09-07T07:16:58.8323527Z 2025-09-07T07:16:58.8323666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8323872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8323948Z return mod(**inputs) 2025-09-07T07:16:58.8324212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8324296Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8324536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8324609Z layer_outputs = layer_module( 2025-09-07T07:16:58.8324841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8324920Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8325169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8325263Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8325498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8325589Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8325827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8325945Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8325948Z 2025-09-07T07:16:58.8326051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8326255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8326332Z return mod(**inputs) 2025-09-07T07:16:58.8326572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8326655Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8326918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8326998Z layer_outputs = layer_module( 2025-09-07T07:16:58.8327225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8327303Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8327545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8327625Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8327869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8327971Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8328213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8328332Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8328336Z 2025-09-07T07:16:58.8328439Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8328652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8328719Z return mod(**inputs) 2025-09-07T07:16:58.8328962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8329042Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8329283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8329362Z layer_outputs = layer_module( 2025-09-07T07:16:58.8329589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8329691Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8329933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8330013Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8330280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8330363Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8330602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8330707Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8330711Z 2025-09-07T07:16:58.8330811Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8331015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8331078Z return mod(**inputs) 2025-09-07T07:16:58.8331320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8331392Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8331641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8331712Z layer_outputs = layer_module( 2025-09-07T07:16:58.8331936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8332022Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8332259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8332346Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8332585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8332688Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8332938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8333015Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8333019Z 2025-09-07T07:16:58.8333128Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8333330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8333395Z return mod(**inputs) 2025-09-07T07:16:58.8333636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8333706Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8333948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8334044Z layer_outputs = layer_module( 2025-09-07T07:16:58.8334292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8334376Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8334630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8334724Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8334974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward 2025-09-07T07:16:58.8335122Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-09-07T07:16:58.8335125Z 2025-09-07T07:16:58.8335212Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8335320Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8335543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8335614Z return mod(**inputs) 2025-09-07T07:16:58.8335900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8335975Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8336224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8336305Z layer_outputs = layer_module( 2025-09-07T07:16:58.8336536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8336624Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8336865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8336976Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8337211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8337311Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8337551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8337628Z return self.weight * hidden_states 2025-09-07T07:16:58.8337633Z 2025-09-07T07:16:58.8337740Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8337942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8338008Z return mod(**inputs) 2025-09-07T07:16:58.8338252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8338324Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8338568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8338637Z layer_outputs = layer_module( 2025-09-07T07:16:58.8339278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8339366Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8339608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8339712Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8339955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8340087Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8340327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8340431Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8340435Z 2025-09-07T07:16:58.8340548Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8340752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8340827Z return mod(**inputs) 2025-09-07T07:16:58.8341073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8341149Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8341413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8341490Z layer_outputs = layer_module( 2025-09-07T07:16:58.8341733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8341815Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8342067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8342170Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8342441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8342577Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8342881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8342980Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8342984Z 2025-09-07T07:16:58.8343094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8343308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8343387Z return mod(**inputs) 2025-09-07T07:16:58.8343644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8343731Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8343989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8344069Z layer_outputs = layer_module( 2025-09-07T07:16:58.8344321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8344407Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8344665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8344759Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8345021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8345144Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8345398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8345493Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8345521Z 2025-09-07T07:16:58.8345608Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8345792Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8346018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8346089Z return mod(**inputs) 2025-09-07T07:16:58.8346353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8346431Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8346693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8346780Z layer_outputs = layer_module( 2025-09-07T07:16:58.8347030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8347121Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8347368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8347460Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8347717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.8347843Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8348098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8348183Z return self.weight * hidden_states 2025-09-07T07:16:58.8348186Z 2025-09-07T07:16:58.8348305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8348522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8348598Z return mod(**inputs) 2025-09-07T07:16:58.8348870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8348951Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8349233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8349310Z layer_outputs = layer_module( 2025-09-07T07:16:58.8349556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8349639Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8349893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8349985Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8350238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8350336Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8350589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8350677Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8350681Z 2025-09-07T07:16:58.8350793Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8351006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8351084Z return mod(**inputs) 2025-09-07T07:16:58.8351341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8351427Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8351681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8351758Z layer_outputs = layer_module( 2025-09-07T07:16:58.8352004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8352105Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8352368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8352453Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8352706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8352801Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8353055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8353146Z key_states = self.k(current_states) 2025-09-07T07:16:58.8353167Z 2025-09-07T07:16:58.8353278Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8353501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8353573Z return mod(**inputs) 2025-09-07T07:16:58.8353819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8353901Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8354137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8354215Z layer_outputs = layer_module( 2025-09-07T07:16:58.8354434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8354512Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8354750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8354830Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8355086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8355169Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8355421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8355554Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8355557Z 2025-09-07T07:16:58.8355659Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8355866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8355930Z return mod(**inputs) 2025-09-07T07:16:58.8356173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8356246Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8356482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8356563Z layer_outputs = layer_module( 2025-09-07T07:16:58.8356781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8356868Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8357103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8357180Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8357416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8357494Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8357733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8357888Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8357909Z 2025-09-07T07:16:58.8358017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8358214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8358279Z return mod(**inputs) 2025-09-07T07:16:58.8358523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8358594Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8358836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8358905Z layer_outputs = layer_module( 2025-09-07T07:16:58.8359124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8359226Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8359460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8359546Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8359777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8359866Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8360095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8360172Z value_states = self.v(current_states) 2025-09-07T07:16:58.8360176Z 2025-09-07T07:16:58.8360288Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8360486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8360560Z return mod(**inputs) 2025-09-07T07:16:58.8360797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8360888Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8361140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8361211Z layer_outputs = layer_module( 2025-09-07T07:16:58.8361465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8361546Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8361785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8361872Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8362114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8362199Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8362431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8362548Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8362552Z 2025-09-07T07:16:58.8362651Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8362856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8362929Z return mod(**inputs) 2025-09-07T07:16:58.8363164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8363241Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8363475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8363546Z layer_outputs = layer_module( 2025-09-07T07:16:58.8363771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8363870Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8364110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8364186Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8364421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8364510Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8364742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8364856Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8364859Z 2025-09-07T07:16:58.8364960Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8365183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8365248Z return mod(**inputs) 2025-09-07T07:16:58.8365485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8365565Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8365804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8365882Z layer_outputs = layer_module( 2025-09-07T07:16:58.8366100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8366178Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8366420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8366500Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8366742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8366841Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8367075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8367203Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8367207Z 2025-09-07T07:16:58.8367309Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8367515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8367580Z return mod(**inputs) 2025-09-07T07:16:58.8367822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8367895Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8368133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8368212Z layer_outputs = layer_module( 2025-09-07T07:16:58.8368437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8368524Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8368760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8368842Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8369084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8369163Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8369404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8369482Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8369485Z 2025-09-07T07:16:58.8369575Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8369678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8369894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8369968Z return mod(**inputs) 2025-09-07T07:16:58.8370205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8370285Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8370520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8370588Z layer_outputs = layer_module( 2025-09-07T07:16:58.8370816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8370910Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8371154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8371238Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8371478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-09-07T07:16:58.8371596Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8371834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8371919Z return self.weight * hidden_states 2025-09-07T07:16:58.8371923Z 2025-09-07T07:16:58.8372026Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8372234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8372301Z return mod(**inputs) 2025-09-07T07:16:58.8372544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8372625Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8372889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8372969Z layer_outputs = layer_module( 2025-09-07T07:16:58.8373207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8373288Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8373535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8373617Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8373863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8373952Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8374191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8374281Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8374285Z 2025-09-07T07:16:58.8374389Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8374602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8374668Z return mod(**inputs) 2025-09-07T07:16:58.8374915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8374988Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8375228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8375308Z layer_outputs = layer_module( 2025-09-07T07:16:58.8375531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8375618Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8375875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8375956Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8376201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8376286Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8376529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8376608Z key_states = self.k(current_states) 2025-09-07T07:16:58.8376612Z 2025-09-07T07:16:58.8376719Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8376919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8377010Z return mod(**inputs) 2025-09-07T07:16:58.8377256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8377331Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8377579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8377652Z layer_outputs = layer_module( 2025-09-07T07:16:58.8377877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8377963Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8378201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8378288Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8378529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8378614Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8378874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8379009Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8379013Z 2025-09-07T07:16:58.8379141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8379344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8379421Z return mod(**inputs) 2025-09-07T07:16:58.8379661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8379733Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8379978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8380052Z layer_outputs = layer_module( 2025-09-07T07:16:58.8380286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8380368Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8380608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8380698Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8380934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8381025Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8381270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8381438Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8381452Z 2025-09-07T07:16:58.8381561Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8381773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8381871Z return mod(**inputs) 2025-09-07T07:16:58.8382127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8382213Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8382476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8382548Z layer_outputs = layer_module( 2025-09-07T07:16:58.8382780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8382858Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8383107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8383212Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8383455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8383550Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8383789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8383874Z value_states = self.v(current_states) 2025-09-07T07:16:58.8383877Z 2025-09-07T07:16:58.8383980Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8384190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8384257Z return mod(**inputs) 2025-09-07T07:16:58.8384496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8384577Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8384844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8384927Z layer_outputs = layer_module( 2025-09-07T07:16:58.8385154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8385249Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8385504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8385589Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8385917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8386012Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8386266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8386395Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8386402Z 2025-09-07T07:16:58.8386515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8386742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8386815Z return mod(**inputs) 2025-09-07T07:16:58.8387078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8387159Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8387414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8387499Z layer_outputs = layer_module( 2025-09-07T07:16:58.8387740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8387839Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8388080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8388188Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8388435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8388519Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8388767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8388877Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8388880Z 2025-09-07T07:16:58.8388983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8389202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8389296Z return mod(**inputs) 2025-09-07T07:16:58.8389562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8389643Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8389914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8389990Z layer_outputs = layer_module( 2025-09-07T07:16:58.8390237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8390326Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8390584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8390675Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8390932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8391022Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8391305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8391423Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8391427Z 2025-09-07T07:16:58.8391544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8391778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8391847Z return mod(**inputs) 2025-09-07T07:16:58.8392098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8392176Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8392439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8392518Z layer_outputs = layer_module( 2025-09-07T07:16:58.8392760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8392847Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8393100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8393193Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8393448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8393545Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8393796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8393881Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8393885Z 2025-09-07T07:16:58.8393982Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8394095Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8394319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8394420Z return mod(**inputs) 2025-09-07T07:16:58.8394678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8394764Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8395029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8395112Z layer_outputs = layer_module( 2025-09-07T07:16:58.8395350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8395442Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8395703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8395826Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8396089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8396196Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8396455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8396540Z return self.weight * hidden_states 2025-09-07T07:16:58.8396544Z 2025-09-07T07:16:58.8396652Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8396874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8396944Z return mod(**inputs) 2025-09-07T07:16:58.8397205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8397284Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8397533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8397636Z layer_outputs = layer_module( 2025-09-07T07:16:58.8397878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8397970Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8398243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8398349Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8398602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8398729Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8398987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8399074Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8399078Z 2025-09-07T07:16:58.8399196Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8399409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8399480Z return mod(**inputs) 2025-09-07T07:16:58.8399743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8399820Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8400083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8400160Z layer_outputs = layer_module( 2025-09-07T07:16:58.8400408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8400493Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8400749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8400868Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8401104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8401231Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8401482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8401568Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8401572Z 2025-09-07T07:16:58.8401688Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8401901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8401979Z return mod(**inputs) 2025-09-07T07:16:58.8402253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8402333Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8402596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8402671Z layer_outputs = layer_module( 2025-09-07T07:16:58.8402922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8403004Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8403268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8403358Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8403595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8403722Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8403976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8404065Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8404069Z 2025-09-07T07:16:58.8404172Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8404390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8404467Z return mod(**inputs) 2025-09-07T07:16:58.8404708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8404789Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8405031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8405109Z layer_outputs = layer_module( 2025-09-07T07:16:58.8405338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8405419Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8405672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8405762Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8406013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-09-07T07:16:58.8406144Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-09-07T07:16:58.8406148Z 2025-09-07T07:16:58.8406233Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8406345Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8406553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8406629Z return mod(**inputs) 2025-09-07T07:16:58.8406869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8406963Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8407210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8407281Z layer_outputs = layer_module( 2025-09-07T07:16:58.8407517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8407597Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8407842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8407924Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8408162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-09-07T07:16:58.8408301Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8408541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8408629Z return self.weight * hidden_states 2025-09-07T07:16:58.8408632Z 2025-09-07T07:16:58.8408734Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8408934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8409008Z return mod(**inputs) 2025-09-07T07:16:58.8409245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8409327Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8409566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8409641Z layer_outputs = layer_module( 2025-09-07T07:16:58.8409872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8409972Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8410220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8410303Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8410578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8410672Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8410919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8411009Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8411013Z 2025-09-07T07:16:58.8411123Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8411347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8411419Z return mod(**inputs) 2025-09-07T07:16:58.8411676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8411756Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8411995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8412074Z layer_outputs = layer_module( 2025-09-07T07:16:58.8412301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8412383Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8412628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8412709Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8412955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8413061Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8413309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8413391Z key_states = self.k(current_states) 2025-09-07T07:16:58.8413396Z 2025-09-07T07:16:58.8413503Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8413716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8413786Z return mod(**inputs) 2025-09-07T07:16:58.8414035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8414110Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8414355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8414457Z layer_outputs = layer_module( 2025-09-07T07:16:58.8414686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8414774Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8415014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8415095Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8415341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8415423Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8415667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8415800Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8415807Z 2025-09-07T07:16:58.8415918Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8416139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8416210Z return mod(**inputs) 2025-09-07T07:16:58.8416470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8416563Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8416824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8416899Z layer_outputs = layer_module( 2025-09-07T07:16:58.8417142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8417229Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8417466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8417555Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8417794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8417882Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8418119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8418276Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8418279Z 2025-09-07T07:16:58.8418390Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8418590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8418663Z return mod(**inputs) 2025-09-07T07:16:58.8418901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8418975Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8419223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8419312Z layer_outputs = layer_module( 2025-09-07T07:16:58.8419693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8419820Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8420199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8420298Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8420553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8420648Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8420950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8421041Z value_states = self.v(current_states) 2025-09-07T07:16:58.8421048Z 2025-09-07T07:16:58.8421160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8421373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8421457Z return mod(**inputs) 2025-09-07T07:16:58.8421720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8421805Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8422060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8422135Z layer_outputs = layer_module( 2025-09-07T07:16:58.8422384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8422472Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8422758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8422847Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8423135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8423223Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8423475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8423599Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8423603Z 2025-09-07T07:16:58.8423714Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8423934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8424007Z return mod(**inputs) 2025-09-07T07:16:58.8424262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8424349Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8424605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8424690Z layer_outputs = layer_module( 2025-09-07T07:16:58.8424937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8425024Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8425290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8425377Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8425644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8425777Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8426055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8426208Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8426213Z 2025-09-07T07:16:58.8426325Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8426555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8426628Z return mod(**inputs) 2025-09-07T07:16:58.8426901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8426981Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8427236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8427341Z layer_outputs = layer_module( 2025-09-07T07:16:58.8427581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8427677Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8427930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8428016Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8428277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8428364Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8428619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8428734Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8428741Z 2025-09-07T07:16:58.8428859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8429071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8429162Z return mod(**inputs) 2025-09-07T07:16:58.8429427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8429506Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8429786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8429864Z layer_outputs = layer_module( 2025-09-07T07:16:58.8430103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8430195Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8430448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-09-07T07:16:58.8430541Z self_attention_outputs = self.layer[0]( 2025-09-07T07:16:58.8430797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-09-07T07:16:58.8430893Z attention_output = self.SelfAttention( 2025-09-07T07:16:58.8431147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8431232Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8431236Z 2025-09-07T07:16:58.8431334Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8431445Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8431667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8431738Z return mod(**inputs) 2025-09-07T07:16:58.8431997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8432084Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8432335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8432437Z layer_outputs = layer_module( 2025-09-07T07:16:58.8432661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8432739Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8432984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8433064Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8433305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-09-07T07:16:58.8433410Z normed_hidden_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8433652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8433766Z return self.weight * hidden_states 2025-09-07T07:16:58.8433770Z 2025-09-07T07:16:58.8433874Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8434080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8434144Z return mod(**inputs) 2025-09-07T07:16:58.8434387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8434459Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8434692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8434767Z layer_outputs = layer_module( 2025-09-07T07:16:58.8434986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8435071Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8435301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8435400Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8435640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8435740Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8435982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-09-07T07:16:58.8436058Z query_states = self.q(hidden_states) 2025-09-07T07:16:58.8436062Z 2025-09-07T07:16:58.8436171Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8436371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8436440Z return mod(**inputs) 2025-09-07T07:16:58.8436707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8436783Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8437051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8437127Z layer_outputs = layer_module( 2025-09-07T07:16:58.8437370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8437460Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8437716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8437809Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8438062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8438148Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8438396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-09-07T07:16:58.8438507Z key_states = self.k(current_states) 2025-09-07T07:16:58.8438511Z 2025-09-07T07:16:58.8438619Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8438817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8438891Z return mod(**inputs) 2025-09-07T07:16:58.8439122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8439194Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8439435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8439505Z layer_outputs = layer_module( 2025-09-07T07:16:58.8439747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8439824Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8440060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8440147Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8440378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8440468Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8440697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-09-07T07:16:58.8440826Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-09-07T07:16:58.8440838Z 2025-09-07T07:16:58.8440938Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8441135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8441208Z return mod(**inputs) 2025-09-07T07:16:58.8441481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8441565Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8441813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8441885Z layer_outputs = layer_module( 2025-09-07T07:16:58.8442109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8442184Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8442422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8442511Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8442745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8442836Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8443067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-09-07T07:16:58.8443227Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-09-07T07:16:58.8443233Z 2025-09-07T07:16:58.8443335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8443541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8443606Z return mod(**inputs) 2025-09-07T07:16:58.8443841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8443925Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8444158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8444234Z layer_outputs = layer_module( 2025-09-07T07:16:58.8444479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8444556Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8444799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8444878Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8445115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8445196Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8445429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-09-07T07:16:58.8445529Z value_states = self.v(current_states) 2025-09-07T07:16:58.8445533Z 2025-09-07T07:16:58.8445631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8445839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8445905Z return mod(**inputs) 2025-09-07T07:16:58.8446149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8446221Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8446457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8446534Z layer_outputs = layer_module( 2025-09-07T07:16:58.8446753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8446837Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8447070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8447151Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8447407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8447490Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8447749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8447859Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8447862Z 2025-09-07T07:16:58.8447969Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8448167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8448232Z return mod(**inputs) 2025-09-07T07:16:58.8448470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8448542Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8448782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8448853Z layer_outputs = layer_module( 2025-09-07T07:16:58.8449073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8449157Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8449389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8449474Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8449703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8449784Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8450026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-09-07T07:16:58.8450131Z attn_output = torch.matmul(attn_weights, value_states) 2025-09-07T07:16:58.8450157Z 2025-09-07T07:16:58.8450266Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8450463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8450537Z return mod(**inputs) 2025-09-07T07:16:58.8450772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8450844Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8451086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8451156Z layer_outputs = layer_module( 2025-09-07T07:16:58.8451382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8451478Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8451709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8451798Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8452029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8452118Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8452347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-09-07T07:16:58.8452454Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-09-07T07:16:58.8452464Z 2025-09-07T07:16:58.8452567Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8452774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8452850Z return mod(**inputs) 2025-09-07T07:16:58.8453095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8453176Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8453404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8453496Z layer_outputs = layer_module( 2025-09-07T07:16:58.8453726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8453803Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8454041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-09-07T07:16:58.8454119Z cross_attention_outputs = self.layer[1]( 2025-09-07T07:16:58.8454351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-09-07T07:16:58.8454442Z attention_output = self.EncDecAttention( 2025-09-07T07:16:58.8454675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-09-07T07:16:58.8454761Z attn_output = self.o(attn_output) 2025-09-07T07:16:58.8454764Z 2025-09-07T07:16:58.8454844Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8454946Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8455150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8455215Z return mod(**inputs) 2025-09-07T07:16:58.8455455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8455526Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8455767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8455840Z layer_outputs = layer_module( 2025-09-07T07:16:58.8456058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8456158Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8456389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8456488Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8456718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-09-07T07:16:58.8456814Z forwarded_states = self.layer_norm(hidden_states) 2025-09-07T07:16:58.8457061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-09-07T07:16:58.8457136Z return self.weight * hidden_states 2025-09-07T07:16:58.8457157Z 2025-09-07T07:16:58.8457265Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8457459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8457526Z return mod(**inputs) 2025-09-07T07:16:58.8457765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8457838Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8458079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8458148Z layer_outputs = layer_module( 2025-09-07T07:16:58.8458369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8458444Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8458673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8458772Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8459016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8459141Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8459387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-09-07T07:16:58.8459472Z hidden_states = self.wi(hidden_states) 2025-09-07T07:16:58.8459482Z 2025-09-07T07:16:58.8459588Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8459791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8459866Z return mod(**inputs) 2025-09-07T07:16:58.8460126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8460214Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8460500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8460578Z layer_outputs = layer_module( 2025-09-07T07:16:58.8460828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8460913Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8461176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8461271Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8461528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8461658Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8461918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-09-07T07:16:58.8462015Z hidden_states = self.act(hidden_states) 2025-09-07T07:16:58.8462038Z 2025-09-07T07:16:58.8462151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8462365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8462444Z return mod(**inputs) 2025-09-07T07:16:58.8462699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-09-07T07:16:58.8462785Z decoder_outputs = self.decoder( 2025-09-07T07:16:58.8463038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-09-07T07:16:58.8463119Z layer_outputs = layer_module( 2025-09-07T07:16:58.8463356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:16:58.8463456Z return super().__call__(*args, **kwargs) 2025-09-07T07:16:58.8463718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-09-07T07:16:58.8463815Z hidden_states = self.layer[-1](hidden_states) 2025-09-07T07:16:58.8464084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-09-07T07:16:58.8464209Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-09-07T07:16:58.8464462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-09-07T07:16:58.8464555Z hidden_states = self.wo(hidden_states) 2025-09-07T07:16:58.8464559Z 2025-09-07T07:16:58.8464646Z cudagraph partition due to non gpu ops 2025-09-07T07:16:58.8464763Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8464983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8465056Z return mod(**inputs) 2025-09-07T07:16:58.8465340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward 2025-09-07T07:16:58.8465475Z sequence_output = sequence_output * (self.model_dim**-0.5) 2025-09-07T07:16:58.8465479Z 2025-09-07T07:16:58.8465595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8465927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8466021Z return mod(**inputs) 2025-09-07T07:16:58.8466288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward 2025-09-07T07:16:58.8466384Z lm_logits = self.lm_head(sequence_output) 2025-09-07T07:16:58.8466391Z 2025-09-07T07:16:58.8466513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8466734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8466815Z return mod(**inputs) 2025-09-07T07:16:58.8467082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-09-07T07:16:58.8467245Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-09-07T07:16:58.8467256Z 2025-09-07T07:16:58.8467373Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8467605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8467687Z return mod(**inputs) 2025-09-07T07:16:58.8467945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-09-07T07:16:58.8468088Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-09-07T07:16:58.8468092Z 2025-09-07T07:16:58.8468192Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:16:58.8468401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:16:58.8468485Z return mod(**inputs) 2025-09-07T07:16:58.8468779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-09-07T07:16:58.8468933Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-09-07T07:16:58.8468939Z 2025-09-07T07:17:11.2178630Z Compilation time (from dynamo_timed): 20.680033084 2025-09-07T07:17:11.2297285Z pass 2025-09-07T07:17:11.2297846Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:11.2302632Z TIMING: _recursive_pre_grad_passes:0.01211 _recursive_joint_graph_passes:0.58134 _recursive_post_grad_passes:0.18883 async_compile.wait:0.78326 code_gen:12.06346 inductor_compile:13.77271 backend_compile:17.82794 gc:0.00123 entire_frame_compile:20.68003 total_wall_time:20.68003 2025-09-07T07:17:11.2304030Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:20423 | FakeTensor.__torch_dispatch__:5324 | ProxyTorchDispatchMode.__torch_dispatch__:7292 2025-09-07T07:17:11.2304600Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-09-07T07:17:14.0091496Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:17:14.0092443Z import pynvml # type: ignore[import] 2025-09-07T07:17:16.8143608Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:17:16.8144649Z from pkg_resources import resource_filename 2025-09-07T07:17:17.4788158Z 2025-09-07T07:17:18.4664885Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:17:18.4665612Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:17:18.4678977Z cpu eval T5Small 2025-09-07T07:17:20.0582064Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:20.5059100Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:20.9340850Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:34.8297553Z Compilation time (from dynamo_timed): 12.256734653 2025-09-07T07:17:34.8497361Z pass 2025-09-07T07:17:34.8498068Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:34.8502289Z TIMING: _recursive_pre_grad_passes:0.01196 async_compile.wait:0.00649 backend_compile:9.36835 gc:0.00042 entire_frame_compile:12.25673 total_wall_time:12.25673 2025-09-07T07:17:34.8503298Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:2289 | FakeTensor.__torch_dispatch__:17 2025-09-07T07:17:34.8503856Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-09-07T07:17:37.2147679Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:17:37.2152734Z import pynvml # type: ignore[import] 2025-09-07T07:17:40.0035221Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:17:40.0036529Z from pkg_resources import resource_filename 2025-09-07T07:17:40.6751018Z 2025-09-07T07:17:43.1344111Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:17:43.1344535Z loading model: 0it [00:02, ?it/s] 2025-09-07T07:17:43.1365324Z cpu eval TrOCRForCausalLM 2025-09-07T07:17:43.2948297Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-09-07T07:17:43.3391259Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:43.6134456Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:43.8760696Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:17:51.7550344Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7553360Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7554083Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7556749Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7557084Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7557423Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7557771Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7558071Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7558307Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7558551Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7558869Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7563367Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7563668Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7564142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7564520Z return mod(**inputs) 2025-09-07T07:17:51.7564963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7565412Z outputs = self.model.decoder( 2025-09-07T07:17:51.7565916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7566354Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7566787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7567208Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7567641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7568093Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7568548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7569019Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7569211Z 2025-09-07T07:17:51.7569337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7569738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7570075Z return mod(**inputs) 2025-09-07T07:17:51.7570454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7570849Z outputs = self.model.decoder( 2025-09-07T07:17:51.7571239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7571632Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7571988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7572378Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7572943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7573418Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7573941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7574389Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7574553Z 2025-09-07T07:17:51.7574678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7575084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7575434Z return mod(**inputs) 2025-09-07T07:17:51.7575814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7576227Z outputs = self.model.decoder( 2025-09-07T07:17:51.7576630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7577061Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7577422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7577792Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7578191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7578615Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7579036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7579440Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7579595Z 2025-09-07T07:17:51.7579682Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7579916Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7580147Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7580402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7580811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7581171Z return mod(**inputs) 2025-09-07T07:17:51.7581576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7582038Z outputs = self.model.decoder( 2025-09-07T07:17:51.7582614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7583056Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7583442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7583848Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7584272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7584714Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7585165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7585606Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7585914Z 2025-09-07T07:17:51.7586047Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7586452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7586829Z return mod(**inputs) 2025-09-07T07:17:51.7587254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7587689Z outputs = self.model.decoder( 2025-09-07T07:17:51.7588127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7588560Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7588958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7589389Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7589823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7590311Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7590506Z 2025-09-07T07:17:51.7590624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7591024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7591401Z return mod(**inputs) 2025-09-07T07:17:51.7591823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7592317Z outputs = self.model.decoder( 2025-09-07T07:17:51.7592756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7593190Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7593577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7593984Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7594411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7594897Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7595332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7595717Z return self.act(input) 2025-09-07T07:17:51.7595843Z 2025-09-07T07:17:51.7595967Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7596365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7596737Z return mod(**inputs) 2025-09-07T07:17:51.7597131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7597532Z outputs = self.model.decoder( 2025-09-07T07:17:51.7598616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7599010Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7599370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7599742Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7600139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7600553Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7600710Z 2025-09-07T07:17:51.7600821Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7601215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7601569Z return mod(**inputs) 2025-09-07T07:17:51.7601965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7602380Z outputs = self.model.decoder( 2025-09-07T07:17:51.7602790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7603206Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7603582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7603983Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7604403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7604831Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7605331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7605773Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7605944Z 2025-09-07T07:17:51.7606050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7606420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7606755Z return mod(**inputs) 2025-09-07T07:17:51.7607125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7607526Z outputs = self.model.decoder( 2025-09-07T07:17:51.7607934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7608328Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7608690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7609072Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7609489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7609936Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7610378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7610804Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7610949Z 2025-09-07T07:17:51.7611071Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7611453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7611811Z return mod(**inputs) 2025-09-07T07:17:51.7612229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7612653Z outputs = self.model.decoder( 2025-09-07T07:17:51.7613087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7613499Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7613878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7614267Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7614673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7615113Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7615560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7615992Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7616146Z 2025-09-07T07:17:51.7616243Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7616475Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7616695Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7616949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7617340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7617694Z return mod(**inputs) 2025-09-07T07:17:51.7618078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7618493Z outputs = self.model.decoder( 2025-09-07T07:17:51.7618905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7619342Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7619997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7620464Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7620896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7621351Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7621799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7622233Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7622396Z 2025-09-07T07:17:51.7622513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7622922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7623326Z return mod(**inputs) 2025-09-07T07:17:51.7623729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7624167Z outputs = self.model.decoder( 2025-09-07T07:17:51.7624582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7625005Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7625385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7625823Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7626253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7626737Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7626936Z 2025-09-07T07:17:51.7627061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7627459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7627859Z return mod(**inputs) 2025-09-07T07:17:51.7628261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7628689Z outputs = self.model.decoder( 2025-09-07T07:17:51.7629136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7629551Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7629930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7630323Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7630747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7631218Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7631631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7631999Z return self.act(input) 2025-09-07T07:17:51.7632127Z 2025-09-07T07:17:51.7632239Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7632634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7632991Z return mod(**inputs) 2025-09-07T07:17:51.7633383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7633806Z outputs = self.model.decoder( 2025-09-07T07:17:51.7634220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7634647Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7635008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7635418Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7635836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7636262Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7636407Z 2025-09-07T07:17:51.7636525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7636896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7637238Z return mod(**inputs) 2025-09-07T07:17:51.7637624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7638033Z outputs = self.model.decoder( 2025-09-07T07:17:51.7638454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7638869Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7639245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7639636Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7640053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7640494Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7640943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7641412Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7641592Z 2025-09-07T07:17:51.7641715Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7642110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7642458Z return mod(**inputs) 2025-09-07T07:17:51.7642879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7643304Z outputs = self.model.decoder( 2025-09-07T07:17:51.7643735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7644144Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7644521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7644891Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7645294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7645714Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7646130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7646533Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7646676Z 2025-09-07T07:17:51.7646779Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7647152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7647488Z return mod(**inputs) 2025-09-07T07:17:51.7647860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7648268Z outputs = self.model.decoder( 2025-09-07T07:17:51.7648669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7649061Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7649411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7649784Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7650223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7650646Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7651076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7651498Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7651665Z 2025-09-07T07:17:51.7651748Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7651968Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7652184Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7652418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7652788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7653144Z return mod(**inputs) 2025-09-07T07:17:51.7653519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7653916Z outputs = self.model.decoder( 2025-09-07T07:17:51.7654302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7654698Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7655055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7655430Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7655823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7656245Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7656667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7657095Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7657238Z 2025-09-07T07:17:51.7657352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7657720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7658076Z return mod(**inputs) 2025-09-07T07:17:51.7658449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7658845Z outputs = self.model.decoder( 2025-09-07T07:17:51.7659235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7659624Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7659984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7660384Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7660812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7661299Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7661498Z 2025-09-07T07:17:51.7661611Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7662001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7662359Z return mod(**inputs) 2025-09-07T07:17:51.7662761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7663185Z outputs = self.model.decoder( 2025-09-07T07:17:51.7663601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7664027Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7664408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7664829Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7665245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7665792Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7666236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7666619Z return self.act(input) 2025-09-07T07:17:51.7666746Z 2025-09-07T07:17:51.7666871Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7667262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7667644Z return mod(**inputs) 2025-09-07T07:17:51.7668040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7668466Z outputs = self.model.decoder( 2025-09-07T07:17:51.7668872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7669288Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7669670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7670065Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7670489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7670912Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7671068Z 2025-09-07T07:17:51.7671180Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7671574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7671924Z return mod(**inputs) 2025-09-07T07:17:51.7672337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7672754Z outputs = self.model.decoder( 2025-09-07T07:17:51.7673180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7673600Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7673974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7674357Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7674778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7675227Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7675673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7676139Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7676317Z 2025-09-07T07:17:51.7676430Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7676821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7677187Z return mod(**inputs) 2025-09-07T07:17:51.7677579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7677995Z outputs = self.model.decoder( 2025-09-07T07:17:51.7678409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7678827Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7679208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7679601Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7680042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7680500Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7680917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7681320Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7681457Z 2025-09-07T07:17:51.7681570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7681943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7682293Z return mod(**inputs) 2025-09-07T07:17:51.7682706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7683123Z outputs = self.model.decoder( 2025-09-07T07:17:51.7683530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7683945Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7684331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7684705Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7685105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7685517Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7685936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7686347Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7686492Z 2025-09-07T07:17:51.7686584Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7686820Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7687041Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7687287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7687680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7688018Z return mod(**inputs) 2025-09-07T07:17:51.7688393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7688800Z outputs = self.model.decoder( 2025-09-07T07:17:51.7689195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7689598Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7689959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7690335Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7690744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7691190Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7691620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7692024Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7692176Z 2025-09-07T07:17:51.7692284Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7692660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7693004Z return mod(**inputs) 2025-09-07T07:17:51.7693403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7693827Z outputs = self.model.decoder( 2025-09-07T07:17:51.7694225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7694643Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7695004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7695374Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7695773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7696221Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7696400Z 2025-09-07T07:17:51.7696516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7696886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7697230Z return mod(**inputs) 2025-09-07T07:17:51.7697609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7698006Z outputs = self.model.decoder( 2025-09-07T07:17:51.7698396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7698793Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7699147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7699516Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7699940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7700406Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7700825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7701214Z return self.act(input) 2025-09-07T07:17:51.7701343Z 2025-09-07T07:17:51.7701515Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7701906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7702271Z return mod(**inputs) 2025-09-07T07:17:51.7702692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7703119Z outputs = self.model.decoder( 2025-09-07T07:17:51.7703532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7703951Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7704329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7704731Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7705158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7705591Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7705816Z 2025-09-07T07:17:51.7705944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7706335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7706703Z return mod(**inputs) 2025-09-07T07:17:51.7707107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7707547Z outputs = self.model.decoder( 2025-09-07T07:17:51.7707961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7708380Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7708761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7709163Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7709605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7710041Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7710489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7710954Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7711133Z 2025-09-07T07:17:51.7711253Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7711643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7712008Z return mod(**inputs) 2025-09-07T07:17:51.7712428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7712854Z outputs = self.model.decoder( 2025-09-07T07:17:51.7713289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7713713Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7714090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7714529Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7714956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7715413Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7715855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7716303Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7716464Z 2025-09-07T07:17:51.7716578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7717009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7717360Z return mod(**inputs) 2025-09-07T07:17:51.7717757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7718174Z outputs = self.model.decoder( 2025-09-07T07:17:51.7718583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7719000Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7719381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7719908Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7720498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7720961Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7721413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7721844Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7722010Z 2025-09-07T07:17:51.7722098Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7722336Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7722570Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7722835Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7723219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7723575Z return mod(**inputs) 2025-09-07T07:17:51.7723971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7724392Z outputs = self.model.decoder( 2025-09-07T07:17:51.7724855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7725278Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7725659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7726058Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7726476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7726927Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7727382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7727824Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7727965Z 2025-09-07T07:17:51.7728082Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7728457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7728790Z return mod(**inputs) 2025-09-07T07:17:51.7729160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7729556Z outputs = self.model.decoder( 2025-09-07T07:17:51.7729967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7730376Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7730756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7731150Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7731581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7732049Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7732239Z 2025-09-07T07:17:51.7732352Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7732742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7733133Z return mod(**inputs) 2025-09-07T07:17:51.7733532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7733962Z outputs = self.model.decoder( 2025-09-07T07:17:51.7734351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7734747Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7735110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7735490Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7735888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7736334Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7736740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7737100Z return self.act(input) 2025-09-07T07:17:51.7737215Z 2025-09-07T07:17:51.7737330Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7737695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7738031Z return mod(**inputs) 2025-09-07T07:17:51.7738406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7738807Z outputs = self.model.decoder( 2025-09-07T07:17:51.7739196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7739617Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7739972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7740346Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7740745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7741141Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7741297Z 2025-09-07T07:17:51.7741409Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7741799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7742170Z return mod(**inputs) 2025-09-07T07:17:51.7742566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7742977Z outputs = self.model.decoder( 2025-09-07T07:17:51.7743395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7743811Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7744196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7744584Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7745011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7745462Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7745968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7746445Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7746625Z 2025-09-07T07:17:51.7746758Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7747152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7747502Z return mod(**inputs) 2025-09-07T07:17:51.7747921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7748349Z outputs = self.model.decoder( 2025-09-07T07:17:51.7748754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7749172Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7749561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7749957Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7750372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7750828Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7751276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7751707Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7751854Z 2025-09-07T07:17:51.7751975Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7752354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7752727Z return mod(**inputs) 2025-09-07T07:17:51.7753114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7753531Z outputs = self.model.decoder( 2025-09-07T07:17:51.7753943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7754354Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7754756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7755148Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7755568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7756008Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7756452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7756886Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7757038Z 2025-09-07T07:17:51.7757133Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7757385Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7757609Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7757867Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7758259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7758596Z return mod(**inputs) 2025-09-07T07:17:51.7758964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7759379Z outputs = self.model.decoder( 2025-09-07T07:17:51.7759790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7760197Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7760555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7760918Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7761322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7761785Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7762212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7762632Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7762817Z 2025-09-07T07:17:51.7762931Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7763322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7763674Z return mod(**inputs) 2025-09-07T07:17:51.7764045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7764434Z outputs = self.model.decoder( 2025-09-07T07:17:51.7764832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7765231Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7765595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7765967Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7766368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7766815Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7767001Z 2025-09-07T07:17:51.7767108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7767477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7767806Z return mod(**inputs) 2025-09-07T07:17:51.7768177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7768582Z outputs = self.model.decoder( 2025-09-07T07:17:51.7768970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7769388Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7769737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7770111Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7770506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7770942Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7771335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7771695Z return self.act(input) 2025-09-07T07:17:51.7771841Z 2025-09-07T07:17:51.7771953Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7772356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7772718Z return mod(**inputs) 2025-09-07T07:17:51.7773104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7773529Z outputs = self.model.decoder( 2025-09-07T07:17:51.7773987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7774404Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7774788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7775185Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7775611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7776038Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7776187Z 2025-09-07T07:17:51.7776335Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7776749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7777101Z return mod(**inputs) 2025-09-07T07:17:51.7777516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7777936Z outputs = self.model.decoder( 2025-09-07T07:17:51.7778350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7778758Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7779137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7779533Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7779956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7780409Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7780843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7781308Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7781493Z 2025-09-07T07:17:51.7781605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7781996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7782366Z return mod(**inputs) 2025-09-07T07:17:51.7782749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7783170Z outputs = self.model.decoder( 2025-09-07T07:17:51.7783579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7784022Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7784391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7784783Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7785207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7785660Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7786192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7786614Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7786770Z 2025-09-07T07:17:51.7786884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7787314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7787679Z return mod(**inputs) 2025-09-07T07:17:51.7788065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7788489Z outputs = self.model.decoder( 2025-09-07T07:17:51.7788897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7789322Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7789695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7790090Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7790490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7790919Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7791362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7791781Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7791927Z 2025-09-07T07:17:51.7792009Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7792232Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7792467Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7792719Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7793106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7793482Z return mod(**inputs) 2025-09-07T07:17:51.7793880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7794315Z outputs = self.model.decoder( 2025-09-07T07:17:51.7794735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7795128Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7795490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7795870Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7796297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7796740Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7797190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7797620Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7797766Z 2025-09-07T07:17:51.7797887Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7798283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7798650Z return mod(**inputs) 2025-09-07T07:17:51.7799070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7799492Z outputs = self.model.decoder( 2025-09-07T07:17:51.7799906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7800322Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7800695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7801102Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7801532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7802022Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7802207Z 2025-09-07T07:17:51.7802322Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7802711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7803063Z return mod(**inputs) 2025-09-07T07:17:51.7803459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7803881Z outputs = self.model.decoder( 2025-09-07T07:17:51.7804284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7804697Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7805075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7805461Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7805872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7806358Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7806785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7807169Z return self.act(input) 2025-09-07T07:17:51.7807289Z 2025-09-07T07:17:51.7807427Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7807809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7808164Z return mod(**inputs) 2025-09-07T07:17:51.7808555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7808974Z outputs = self.model.decoder( 2025-09-07T07:17:51.7809382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7809793Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7810170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7810559Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7810979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7811395Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7811552Z 2025-09-07T07:17:51.7811666Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7812060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7812435Z return mod(**inputs) 2025-09-07T07:17:51.7812830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7813247Z outputs = self.model.decoder( 2025-09-07T07:17:51.7813657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7814105Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7814484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7814881Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7815294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7815745Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7816193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7816665Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7816845Z 2025-09-07T07:17:51.7816990Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7817383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7817756Z return mod(**inputs) 2025-09-07T07:17:51.7818151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7818571Z outputs = self.model.decoder( 2025-09-07T07:17:51.7818978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7819404Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7819981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7820394Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7820819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7821264Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7821770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7822204Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7822351Z 2025-09-07T07:17:51.7822473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7822893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7823254Z return mod(**inputs) 2025-09-07T07:17:51.7823650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7824075Z outputs = self.model.decoder( 2025-09-07T07:17:51.7824489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7824905Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7825284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7825678Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7826157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7826618Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7827068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7827513Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7827679Z 2025-09-07T07:17:51.7827778Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7828013Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7828234Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7828492Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7828882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7829244Z return mod(**inputs) 2025-09-07T07:17:51.7829681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7830112Z outputs = self.model.decoder( 2025-09-07T07:17:51.7830537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7830969Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7831360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7831756Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7832191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7832686Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7833143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7833587Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7833739Z 2025-09-07T07:17:51.7833855Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7834259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7834623Z return mod(**inputs) 2025-09-07T07:17:51.7835033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7835464Z outputs = self.model.decoder( 2025-09-07T07:17:51.7835890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7836327Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7836724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7837157Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7837588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7838076Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7838300Z 2025-09-07T07:17:51.7838419Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7838820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7839186Z return mod(**inputs) 2025-09-07T07:17:51.7839576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7839993Z outputs = self.model.decoder( 2025-09-07T07:17:51.7840405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7840826Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7841200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7841597Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7842019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7842488Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7842910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7843288Z return self.act(input) 2025-09-07T07:17:51.7843417Z 2025-09-07T07:17:51.7843527Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7843915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7844282Z return mod(**inputs) 2025-09-07T07:17:51.7844673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7845155Z outputs = self.model.decoder( 2025-09-07T07:17:51.7845563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7846001Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7846378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7846763Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7847183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7847627Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7847795Z 2025-09-07T07:17:51.7847915Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7848311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7848677Z return mod(**inputs) 2025-09-07T07:17:51.7849076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7849514Z outputs = self.model.decoder( 2025-09-07T07:17:51.7849934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7850364Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7850737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7851132Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7851559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7852030Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7852480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7852919Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7853096Z 2025-09-07T07:17:51.7853200Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7853604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7853967Z return mod(**inputs) 2025-09-07T07:17:51.7854329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7854731Z outputs = self.model.decoder( 2025-09-07T07:17:51.7855119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7855515Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7855868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7856238Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7856637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7857059Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7857479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7857875Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7858021Z 2025-09-07T07:17:51.7858126Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7858510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7858849Z return mod(**inputs) 2025-09-07T07:17:51.7859231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7859645Z outputs = self.model.decoder( 2025-09-07T07:17:51.7860036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7860431Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7860789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7861161Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7861591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7862041Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7862483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7862934Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7863088Z 2025-09-07T07:17:51.7863177Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7863410Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7863637Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7863895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7864293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7864658Z return mod(**inputs) 2025-09-07T07:17:51.7865052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7865522Z outputs = self.model.decoder( 2025-09-07T07:17:51.7866018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7866450Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7866835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7867263Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7867691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7868187Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7868627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7869076Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7869231Z 2025-09-07T07:17:51.7869344Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7869735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7870099Z return mod(**inputs) 2025-09-07T07:17:51.7870492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7870926Z outputs = self.model.decoder( 2025-09-07T07:17:51.7871338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7871756Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7872136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7872539Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7872971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7873474Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7873662Z 2025-09-07T07:17:51.7873780Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7874162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7874513Z return mod(**inputs) 2025-09-07T07:17:51.7874939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7875361Z outputs = self.model.decoder( 2025-09-07T07:17:51.7875765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7876179Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7876556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7876947Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7877368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7877856Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7878281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7878657Z return self.act(input) 2025-09-07T07:17:51.7878779Z 2025-09-07T07:17:51.7878900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7879287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7879651Z return mod(**inputs) 2025-09-07T07:17:51.7880048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7880465Z outputs = self.model.decoder( 2025-09-07T07:17:51.7880870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7881281Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7881660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7882057Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7882505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7882938Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7883086Z 2025-09-07T07:17:51.7883214Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7883610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7883962Z return mod(**inputs) 2025-09-07T07:17:51.7884355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7884771Z outputs = self.model.decoder( 2025-09-07T07:17:51.7885194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7885616Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7885998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7886393Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7886814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7887261Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7887711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7888175Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7888352Z 2025-09-07T07:17:51.7888470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7888856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7889213Z return mod(**inputs) 2025-09-07T07:17:51.7889608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7890054Z outputs = self.model.decoder( 2025-09-07T07:17:51.7890466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7890862Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7891218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7891583Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7891981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7892398Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7892821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7893255Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7893397Z 2025-09-07T07:17:51.7893517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7893889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7894220Z return mod(**inputs) 2025-09-07T07:17:51.7894592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7894989Z outputs = self.model.decoder( 2025-09-07T07:17:51.7895378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7895765Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7896125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7896496Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7896913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7897338Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7897749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7898171Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7898324Z 2025-09-07T07:17:51.7898407Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7898630Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7898839Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7899080Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7899447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7899781Z return mod(**inputs) 2025-09-07T07:17:51.7900158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7900551Z outputs = self.model.decoder( 2025-09-07T07:17:51.7900945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7901345Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7901706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7902076Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7902474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7902901Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7903320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7903748Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7903894Z 2025-09-07T07:17:51.7904013Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7904433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7904810Z return mod(**inputs) 2025-09-07T07:17:51.7905218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7905654Z outputs = self.model.decoder( 2025-09-07T07:17:51.7906163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7906617Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7907007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7907451Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7907868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7908379Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7908583Z 2025-09-07T07:17:51.7908700Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7909106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7909478Z return mod(**inputs) 2025-09-07T07:17:51.7909878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7910322Z outputs = self.model.decoder( 2025-09-07T07:17:51.7910756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7911198Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7911587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7912014Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7912461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7912958Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7913417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7913803Z return self.act(input) 2025-09-07T07:17:51.7913938Z 2025-09-07T07:17:51.7914052Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7914453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7914831Z return mod(**inputs) 2025-09-07T07:17:51.7915240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7915685Z outputs = self.model.decoder( 2025-09-07T07:17:51.7916113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7916543Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7916936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7917341Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7917776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7918216Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7918377Z 2025-09-07T07:17:51.7918493Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7918897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7919256Z return mod(**inputs) 2025-09-07T07:17:51.7919798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7920301Z outputs = self.model.decoder( 2025-09-07T07:17:51.7920730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7921165Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7921548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7921955Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7922398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7922863Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7923361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7923842Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7924037Z 2025-09-07T07:17:51.7924152Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7924532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7924864Z return mod(**inputs) 2025-09-07T07:17:51.7925227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7925623Z outputs = self.model.decoder( 2025-09-07T07:17:51.7926003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7926397Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7926751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7927121Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7927553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7927967Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7928410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7928821Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7928959Z 2025-09-07T07:17:51.7929065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7929436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7929778Z return mod(**inputs) 2025-09-07T07:17:51.7930150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7930546Z outputs = self.model.decoder( 2025-09-07T07:17:51.7930945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7931346Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7931695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7932057Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7932439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7932852Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7933272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7933681Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7933825Z 2025-09-07T07:17:51.7933914Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7934128Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7934354Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7934634Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7935023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7935356Z return mod(**inputs) 2025-09-07T07:17:51.7935750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7936179Z outputs = self.model.decoder( 2025-09-07T07:17:51.7936593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7937013Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7937401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7937822Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7938227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7938639Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7939040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7939447Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7939595Z 2025-09-07T07:17:51.7939702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7940086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7940441Z return mod(**inputs) 2025-09-07T07:17:51.7940824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7941243Z outputs = self.model.decoder( 2025-09-07T07:17:51.7941730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7942167Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7942547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7942943Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7943400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7943862Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7944060Z 2025-09-07T07:17:51.7944173Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7944559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7944927Z return mod(**inputs) 2025-09-07T07:17:51.7945324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7945803Z outputs = self.model.decoder( 2025-09-07T07:17:51.7946239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7946668Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7947063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7947462Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7947900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7948383Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7948811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7949187Z return self.act(input) 2025-09-07T07:17:51.7949308Z 2025-09-07T07:17:51.7949422Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7949841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7950193Z return mod(**inputs) 2025-09-07T07:17:51.7950587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7950996Z outputs = self.model.decoder( 2025-09-07T07:17:51.7951403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7951812Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7952157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7952514Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7952912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7953309Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7953454Z 2025-09-07T07:17:51.7953559Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7953930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7954274Z return mod(**inputs) 2025-09-07T07:17:51.7954629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7955012Z outputs = self.model.decoder( 2025-09-07T07:17:51.7955391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7955783Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7956138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7956508Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7956933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7957352Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7957785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-09-07T07:17:51.7958207Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:17:51.7958381Z 2025-09-07T07:17:51.7958482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7958844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7959171Z return mod(**inputs) 2025-09-07T07:17:51.7959540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7959918Z outputs = self.model.decoder( 2025-09-07T07:17:51.7960308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7960729Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7961106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7961492Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7961928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7962046Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7962334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-09-07T07:17:51.7962423Z key_states = self.k_proj(current_states) 2025-09-07T07:17:51.7962437Z 2025-09-07T07:17:51.7962548Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7962763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7962864Z return mod(**inputs) 2025-09-07T07:17:51.7963137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7963224Z outputs = self.model.decoder( 2025-09-07T07:17:51.7963499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7963575Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7963825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7963908Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7964188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7964322Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7964581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-09-07T07:17:51.7964676Z value_states = self.v_proj(current_states) 2025-09-07T07:17:51.7964679Z 2025-09-07T07:17:51.7964761Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7964851Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7964932Z cudagraph partition due to non gpu ops 2025-09-07T07:17:51.7965044Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7965270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7965339Z return mod(**inputs) 2025-09-07T07:17:51.7965621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7965700Z outputs = self.model.decoder( 2025-09-07T07:17:51.7965998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7966080Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7966317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7966428Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7966702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-09-07T07:17:51.7966813Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:17:51.7967081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-09-07T07:17:51.7967169Z attn_output = self.out_proj(attn_output) 2025-09-07T07:17:51.7967175Z 2025-09-07T07:17:51.7967292Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7967506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7967587Z return mod(**inputs) 2025-09-07T07:17:51.7967861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7967936Z outputs = self.model.decoder( 2025-09-07T07:17:51.7968217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7968292Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7968535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7968619Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7968894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7969026Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7969030Z 2025-09-07T07:17:51.7969141Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7969388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7969457Z return mod(**inputs) 2025-09-07T07:17:51.7969736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7969814Z outputs = self.model.decoder( 2025-09-07T07:17:51.7970084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7970168Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7970404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7970516Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7970788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-09-07T07:17:51.7970925Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:17:51.7971155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:17:51.7971233Z return self.act(input) 2025-09-07T07:17:51.7971237Z 2025-09-07T07:17:51.7971355Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7971567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7971645Z return mod(**inputs) 2025-09-07T07:17:51.7971919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-09-07T07:17:51.7971999Z outputs = self.model.decoder( 2025-09-07T07:17:51.7972280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-09-07T07:17:51.7972357Z layer_outputs = decoder_layer( 2025-09-07T07:17:51.7972632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:17:51.7972720Z return super().__call__(*args, **kwargs) 2025-09-07T07:17:51.7973006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-09-07T07:17:51.7973104Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:17:51.7973108Z 2025-09-07T07:17:51.7973217Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7973440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7973511Z return mod(**inputs) 2025-09-07T07:17:51.7973790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 839, in forward 2025-09-07T07:17:51.7973894Z logits = self.output_projection(outputs[0]) 2025-09-07T07:17:51.7973897Z 2025-09-07T07:17:51.7974008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:17:51.7974230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:17:51.7974298Z return mod(**inputs) 2025-09-07T07:17:51.7974581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 844, in forward 2025-09-07T07:17:51.7974744Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:17:51.7974748Z 2025-09-07T07:18:03.2696735Z Compilation time (from dynamo_timed): 18.008004511 2025-09-07T07:18:03.2743641Z pass 2025-09-07T07:18:03.2744036Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:18:03.2744931Z TIMING: _recursive_pre_grad_passes:0.00827 _recursive_joint_graph_passes:0.4759 _recursive_post_grad_passes:0.07702 async_compile.wait:0.82484 code_gen:10.48364 inductor_compile:11.73292 backend_compile:15.27858 gc:0.00139 entire_frame_compile:18.008 total_wall_time:18.008 2025-09-07T07:18:03.2746316Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:14341 | FakeTensor.__torch_dispatch__:4316 | ProxyTorchDispatchMode.__torch_dispatch__:5467 2025-09-07T07:18:03.2746869Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-09-07T07:18:05.8597607Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:18:05.8598537Z import pynvml # type: ignore[import] 2025-09-07T07:18:08.6341041Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:18:08.6342200Z from pkg_resources import resource_filename 2025-09-07T07:18:09.2859558Z 2025-09-07T07:18:15.6798191Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:18:15.6798491Z loading model: 0it [00:06, ?it/s] 2025-09-07T07:18:15.6818926Z cpu eval XGLMForCausalLM 2025-09-07T07:18:16.0846663Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-09-07T07:18:16.1895181Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:18:16.7359151Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:18:17.2380762Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:18:32.0399153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0400017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0400405Z return mod(**inputs) 2025-09-07T07:18:32.0400841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0401385Z outputs = self.model( 2025-09-07T07:18:32.0401806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0402313Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0402713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0403129Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0403574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0404124Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0404599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0405078Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0405277Z 2025-09-07T07:18:32.0405403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0405814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0406182Z return mod(**inputs) 2025-09-07T07:18:32.0406597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0407026Z outputs = self.model( 2025-09-07T07:18:32.0407439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0407873Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0408284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0408764Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0409192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0409648Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0410207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0410637Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0410788Z 2025-09-07T07:18:32.0410906Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0411305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0411732Z return mod(**inputs) 2025-09-07T07:18:32.0412131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0412558Z outputs = self.model( 2025-09-07T07:18:32.0412951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0413376Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0413775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0414200Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0414623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0415078Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0415534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0416002Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0416189Z 2025-09-07T07:18:32.0416359Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0416759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0417143Z return mod(**inputs) 2025-09-07T07:18:32.0417568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0418003Z outputs = self.model( 2025-09-07T07:18:32.0418396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0418818Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0419206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0419763Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0420194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0420645Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0421087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0421588Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0421814Z 2025-09-07T07:18:32.0421933Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0422335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0422694Z return mod(**inputs) 2025-09-07T07:18:32.0423115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0423538Z outputs = self.model( 2025-09-07T07:18:32.0423937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0424402Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0424787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0425190Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0425623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0426411Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0426870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0427289Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0427450Z 2025-09-07T07:18:32.0427564Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0428001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0428357Z return mod(**inputs) 2025-09-07T07:18:32.0428746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0429157Z outputs = self.model( 2025-09-07T07:18:32.0429544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0429956Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0430335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0430718Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0431135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0431579Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0432015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0432491Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0432648Z 2025-09-07T07:18:32.0432754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0433149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0433481Z return mod(**inputs) 2025-09-07T07:18:32.0433842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0434219Z outputs = self.model( 2025-09-07T07:18:32.0434587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0434975Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0435331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0435708Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0436084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0436486Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0436888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0437361Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0437565Z 2025-09-07T07:18:32.0437678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0438040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0438382Z return mod(**inputs) 2025-09-07T07:18:32.0438748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0439151Z outputs = self.model( 2025-09-07T07:18:32.0439537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0439944Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0440300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0440665Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0441079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0441489Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0441887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0442330Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0442485Z 2025-09-07T07:18:32.0442597Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0442979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0443315Z return mod(**inputs) 2025-09-07T07:18:32.0443672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0444070Z outputs = self.model( 2025-09-07T07:18:32.0444460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0444851Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0445206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0445579Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0445980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0446480Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0446680Z 2025-09-07T07:18:32.0446794Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0447154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0447506Z return mod(**inputs) 2025-09-07T07:18:32.0447878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0448266Z outputs = self.model( 2025-09-07T07:18:32.0448619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0449009Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0449364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0449734Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0450129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0450567Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0450965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0451320Z return self.act(input) 2025-09-07T07:18:32.0451439Z 2025-09-07T07:18:32.0451560Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0451950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0452291Z return mod(**inputs) 2025-09-07T07:18:32.0452673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0453081Z outputs = self.model( 2025-09-07T07:18:32.0453462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0453869Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0454225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0454591Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0454986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0455383Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0455524Z 2025-09-07T07:18:32.0455628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0456009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0456360Z return mod(**inputs) 2025-09-07T07:18:32.0456770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0457173Z outputs = self.model( 2025-09-07T07:18:32.0457550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0457966Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0458330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0458703Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0459090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0459516Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0459954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0460411Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0460589Z 2025-09-07T07:18:32.0460707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0461149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0461515Z return mod(**inputs) 2025-09-07T07:18:32.0461932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0462365Z outputs = self.model( 2025-09-07T07:18:32.0462753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0463177Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0463570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0463974Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0464394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0464841Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0465308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0465961Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0466116Z 2025-09-07T07:18:32.0466243Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0466659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0467038Z return mod(**inputs) 2025-09-07T07:18:32.0467448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0467867Z outputs = self.model( 2025-09-07T07:18:32.0468270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0468698Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0469103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0469543Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0469978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0470435Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0470879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0471350Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0471541Z 2025-09-07T07:18:32.0471661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0472068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0472464Z return mod(**inputs) 2025-09-07T07:18:32.0472859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0473279Z outputs = self.model( 2025-09-07T07:18:32.0473683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0474115Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0474499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0474907Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0475335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0475792Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0476243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0476736Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0476989Z 2025-09-07T07:18:32.0477108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0477507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0477884Z return mod(**inputs) 2025-09-07T07:18:32.0478305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0478721Z outputs = self.model( 2025-09-07T07:18:32.0479107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0479519Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0479895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0480282Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0480701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0481146Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0481583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0482010Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0482163Z 2025-09-07T07:18:32.0482273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0482663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0483015Z return mod(**inputs) 2025-09-07T07:18:32.0483403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0483815Z outputs = self.model( 2025-09-07T07:18:32.0484194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0484670Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0485050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0485442Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0485848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0486291Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0486727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0487169Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0487331Z 2025-09-07T07:18:32.0487473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0487853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0488209Z return mod(**inputs) 2025-09-07T07:18:32.0488592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0489005Z outputs = self.model( 2025-09-07T07:18:32.0489397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0489797Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0490173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0490562Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0490974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0491406Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0491870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0492346Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0492538Z 2025-09-07T07:18:32.0492657Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0493086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0493447Z return mod(**inputs) 2025-09-07T07:18:32.0493857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0494289Z outputs = self.model( 2025-09-07T07:18:32.0494698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0495134Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0495528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0495936Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0496374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0496835Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0497289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0497734Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0497899Z 2025-09-07T07:18:32.0498016Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0498422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0498809Z return mod(**inputs) 2025-09-07T07:18:32.0499207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0499637Z outputs = self.model( 2025-09-07T07:18:32.0500059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0500491Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0500886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0501291Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0501721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0502212Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0502404Z 2025-09-07T07:18:32.0502531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0502940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0503337Z return mod(**inputs) 2025-09-07T07:18:32.0503739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0504173Z outputs = self.model( 2025-09-07T07:18:32.0504586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0505011Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0505412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0505930Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0506367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0506852Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0507280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0507663Z return self.act(input) 2025-09-07T07:18:32.0507823Z 2025-09-07T07:18:32.0507942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0508344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0508712Z return mod(**inputs) 2025-09-07T07:18:32.0509127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0509553Z outputs = self.model( 2025-09-07T07:18:32.0509948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0510382Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0510773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0511179Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0511612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0512033Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0512187Z 2025-09-07T07:18:32.0512311Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0512707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0513068Z return mod(**inputs) 2025-09-07T07:18:32.0513463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0513879Z outputs = self.model( 2025-09-07T07:18:32.0514270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0514679Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0515058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0515447Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0515883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0516316Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0516752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0517203Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0517381Z 2025-09-07T07:18:32.0517499Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0517881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0518215Z return mod(**inputs) 2025-09-07T07:18:32.0519499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0520067Z outputs = self.model( 2025-09-07T07:18:32.0520438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0520829Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0521217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0521608Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0522007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0522489Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0522931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0523359Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0523513Z 2025-09-07T07:18:32.0523626Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0524105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0524434Z return mod(**inputs) 2025-09-07T07:18:32.0524828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0525225Z outputs = self.model( 2025-09-07T07:18:32.0525614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0526026Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0526380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0526771Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0527189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0527609Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0528016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0528445Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0528622Z 2025-09-07T07:18:32.0528729Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0529098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0529429Z return mod(**inputs) 2025-09-07T07:18:32.0529788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0530175Z outputs = self.model( 2025-09-07T07:18:32.0530540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0530934Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0531291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0531688Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0532103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0532543Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0532976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0533459Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0533669Z 2025-09-07T07:18:32.0533786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0534149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0534529Z return mod(**inputs) 2025-09-07T07:18:32.0534915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0535315Z outputs = self.model( 2025-09-07T07:18:32.0535699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0536105Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0536483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0536879Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0537287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0537722Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0538155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0538608Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0538769Z 2025-09-07T07:18:32.0538888Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0539274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0539660Z return mod(**inputs) 2025-09-07T07:18:32.0540048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0540465Z outputs = self.model( 2025-09-07T07:18:32.0540842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0541264Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0541640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0542033Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0542446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0542879Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0543308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0543742Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0543902Z 2025-09-07T07:18:32.0544019Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0544398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0544750Z return mod(**inputs) 2025-09-07T07:18:32.0545131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0545539Z outputs = self.model( 2025-09-07T07:18:32.0545987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0546424Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0546811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0547215Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0547640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0548093Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0548531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0549006Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0549220Z 2025-09-07T07:18:32.0549326Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0549701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0550032Z return mod(**inputs) 2025-09-07T07:18:32.0550399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0550787Z outputs = self.model( 2025-09-07T07:18:32.0551157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0551546Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0551911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0552304Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0552717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0553157Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0553618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0554034Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0554181Z 2025-09-07T07:18:32.0554287Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0554666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0555001Z return mod(**inputs) 2025-09-07T07:18:32.0555359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0555749Z outputs = self.model( 2025-09-07T07:18:32.0556116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0556516Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0556876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0557242Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0557638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0558083Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0558261Z 2025-09-07T07:18:32.0558373Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0558742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0559070Z return mod(**inputs) 2025-09-07T07:18:32.0559440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0559818Z outputs = self.model( 2025-09-07T07:18:32.0560173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0560546Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0560914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0561271Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0561653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0562074Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0562459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0562805Z return self.act(input) 2025-09-07T07:18:32.0562926Z 2025-09-07T07:18:32.0563030Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0563398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0563760Z return mod(**inputs) 2025-09-07T07:18:32.0564161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0564553Z outputs = self.model( 2025-09-07T07:18:32.0564923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0565314Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0565702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0566099Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0566512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0566933Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0567082Z 2025-09-07T07:18:32.0567211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0567570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0567920Z return mod(**inputs) 2025-09-07T07:18:32.0568287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0568673Z outputs = self.model( 2025-09-07T07:18:32.0569054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0569441Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0569801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0570162Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0570558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.0570958Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.0571098Z 2025-09-07T07:18:32.0571202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0571569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0571905Z return mod(**inputs) 2025-09-07T07:18:32.0572291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0572701Z outputs = self.model( 2025-09-07T07:18:32.0573098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0573518Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0573900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0574282Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0574701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0575155Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0575592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0576047Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0576225Z 2025-09-07T07:18:32.0576337Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0576726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0577077Z return mod(**inputs) 2025-09-07T07:18:32.0577459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0577868Z outputs = self.model( 2025-09-07T07:18:32.0578273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0578681Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0579061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0579453Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0579869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0580321Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0580768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0581203Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0581352Z 2025-09-07T07:18:32.0581474Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0581866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0582225Z return mod(**inputs) 2025-09-07T07:18:32.0582636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0583055Z outputs = self.model( 2025-09-07T07:18:32.0583475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0583908Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0584297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0584704Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0585127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0585582Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0586117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0586600Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0586784Z 2025-09-07T07:18:32.0586906Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0587312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0587672Z return mod(**inputs) 2025-09-07T07:18:32.0588064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0588480Z outputs = self.model( 2025-09-07T07:18:32.0588873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0589287Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0589664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0590065Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0590485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0590970Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0591431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0591917Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0592126Z 2025-09-07T07:18:32.0592238Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0592626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0592986Z return mod(**inputs) 2025-09-07T07:18:32.0593384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0593815Z outputs = self.model( 2025-09-07T07:18:32.0594202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0594609Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0594963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0595337Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0595730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0596143Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0596553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0596947Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0597099Z 2025-09-07T07:18:32.0597206Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0597590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0597925Z return mod(**inputs) 2025-09-07T07:18:32.0598280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0598682Z outputs = self.model( 2025-09-07T07:18:32.0599045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0599434Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0599789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0600154Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0600546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0600967Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0601380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0601796Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0601951Z 2025-09-07T07:18:32.0602556Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0602925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0603257Z return mod(**inputs) 2025-09-07T07:18:32.0603623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0604000Z outputs = self.model( 2025-09-07T07:18:32.0604368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0604762Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0605121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0605511Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0605897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0606312Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0606720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0607161Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0607342Z 2025-09-07T07:18:32.0607453Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0607822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0608192Z return mod(**inputs) 2025-09-07T07:18:32.0608575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0608991Z outputs = self.model( 2025-09-07T07:18:32.0609350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0609742Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0610101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0610471Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0610859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0611262Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0611671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0612078Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0612213Z 2025-09-07T07:18:32.0612339Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0612709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0613036Z return mod(**inputs) 2025-09-07T07:18:32.0613420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0613796Z outputs = self.model( 2025-09-07T07:18:32.0614148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0614524Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0614884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0615255Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0615646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0616084Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0616272Z 2025-09-07T07:18:32.0616374Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0616731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0617062Z return mod(**inputs) 2025-09-07T07:18:32.0617423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0617808Z outputs = self.model( 2025-09-07T07:18:32.0618163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0618551Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0618912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0619282Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0619862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0620313Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0620716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0621073Z return self.act(input) 2025-09-07T07:18:32.0621189Z 2025-09-07T07:18:32.0621304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0621692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0622043Z return mod(**inputs) 2025-09-07T07:18:32.0622428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0622884Z outputs = self.model( 2025-09-07T07:18:32.0623263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0623680Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0624062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0624465Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0624886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0625300Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0625458Z 2025-09-07T07:18:32.0625571Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0626027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0626396Z return mod(**inputs) 2025-09-07T07:18:32.0626792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0627248Z outputs = self.model( 2025-09-07T07:18:32.0627606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0628088Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0628525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0628927Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0629362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0629816Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0630266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0630738Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0630921Z 2025-09-07T07:18:32.0631038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0631447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0631810Z return mod(**inputs) 2025-09-07T07:18:32.0632208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0632630Z outputs = self.model( 2025-09-07T07:18:32.0633020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0633440Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0633833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0634242Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0634662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0635148Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0635602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0636039Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0636200Z 2025-09-07T07:18:32.0636318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0636701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0637053Z return mod(**inputs) 2025-09-07T07:18:32.0637436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0637839Z outputs = self.model( 2025-09-07T07:18:32.0638235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0638646Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0639023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0639413Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0639829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0640260Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0640695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0641152Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0641327Z 2025-09-07T07:18:32.0641445Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0641835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0642199Z return mod(**inputs) 2025-09-07T07:18:32.0642603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0643020Z outputs = self.model( 2025-09-07T07:18:32.0643403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0643788Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0644148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0644516Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0644911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0645331Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0645736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0646193Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0646393Z 2025-09-07T07:18:32.0646499Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0646869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0647199Z return mod(**inputs) 2025-09-07T07:18:32.0647557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0647939Z outputs = self.model( 2025-09-07T07:18:32.0648307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0648700Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0649056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0649428Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0649849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0650269Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0650683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0651084Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0651237Z 2025-09-07T07:18:32.0651343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0651719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0652058Z return mod(**inputs) 2025-09-07T07:18:32.0652424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0652821Z outputs = self.model( 2025-09-07T07:18:32.0653188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0653583Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0653942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0654307Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0654699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0655115Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0655532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0655943Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0656096Z 2025-09-07T07:18:32.0656198Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0656585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0656931Z return mod(**inputs) 2025-09-07T07:18:32.0657281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0657681Z outputs = self.model( 2025-09-07T07:18:32.0658040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0658432Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0658781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0659141Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0659517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0659929Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0660342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0660792Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0660975Z 2025-09-07T07:18:32.0661089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0661448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0661801Z return mod(**inputs) 2025-09-07T07:18:32.0662192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0662614Z outputs = self.model( 2025-09-07T07:18:32.0662991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0663405Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0663783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0664199Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0664616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0665049Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0665484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0665987Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0666141Z 2025-09-07T07:18:32.0666267Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0666671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0667058Z return mod(**inputs) 2025-09-07T07:18:32.0667453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0667852Z outputs = self.model( 2025-09-07T07:18:32.0668221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0668601Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0668965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0669334Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0669730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0670170Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0670346Z 2025-09-07T07:18:32.0670455Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0670822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0671203Z return mod(**inputs) 2025-09-07T07:18:32.0671599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0671984Z outputs = self.model( 2025-09-07T07:18:32.0672383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0672797Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0673175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0673569Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0673989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0674463Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0674876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0675229Z return self.act(input) 2025-09-07T07:18:32.0675341Z 2025-09-07T07:18:32.0675451Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0675818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0676152Z return mod(**inputs) 2025-09-07T07:18:32.0676518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0676902Z outputs = self.model( 2025-09-07T07:18:32.0677259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0677648Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0678009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0678377Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0678790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0679188Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0679336Z 2025-09-07T07:18:32.0679445Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0679815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0680151Z return mod(**inputs) 2025-09-07T07:18:32.0680517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0680899Z outputs = self.model( 2025-09-07T07:18:32.0681265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0681696Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0682078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0682462Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0682878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.0683300Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.0683450Z 2025-09-07T07:18:32.0683567Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0683965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0684287Z return mod(**inputs) 2025-09-07T07:18:32.0684650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0685033Z outputs = self.model( 2025-09-07T07:18:32.0685399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0685807Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0686184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0686574Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0687010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0687455Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0687888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0688324Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0688502Z 2025-09-07T07:18:32.0688606Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0688984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0689334Z return mod(**inputs) 2025-09-07T07:18:32.0689719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0690105Z outputs = self.model( 2025-09-07T07:18:32.0690495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0690908Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0691289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0691663Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0692052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0692471Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0692908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0693341Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0693494Z 2025-09-07T07:18:32.0693605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0693993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0694343Z return mod(**inputs) 2025-09-07T07:18:32.0694725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0695122Z outputs = self.model( 2025-09-07T07:18:32.0695508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0695916Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0696320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0696709Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0697133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0697577Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0698020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0698474Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0698656Z 2025-09-07T07:18:32.0698767Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0699158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0699511Z return mod(**inputs) 2025-09-07T07:18:32.0699897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0700310Z outputs = self.model( 2025-09-07T07:18:32.0700702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0701114Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0701509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0701912Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0702331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0702783Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0703233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0703724Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0703932Z 2025-09-07T07:18:32.0704051Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0704442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0704803Z return mod(**inputs) 2025-09-07T07:18:32.0705200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0705621Z outputs = self.model( 2025-09-07T07:18:32.0706079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0706507Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0706898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0707304Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0707737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0708183Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0708652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0709092Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0709251Z 2025-09-07T07:18:32.0709378Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0709792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0710203Z return mod(**inputs) 2025-09-07T07:18:32.0710611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0711036Z outputs = self.model( 2025-09-07T07:18:32.0711434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0711878Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0712278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0712692Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0713122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0713590Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0714038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0714500Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0714671Z 2025-09-07T07:18:32.0714784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0715181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0715558Z return mod(**inputs) 2025-09-07T07:18:32.0715979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0716360Z outputs = self.model( 2025-09-07T07:18:32.0716720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0717115Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0717470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0717844Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0718251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0718675Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0719094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0719541Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0719888Z 2025-09-07T07:18:32.0720004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0720383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0720710Z return mod(**inputs) 2025-09-07T07:18:32.0721069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0721436Z outputs = self.model( 2025-09-07T07:18:32.0721801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0722184Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0722542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0722905Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0723302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0723773Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0724177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0724568Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0724704Z 2025-09-07T07:18:32.0724808Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0725171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0725504Z return mod(**inputs) 2025-09-07T07:18:32.0725872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0726284Z outputs = self.model( 2025-09-07T07:18:32.0726646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0727047Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0727394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0727748Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0728124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0728549Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0728727Z 2025-09-07T07:18:32.0728829Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0729193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0729520Z return mod(**inputs) 2025-09-07T07:18:32.0729879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0730258Z outputs = self.model( 2025-09-07T07:18:32.0730659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0731036Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0731417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0731813Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0732238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0732714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0733148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0733533Z return self.act(input) 2025-09-07T07:18:32.0733665Z 2025-09-07T07:18:32.0733782Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0734179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0734510Z return mod(**inputs) 2025-09-07T07:18:32.0734907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0735322Z outputs = self.model( 2025-09-07T07:18:32.0735721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0736145Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0736537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0736937Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0737369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0737808Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0737986Z 2025-09-07T07:18:32.0738109Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0738510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0738862Z return mod(**inputs) 2025-09-07T07:18:32.0739256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0739668Z outputs = self.model( 2025-09-07T07:18:32.0740059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0740469Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0740858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0741290Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0741699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0742115Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0742521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0742973Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0743158Z 2025-09-07T07:18:32.0743271Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0743661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0744007Z return mod(**inputs) 2025-09-07T07:18:32.0744380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0744782Z outputs = self.model( 2025-09-07T07:18:32.0745165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0745591Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0746052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0746482Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0746913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0747356Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0747770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0748162Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0748307Z 2025-09-07T07:18:32.0748416Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0748790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0749128Z return mod(**inputs) 2025-09-07T07:18:32.0749491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0749870Z outputs = self.model( 2025-09-07T07:18:32.0750238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0750631Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0750991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0751413Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0751809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0752231Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0752649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0753103Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0753271Z 2025-09-07T07:18:32.0753377Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0753754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0754092Z return mod(**inputs) 2025-09-07T07:18:32.0754455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0754840Z outputs = self.model( 2025-09-07T07:18:32.0755205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0755631Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0755983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0756371Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0756760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0757174Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0757589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0758055Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0758242Z 2025-09-07T07:18:32.0758349Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0758701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0759029Z return mod(**inputs) 2025-09-07T07:18:32.0759388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0759765Z outputs = self.model( 2025-09-07T07:18:32.0760144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0760517Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0760882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0761243Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0761621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0762016Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0762416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0762809Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0762951Z 2025-09-07T07:18:32.0763061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0763422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0763738Z return mod(**inputs) 2025-09-07T07:18:32.0764092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0764462Z outputs = self.model( 2025-09-07T07:18:32.0764812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0765193Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0765531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0765894Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0766288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0766729Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0767181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0767629Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0767804Z 2025-09-07T07:18:32.0767909Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0768285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0768623Z return mod(**inputs) 2025-09-07T07:18:32.0768985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0769364Z outputs = self.model( 2025-09-07T07:18:32.0769718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0770112Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0770453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0770817Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0771211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0771627Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0772040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0772498Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0772698Z 2025-09-07T07:18:32.0772808Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0773194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0773545Z return mod(**inputs) 2025-09-07T07:18:32.0773951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0774332Z outputs = self.model( 2025-09-07T07:18:32.0774713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0775111Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0775488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0775871Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0776289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0776737Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0777183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0777607Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0777759Z 2025-09-07T07:18:32.0777870Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0778255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0778603Z return mod(**inputs) 2025-09-07T07:18:32.0778985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0779388Z outputs = self.model( 2025-09-07T07:18:32.0779765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0780177Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0780551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0780941Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0781344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0781829Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0782023Z 2025-09-07T07:18:32.0782135Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0782525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0782877Z return mod(**inputs) 2025-09-07T07:18:32.0783257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0783675Z outputs = self.model( 2025-09-07T07:18:32.0784069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0784511Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0784892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0785302Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0785799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0786294Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0786731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0787108Z return self.act(input) 2025-09-07T07:18:32.0787241Z 2025-09-07T07:18:32.0787356Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0787758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0788119Z return mod(**inputs) 2025-09-07T07:18:32.0788522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0788935Z outputs = self.model( 2025-09-07T07:18:32.0789362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0789791Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0790209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0790607Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0791036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0791470Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0791622Z 2025-09-07T07:18:32.0791745Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0792147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0792501Z return mod(**inputs) 2025-09-07T07:18:32.0792899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0793324Z outputs = self.model( 2025-09-07T07:18:32.0793720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0794143Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0794523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0794926Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0795355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.0795795Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.0795942Z 2025-09-07T07:18:32.0796055Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0796443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0796829Z return mod(**inputs) 2025-09-07T07:18:32.0797221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0797629Z outputs = self.model( 2025-09-07T07:18:32.0798006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0798413Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0798796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0799182Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0799590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0800048Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0800485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0800939Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0801116Z 2025-09-07T07:18:32.0801234Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0801616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0801966Z return mod(**inputs) 2025-09-07T07:18:32.0802348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0802753Z outputs = self.model( 2025-09-07T07:18:32.0803133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0803212Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0803452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0803567Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0803830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0803960Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0804234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0804318Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0804322Z 2025-09-07T07:18:32.0804434Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0804637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0804712Z return mod(**inputs) 2025-09-07T07:18:32.0804963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0805031Z outputs = self.model( 2025-09-07T07:18:32.0805293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0805365Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0805599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0805678Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0805933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0806033Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0806281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0806401Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0806405Z 2025-09-07T07:18:32.0806508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0806732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0806798Z return mod(**inputs) 2025-09-07T07:18:32.0807046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0807120Z outputs = self.model( 2025-09-07T07:18:32.0807366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0807446Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0807669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0807753Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0808038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0808138Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0808398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0808539Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0808542Z 2025-09-07T07:18:32.0808653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0808855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0808921Z return mod(**inputs) 2025-09-07T07:18:32.0809181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0809248Z outputs = self.model( 2025-09-07T07:18:32.0809506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0809580Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0809821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0809914Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0810182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0810292Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0810542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0810637Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0810641Z 2025-09-07T07:18:32.0810743Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0810949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0811024Z return mod(**inputs) 2025-09-07T07:18:32.0811278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0811355Z outputs = self.model( 2025-09-07T07:18:32.0811606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0811680Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0811912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0811992Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0812251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0812348Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0812610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0812708Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0812728Z 2025-09-07T07:18:32.0812831Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0813039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0813107Z return mod(**inputs) 2025-09-07T07:18:32.0813364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0813432Z outputs = self.model( 2025-09-07T07:18:32.0813685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0813767Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0813992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0814096Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0814351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0814450Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0814707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0814835Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0814839Z 2025-09-07T07:18:32.0814948Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0815149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0815222Z return mod(**inputs) 2025-09-07T07:18:32.0815479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0815549Z outputs = self.model( 2025-09-07T07:18:32.0815824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0815902Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0816139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0816235Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0816483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0816588Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0816839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0816929Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0816934Z 2025-09-07T07:18:32.0817036Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0817247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0817316Z return mod(**inputs) 2025-09-07T07:18:32.0817564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0817642Z outputs = self.model( 2025-09-07T07:18:32.0817889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0817969Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0818194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0818272Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0818531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0818653Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0818657Z 2025-09-07T07:18:32.0818785Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0818985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0819060Z return mod(**inputs) 2025-09-07T07:18:32.0819311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0819382Z outputs = self.model( 2025-09-07T07:18:32.0819821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0819901Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0820146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0820271Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0820517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0820648Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0820865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0820946Z return self.act(input) 2025-09-07T07:18:32.0820950Z 2025-09-07T07:18:32.0821054Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0821255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0821330Z return mod(**inputs) 2025-09-07T07:18:32.0821580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0821653Z outputs = self.model( 2025-09-07T07:18:32.0821906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0821992Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0822260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0822348Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0822655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0822745Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0822749Z 2025-09-07T07:18:32.0822864Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0823078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0823149Z return mod(**inputs) 2025-09-07T07:18:32.0823421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0823494Z outputs = self.model( 2025-09-07T07:18:32.0823767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0823845Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0824084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0824177Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0824439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0824552Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0824818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0824947Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0824952Z 2025-09-07T07:18:32.0825057Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0825272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0825382Z return mod(**inputs) 2025-09-07T07:18:32.0825649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0825780Z outputs = self.model( 2025-09-07T07:18:32.0826063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0826145Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0826397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0826483Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0826762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0826890Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0827164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0827254Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0827258Z 2025-09-07T07:18:32.0827361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0827568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0827635Z return mod(**inputs) 2025-09-07T07:18:32.0827889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0827956Z outputs = self.model( 2025-09-07T07:18:32.0828205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0828290Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0828533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0828625Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0828880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0829001Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0829261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0829371Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0829374Z 2025-09-07T07:18:32.0829484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0829686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0829763Z return mod(**inputs) 2025-09-07T07:18:32.0830025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0830094Z outputs = self.model( 2025-09-07T07:18:32.0830343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0830415Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0830640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0830720Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0830962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0831065Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0831309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0831453Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0831456Z 2025-09-07T07:18:32.0831578Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0831787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0831855Z return mod(**inputs) 2025-09-07T07:18:32.0832108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0832184Z outputs = self.model( 2025-09-07T07:18:32.0832435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0832516Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0832746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0832846Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0833115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0833223Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0833501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0833591Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0833595Z 2025-09-07T07:18:32.0833696Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0833903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0833969Z return mod(**inputs) 2025-09-07T07:18:32.0834227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0834295Z outputs = self.model( 2025-09-07T07:18:32.0834549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0834641Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0834870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0834959Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0835232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0835342Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0835615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0835716Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0835720Z 2025-09-07T07:18:32.0835835Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0836047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0836122Z return mod(**inputs) 2025-09-07T07:18:32.0836408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0836479Z outputs = self.model( 2025-09-07T07:18:32.0836761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0836839Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0837092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0837176Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0837455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0837557Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0837829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0837997Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0838000Z 2025-09-07T07:18:32.0838108Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0838336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0838405Z return mod(**inputs) 2025-09-07T07:18:32.0838682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0838761Z outputs = self.model( 2025-09-07T07:18:32.0839037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0839120Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0839380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0839469Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0839758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0839864Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0840138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0840225Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0840229Z 2025-09-07T07:18:32.0840345Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0840570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0840640Z return mod(**inputs) 2025-09-07T07:18:32.0840923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0840997Z outputs = self.model( 2025-09-07T07:18:32.0841291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0841371Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0841641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0841734Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0842009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0842141Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0842145Z 2025-09-07T07:18:32.0842251Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0842480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0842551Z return mod(**inputs) 2025-09-07T07:18:32.0842824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0842904Z outputs = self.model( 2025-09-07T07:18:32.0843181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0843266Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0843500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0843583Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0843865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0843990Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0844222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0844299Z return self.act(input) 2025-09-07T07:18:32.0844302Z 2025-09-07T07:18:32.0844437Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0844650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0844719Z return mod(**inputs) 2025-09-07T07:18:32.0844990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0845061Z outputs = self.model( 2025-09-07T07:18:32.0845330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0845405Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0845640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0845750Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0846013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0846111Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0846115Z 2025-09-07T07:18:32.0846223Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0846438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0846516Z return mod(**inputs) 2025-09-07T07:18:32.0846781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0846858Z outputs = self.model( 2025-09-07T07:18:32.0847128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0847212Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0847454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0847539Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0847839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.0847928Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.0847932Z 2025-09-07T07:18:32.0848065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0848278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0848349Z return mod(**inputs) 2025-09-07T07:18:32.0848621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0848692Z outputs = self.model( 2025-09-07T07:18:32.0848964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0849043Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0849288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0849382Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0849646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0849759Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0850026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0850152Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0850156Z 2025-09-07T07:18:32.0850263Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0850478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0850558Z return mod(**inputs) 2025-09-07T07:18:32.0850826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0850920Z outputs = self.model( 2025-09-07T07:18:32.0851185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0851264Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0851511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0851596Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0851864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0851969Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0852242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0852352Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0852356Z 2025-09-07T07:18:32.0852468Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0852691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0852762Z return mod(**inputs) 2025-09-07T07:18:32.0853035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0853106Z outputs = self.model( 2025-09-07T07:18:32.0853371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0853455Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0853692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0853784Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0854068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0854175Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0854451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0854583Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0854587Z 2025-09-07T07:18:32.0854705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0854924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0855001Z return mod(**inputs) 2025-09-07T07:18:32.0855269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0855354Z outputs = self.model( 2025-09-07T07:18:32.0855620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0855697Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0855936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0856016Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0856287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0856402Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0856674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0856827Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0856831Z 2025-09-07T07:18:32.0856942Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0857162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0857232Z return mod(**inputs) 2025-09-07T07:18:32.0857516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0857595Z outputs = self.model( 2025-09-07T07:18:32.0857862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0857947Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0858189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0858275Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0858551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0858675Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0858948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0859042Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0859046Z 2025-09-07T07:18:32.0859160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0859375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0859445Z return mod(**inputs) 2025-09-07T07:18:32.0859713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0859784Z outputs = self.model( 2025-09-07T07:18:32.0860053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0860130Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0860368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0860459Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0860741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0860853Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0861135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0861241Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0861250Z 2025-09-07T07:18:32.0861358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0861571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0861647Z return mod(**inputs) 2025-09-07T07:18:32.0861924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0862003Z outputs = self.model( 2025-09-07T07:18:32.0862275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0862355Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0862616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0862703Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0862986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0863095Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0878790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0879115Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0879124Z 2025-09-07T07:18:32.0879245Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0879594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0879673Z return mod(**inputs) 2025-09-07T07:18:32.0879979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0880058Z outputs = self.model( 2025-09-07T07:18:32.0880323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0880416Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0880652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0880750Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0881044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0881155Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0881420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0881506Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0881511Z 2025-09-07T07:18:32.0881633Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0881845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0881923Z return mod(**inputs) 2025-09-07T07:18:32.0882183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0882259Z outputs = self.model( 2025-09-07T07:18:32.0882541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0882624Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0882901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0882993Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0883290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0883431Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0883436Z 2025-09-07T07:18:32.0883552Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0883783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0883856Z return mod(**inputs) 2025-09-07T07:18:32.0884149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0884224Z outputs = self.model( 2025-09-07T07:18:32.0884478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0884563Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0884804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0884893Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0885143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0885261Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0885487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0885558Z return self.act(input) 2025-09-07T07:18:32.0885563Z 2025-09-07T07:18:32.0885675Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0885878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0885945Z return mod(**inputs) 2025-09-07T07:18:32.0886264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0886335Z outputs = self.model( 2025-09-07T07:18:32.0886594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0886667Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0886898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0886981Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0887229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0887351Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0887355Z 2025-09-07T07:18:32.0887460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0887675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0887744Z return mod(**inputs) 2025-09-07T07:18:32.0888008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0888085Z outputs = self.model( 2025-09-07T07:18:32.0888332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0888413Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0888638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0888719Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0888982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0889094Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0889385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0889513Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0889517Z 2025-09-07T07:18:32.0889647Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0889865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0889937Z return mod(**inputs) 2025-09-07T07:18:32.0890212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0890285Z outputs = self.model( 2025-09-07T07:18:32.0890564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0890644Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0890887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0890980Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0891251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0891360Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0891613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0891702Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0891707Z 2025-09-07T07:18:32.0891812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0892015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0892091Z return mod(**inputs) 2025-09-07T07:18:32.0892345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0892436Z outputs = self.model( 2025-09-07T07:18:32.0892685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0892758Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0892988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0893067Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0893320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0893421Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0893670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0893807Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0893814Z 2025-09-07T07:18:32.0893917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0894126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0894193Z return mod(**inputs) 2025-09-07T07:18:32.0894450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0894519Z outputs = self.model( 2025-09-07T07:18:32.0894767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0894849Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0895070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0895158Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0895427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0895529Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0895789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0895947Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0895951Z 2025-09-07T07:18:32.0896065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0896263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0896336Z return mod(**inputs) 2025-09-07T07:18:32.0896585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0896656Z outputs = self.model( 2025-09-07T07:18:32.0896917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0896993Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0897228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0897310Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0897573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0897681Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0897938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0898032Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0898035Z 2025-09-07T07:18:32.0898140Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0898388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0898496Z return mod(**inputs) 2025-09-07T07:18:32.0898758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0898836Z outputs = self.model( 2025-09-07T07:18:32.0899094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0899175Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0899404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0899484Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0899748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0899863Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0900128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0900228Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0900231Z 2025-09-07T07:18:32.0900332Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0900544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0900611Z return mod(**inputs) 2025-09-07T07:18:32.0900869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0900937Z outputs = self.model( 2025-09-07T07:18:32.0901196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0901271Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0901495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0901599Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0901857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0901962Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0902227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0902362Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0902366Z 2025-09-07T07:18:32.0902475Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0902679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0902753Z return mod(**inputs) 2025-09-07T07:18:32.0903010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0903076Z outputs = self.model( 2025-09-07T07:18:32.0903338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0903410Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0903646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0903726Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0903991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0904088Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0904339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0904431Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0904435Z 2025-09-07T07:18:32.0904538Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0904764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0904830Z return mod(**inputs) 2025-09-07T07:18:32.0905099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0905177Z outputs = self.model( 2025-09-07T07:18:32.0905449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0905534Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0905872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0905972Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0906262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0906392Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0906398Z 2025-09-07T07:18:32.0906516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0906741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0906821Z return mod(**inputs) 2025-09-07T07:18:32.0907104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0907174Z outputs = self.model( 2025-09-07T07:18:32.0907448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0907524Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0907771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0907857Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0908137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0908275Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0908523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0908606Z return self.act(input) 2025-09-07T07:18:32.0908610Z 2025-09-07T07:18:32.0908720Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0908939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0909009Z return mod(**inputs) 2025-09-07T07:18:32.0909272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0909352Z outputs = self.model( 2025-09-07T07:18:32.0909616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0909706Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0909944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0910029Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0910301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0910390Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0910393Z 2025-09-07T07:18:32.0910508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0910719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0910790Z return mod(**inputs) 2025-09-07T07:18:32.0911063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0911135Z outputs = self.model( 2025-09-07T07:18:32.0911433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0911510Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0911755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0911840Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0912101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.0912193Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.0912197Z 2025-09-07T07:18:32.0912304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0912522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0912610Z return mod(**inputs) 2025-09-07T07:18:32.0912872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0912952Z outputs = self.model( 2025-09-07T07:18:32.0913216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0913303Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0913539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0913622Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0913892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0913997Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0914269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0914415Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0914421Z 2025-09-07T07:18:32.0914540Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0914756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0914844Z return mod(**inputs) 2025-09-07T07:18:32.0915120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0915191Z outputs = self.model( 2025-09-07T07:18:32.0915464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0915550Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0915775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0915864Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0916114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0916222Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0916476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0916564Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0916568Z 2025-09-07T07:18:32.0916670Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0916871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0916948Z return mod(**inputs) 2025-09-07T07:18:32.0917198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0917275Z outputs = self.model( 2025-09-07T07:18:32.0917526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0917616Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0917851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0917932Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0918194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0918292Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0918594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0918709Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0918731Z 2025-09-07T07:18:32.0918833Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0919036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0919103Z return mod(**inputs) 2025-09-07T07:18:32.0919351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0919416Z outputs = self.model( 2025-09-07T07:18:32.0919849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0919935Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0920156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0920243Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0920499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0920602Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0920929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0921071Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0921075Z 2025-09-07T07:18:32.0921186Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0921416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0921492Z return mod(**inputs) 2025-09-07T07:18:32.0921749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0921817Z outputs = self.model( 2025-09-07T07:18:32.0922077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0922158Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0922410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0922500Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0922784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0922900Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0923186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0923287Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0923290Z 2025-09-07T07:18:32.0923401Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0923638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0923711Z return mod(**inputs) 2025-09-07T07:18:32.0923988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0924070Z outputs = self.model( 2025-09-07T07:18:32.0924363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0924444Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0924673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0924751Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0925009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0925108Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0925370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0925492Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0925495Z 2025-09-07T07:18:32.0925599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0925810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0925879Z return mod(**inputs) 2025-09-07T07:18:32.0926143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0926217Z outputs = self.model( 2025-09-07T07:18:32.0926506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0926585Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0926838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0926932Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0927219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0927345Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0927625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0927781Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0927786Z 2025-09-07T07:18:32.0927917Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0928145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0928220Z return mod(**inputs) 2025-09-07T07:18:32.0928471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0928538Z outputs = self.model( 2025-09-07T07:18:32.0928797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0928871Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0929106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0929185Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0929441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0929539Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0929791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0929880Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0929883Z 2025-09-07T07:18:32.0929994Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0930201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0930266Z return mod(**inputs) 2025-09-07T07:18:32.0930509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0930604Z outputs = self.model( 2025-09-07T07:18:32.0930855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0930933Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0931159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0931246Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0931497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0931614Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0931636Z 2025-09-07T07:18:32.0931747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0931949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0932023Z return mod(**inputs) 2025-09-07T07:18:32.0932274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0932349Z outputs = self.model( 2025-09-07T07:18:32.0932620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0932696Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0932940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0933031Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0933277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0933403Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0933636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0933718Z return self.act(input) 2025-09-07T07:18:32.0933722Z 2025-09-07T07:18:32.0933826Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0934055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0934123Z return mod(**inputs) 2025-09-07T07:18:32.0934373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0934446Z outputs = self.model( 2025-09-07T07:18:32.0934695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0934776Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0935000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0935081Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0935336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0935422Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0935425Z 2025-09-07T07:18:32.0935533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0935730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0935803Z return mod(**inputs) 2025-09-07T07:18:32.0936049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0936115Z outputs = self.model( 2025-09-07T07:18:32.0936373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0936444Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0936697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0936777Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0937037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0937147Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0937402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0937522Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0937526Z 2025-09-07T07:18:32.0937628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0937847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0937920Z return mod(**inputs) 2025-09-07T07:18:32.0938174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0938251Z outputs = self.model( 2025-09-07T07:18:32.0938502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0938581Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0938806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0938886Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0939142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0939243Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0939503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0939600Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0939606Z 2025-09-07T07:18:32.0939711Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0939922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0940005Z return mod(**inputs) 2025-09-07T07:18:32.0940264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0940330Z outputs = self.model( 2025-09-07T07:18:32.0940582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0940663Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0940886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0940975Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0941227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0941336Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0941587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0941700Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0941704Z 2025-09-07T07:18:32.0941814Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0942015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0942089Z return mod(**inputs) 2025-09-07T07:18:32.0942343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0942411Z outputs = self.model( 2025-09-07T07:18:32.0942670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0943150Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0943391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0943476Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0943761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0943866Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0944136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0944283Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0944303Z 2025-09-07T07:18:32.0944407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0944618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0944686Z return mod(**inputs) 2025-09-07T07:18:32.0944944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0945022Z outputs = self.model( 2025-09-07T07:18:32.0945276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0945357Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0945584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0945664Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0946026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0946141Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0946437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0946534Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0946538Z 2025-09-07T07:18:32.0946675Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0946890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0946961Z return mod(**inputs) 2025-09-07T07:18:32.0947244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0947311Z outputs = self.model( 2025-09-07T07:18:32.0947580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0947656Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0947881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0947973Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0948223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0948330Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0948591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0948702Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0948705Z 2025-09-07T07:18:32.0948812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0949045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0949125Z return mod(**inputs) 2025-09-07T07:18:32.0949390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0949489Z outputs = self.model( 2025-09-07T07:18:32.0949755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0949833Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0950080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0950165Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0950435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0950537Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0950803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0950965Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0950969Z 2025-09-07T07:18:32.0951081Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0951303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0951371Z return mod(**inputs) 2025-09-07T07:18:32.0951647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0951719Z outputs = self.model( 2025-09-07T07:18:32.0951983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0952066Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0952307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0952403Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0952670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0952791Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0953065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0953168Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0953173Z 2025-09-07T07:18:32.0953290Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0953503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0953582Z return mod(**inputs) 2025-09-07T07:18:32.0953847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0953917Z outputs = self.model( 2025-09-07T07:18:32.0954190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0954267Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0954520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0954604Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0954870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0955004Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0955007Z 2025-09-07T07:18:32.0955117Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0955340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0955418Z return mod(**inputs) 2025-09-07T07:18:32.0955672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0955746Z outputs = self.model( 2025-09-07T07:18:32.0955989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0956086Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0956311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0956395Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0956640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0956754Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0956975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0957046Z return self.act(input) 2025-09-07T07:18:32.0957067Z 2025-09-07T07:18:32.0957174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0957372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0957438Z return mod(**inputs) 2025-09-07T07:18:32.0957689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0957756Z outputs = self.model( 2025-09-07T07:18:32.0958007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0958082Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0958304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0958392Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0958639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0958730Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0958733Z 2025-09-07T07:18:32.0958863Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0959072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0959139Z return mod(**inputs) 2025-09-07T07:18:32.0959415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0959491Z outputs = self.model( 2025-09-07T07:18:32.0959735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0959813Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0960032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0960110Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0960362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.0960444Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.0960447Z 2025-09-07T07:18:32.0960556Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0960752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0960823Z return mod(**inputs) 2025-09-07T07:18:32.0961066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0961130Z outputs = self.model( 2025-09-07T07:18:32.0961383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0961454Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0961681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0961758Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0962021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0962125Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0962375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0962493Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0962497Z 2025-09-07T07:18:32.0962601Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0962809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0962885Z return mod(**inputs) 2025-09-07T07:18:32.0963131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0963220Z outputs = self.model( 2025-09-07T07:18:32.0963465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0963543Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0963764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0963841Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0964101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0964198Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0964449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0964527Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0964532Z 2025-09-07T07:18:32.0964631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0964848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0964915Z return mod(**inputs) 2025-09-07T07:18:32.0965163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0965245Z outputs = self.model( 2025-09-07T07:18:32.0965498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0965569Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0965789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0965874Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0966122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0966226Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0966481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0966594Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0966597Z 2025-09-07T07:18:32.0966709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0966909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0966982Z return mod(**inputs) 2025-09-07T07:18:32.0967232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0967299Z outputs = self.model( 2025-09-07T07:18:32.0967554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0967629Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0967859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0967957Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0968214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0968312Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0968567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0968709Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0968713Z 2025-09-07T07:18:32.0968813Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0969019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0969101Z return mod(**inputs) 2025-09-07T07:18:32.0969361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0969438Z outputs = self.model( 2025-09-07T07:18:32.0969694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0969775Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0970006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0970093Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0970356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0970451Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0970709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0970794Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0970798Z 2025-09-07T07:18:32.0970925Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0971123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0971189Z return mod(**inputs) 2025-09-07T07:18:32.0971466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0971537Z outputs = self.model( 2025-09-07T07:18:32.0971792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0971866Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0972087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0972177Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0972427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0972533Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0972781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0972886Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0972890Z 2025-09-07T07:18:32.0972991Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0973194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0973268Z return mod(**inputs) 2025-09-07T07:18:32.0973515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0973590Z outputs = self.model( 2025-09-07T07:18:32.0973840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0973933Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0974166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0974244Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0974503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0974600Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0974854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0974982Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0974985Z 2025-09-07T07:18:32.0975089Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0975312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0975380Z return mod(**inputs) 2025-09-07T07:18:32.0975637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0975705Z outputs = self.model( 2025-09-07T07:18:32.0975955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0976036Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0976257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0976343Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0976592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0976691Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0976962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.0977048Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.0977051Z 2025-09-07T07:18:32.0977161Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0977380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0977455Z return mod(**inputs) 2025-09-07T07:18:32.0977705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0977775Z outputs = self.model( 2025-09-07T07:18:32.0978042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0978120Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0978374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0978453Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0978707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0978832Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0978835Z 2025-09-07T07:18:32.0978939Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0979150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0979215Z return mod(**inputs) 2025-09-07T07:18:32.0979470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0979536Z outputs = self.model( 2025-09-07T07:18:32.0979785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0979865Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0980089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0980191Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0980442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.0980558Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.0980782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.0980852Z return self.act(input) 2025-09-07T07:18:32.0980855Z 2025-09-07T07:18:32.0980964Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0981167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0981260Z return mod(**inputs) 2025-09-07T07:18:32.0981522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0981590Z outputs = self.model( 2025-09-07T07:18:32.0981844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0981917Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0982151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0982234Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0982504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.0982600Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.0982604Z 2025-09-07T07:18:32.0982712Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0982934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0983003Z return mod(**inputs) 2025-09-07T07:18:32.0983311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0983391Z outputs = self.model( 2025-09-07T07:18:32.0983684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0983769Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0984006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0984089Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0984374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0984480Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0984758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0984878Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0984881Z 2025-09-07T07:18:32.0984994Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0985221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0985292Z return mod(**inputs) 2025-09-07T07:18:32.0985576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0985646Z outputs = self.model( 2025-09-07T07:18:32.0986183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0986271Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0986524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0986621Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0986935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0987051Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0987335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.0987425Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.0987430Z 2025-09-07T07:18:32.0987534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0987778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0987851Z return mod(**inputs) 2025-09-07T07:18:32.0988123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0988197Z outputs = self.model( 2025-09-07T07:18:32.0988449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0988526Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0988765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0988844Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0989140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0989238Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0989491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.0989612Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.0989615Z 2025-09-07T07:18:32.0989718Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0989943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0990013Z return mod(**inputs) 2025-09-07T07:18:32.0990289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0990360Z outputs = self.model( 2025-09-07T07:18:32.0990610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0990691Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0990914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0991000Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0991251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0991349Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0991610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.0991745Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.0991749Z 2025-09-07T07:18:32.0991859Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0992060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0992132Z return mod(**inputs) 2025-09-07T07:18:32.0992381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0992447Z outputs = self.model( 2025-09-07T07:18:32.0992699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0992773Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0993004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0993100Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0993347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0993453Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0993701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.0993795Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.0993798Z 2025-09-07T07:18:32.0993900Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0994099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0994191Z return mod(**inputs) 2025-09-07T07:18:32.0994447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0994523Z outputs = self.model( 2025-09-07T07:18:32.0994778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0994858Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0995088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0995166Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0995424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0995522Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0995786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.0995882Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.0995904Z 2025-09-07T07:18:32.0996008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0996217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0996283Z return mod(**inputs) 2025-09-07T07:18:32.0996575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0996645Z outputs = self.model( 2025-09-07T07:18:32.0996910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0996983Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0997208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0997297Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0997549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0997654Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.0997905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.0998035Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.0998039Z 2025-09-07T07:18:32.0998150Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.0998352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.0998426Z return mod(**inputs) 2025-09-07T07:18:32.0998675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.0998745Z outputs = self.model( 2025-09-07T07:18:32.0999004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.0999103Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.0999333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.0999412Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.0999667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.0999763Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1000010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1000097Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1000100Z 2025-09-07T07:18:32.1000218Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1000420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1000487Z return mod(**inputs) 2025-09-07T07:18:32.1000730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1000805Z outputs = self.model( 2025-09-07T07:18:32.1001051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1001128Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1001345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1001422Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1001672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1001788Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1001792Z 2025-09-07T07:18:32.1001898Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1002114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1002188Z return mod(**inputs) 2025-09-07T07:18:32.1002444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1002512Z outputs = self.model( 2025-09-07T07:18:32.1002771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1002843Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1003074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1003153Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1003404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1003531Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1003749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1003825Z return self.act(input) 2025-09-07T07:18:32.1003828Z 2025-09-07T07:18:32.1003932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1004141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1004206Z return mod(**inputs) 2025-09-07T07:18:32.1004470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1004543Z outputs = self.model( 2025-09-07T07:18:32.1004786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1004867Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1005084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1005188Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1005441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1005521Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1005525Z 2025-09-07T07:18:32.1005631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1005833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1005899Z return mod(**inputs) 2025-09-07T07:18:32.1006155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1006239Z outputs = self.model( 2025-09-07T07:18:32.1006504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1006580Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1006817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1006897Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1007154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.1007242Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.1007245Z 2025-09-07T07:18:32.1007351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1007563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1007627Z return mod(**inputs) 2025-09-07T07:18:32.1007886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1007960Z outputs = self.model( 2025-09-07T07:18:32.1008235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1008316Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1008554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1008634Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1008895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1008993Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1009248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1009363Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1009366Z 2025-09-07T07:18:32.1009473Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1009677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1009746Z return mod(**inputs) 2025-09-07T07:18:32.1010004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1010074Z outputs = self.model( 2025-09-07T07:18:32.1010329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1010403Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1010624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1010713Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1010960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1011069Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1011332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1011419Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1011424Z 2025-09-07T07:18:32.1011524Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1011719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1011791Z return mod(**inputs) 2025-09-07T07:18:32.1012034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1012105Z outputs = self.model( 2025-09-07T07:18:32.1012354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1012444Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1012673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1012752Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1013006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1013105Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1013355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1013475Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1013478Z 2025-09-07T07:18:32.1013580Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1013788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1013855Z return mod(**inputs) 2025-09-07T07:18:32.1014134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1014206Z outputs = self.model( 2025-09-07T07:18:32.1014454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1014549Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1014774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1014860Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1015110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1015209Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1015469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1015607Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1015611Z 2025-09-07T07:18:32.1015723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1015922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1015997Z return mod(**inputs) 2025-09-07T07:18:32.1016254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1016318Z outputs = self.model( 2025-09-07T07:18:32.1016574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1016646Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1016879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1016961Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1017212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1017338Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1017591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1017685Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1017688Z 2025-09-07T07:18:32.1017792Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1018001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1018069Z return mod(**inputs) 2025-09-07T07:18:32.1018318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1018409Z outputs = self.model( 2025-09-07T07:18:32.1018661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1018745Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1018971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1019051Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1019309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1019409Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1019855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1019959Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1019966Z 2025-09-07T07:18:32.1020069Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1020278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1020393Z return mod(**inputs) 2025-09-07T07:18:32.1020653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1020722Z outputs = self.model( 2025-09-07T07:18:32.1021007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1021083Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1021310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1021399Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1021654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1021768Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1022048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1022187Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1022190Z 2025-09-07T07:18:32.1022307Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1022534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1022612Z return mod(**inputs) 2025-09-07T07:18:32.1022887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1022965Z outputs = self.model( 2025-09-07T07:18:32.1023241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1023321Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1023566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1023681Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1023967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1024072Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1024357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1024451Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1024455Z 2025-09-07T07:18:32.1024563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1024787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1024879Z return mod(**inputs) 2025-09-07T07:18:32.1025154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1025234Z outputs = self.model( 2025-09-07T07:18:32.1025522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1025604Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1025918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1026014Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1026289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1026413Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1026418Z 2025-09-07T07:18:32.1026533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1026748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1026826Z return mod(**inputs) 2025-09-07T07:18:32.1027121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1027196Z outputs = self.model( 2025-09-07T07:18:32.1027524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1027605Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1027856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1027948Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1028200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1028323Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1028542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1028625Z return self.act(input) 2025-09-07T07:18:32.1028628Z 2025-09-07T07:18:32.1028731Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1028941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1029008Z return mod(**inputs) 2025-09-07T07:18:32.1029256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1029330Z outputs = self.model( 2025-09-07T07:18:32.1029581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1029660Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1029885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1029966Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1030227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1030329Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1030333Z 2025-09-07T07:18:32.1030443Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1030646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1030724Z return mod(**inputs) 2025-09-07T07:18:32.1030973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1031042Z outputs = self.model( 2025-09-07T07:18:32.1031297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1031388Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1031618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1031699Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1031950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1032061Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1032312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1032431Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1032434Z 2025-09-07T07:18:32.1032537Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1032737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1032813Z return mod(**inputs) 2025-09-07T07:18:32.1033063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1033156Z outputs = self.model( 2025-09-07T07:18:32.1033411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1033491Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1033733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1033813Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1034068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1034167Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1034427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1034509Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1034513Z 2025-09-07T07:18:32.1034628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1034833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1034898Z return mod(**inputs) 2025-09-07T07:18:32.1035151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1035220Z outputs = self.model( 2025-09-07T07:18:32.1035466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1035549Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1035770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1035856Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1036107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1036231Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1036484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1036595Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1036598Z 2025-09-07T07:18:32.1036711Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1036914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1036985Z return mod(**inputs) 2025-09-07T07:18:32.1037236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1037303Z outputs = self.model( 2025-09-07T07:18:32.1037577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1037649Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1037883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1037963Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1038230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1038325Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1038568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1038705Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1038708Z 2025-09-07T07:18:32.1038808Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1039012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1039078Z return mod(**inputs) 2025-09-07T07:18:32.1039334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1039411Z outputs = self.model( 2025-09-07T07:18:32.1039675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1039753Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1039972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1040050Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1040300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1040396Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1040642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1040731Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1040734Z 2025-09-07T07:18:32.1040841Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1041037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1041102Z return mod(**inputs) 2025-09-07T07:18:32.1041351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1041418Z outputs = self.model( 2025-09-07T07:18:32.1041669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1041739Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1041963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1042051Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1042299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1042423Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1042674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1042778Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1042781Z 2025-09-07T07:18:32.1042882Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1043085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1043159Z return mod(**inputs) 2025-09-07T07:18:32.1043407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1043502Z outputs = self.model( 2025-09-07T07:18:32.1043765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1043837Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1044063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1044142Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1044396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1044492Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1044738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1044871Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1044876Z 2025-09-07T07:18:32.1044976Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1045194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1045261Z return mod(**inputs) 2025-09-07T07:18:32.1045511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1045594Z outputs = self.model( 2025-09-07T07:18:32.1045843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1045921Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1046140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1046225Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1046469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1046567Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1046819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1046899Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1046903Z 2025-09-07T07:18:32.1047012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1047207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1047278Z return mod(**inputs) 2025-09-07T07:18:32.1047521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1047587Z outputs = self.model( 2025-09-07T07:18:32.1047842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1047915Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1048138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1048231Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1048471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1048595Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1048599Z 2025-09-07T07:18:32.1048697Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1048898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1048963Z return mod(**inputs) 2025-09-07T07:18:32.1049205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1049294Z outputs = self.model( 2025-09-07T07:18:32.1049543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1049625Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1049849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1049935Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1050192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1050306Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1050526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1050596Z return self.act(input) 2025-09-07T07:18:32.1050599Z 2025-09-07T07:18:32.1050708Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1050908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1050971Z return mod(**inputs) 2025-09-07T07:18:32.1051251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1051319Z outputs = self.model( 2025-09-07T07:18:32.1051587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1051660Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1051879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1051960Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1052203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1052289Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1052294Z 2025-09-07T07:18:32.1052393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1052596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1052664Z return mod(**inputs) 2025-09-07T07:18:32.1052922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1052998Z outputs = self.model( 2025-09-07T07:18:32.1053248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1053328Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1053553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1053632Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1053887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.1053970Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.1053974Z 2025-09-07T07:18:32.1054106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1054303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1054376Z return mod(**inputs) 2025-09-07T07:18:32.1054632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1054697Z outputs = self.model( 2025-09-07T07:18:32.1054945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1055017Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1055247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1055345Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1055592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1055704Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1055951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1056071Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1056074Z 2025-09-07T07:18:32.1056177Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1056385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1056452Z return mod(**inputs) 2025-09-07T07:18:32.1056703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1056780Z outputs = self.model( 2025-09-07T07:18:32.1057028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1057137Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1057367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1057446Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1057723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1057825Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1058084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1058165Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1058168Z 2025-09-07T07:18:32.1058269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1058481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1058548Z return mod(**inputs) 2025-09-07T07:18:32.1058808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1058876Z outputs = self.model( 2025-09-07T07:18:32.1059134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1059207Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1059429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1059517Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1059764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1059868Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1060118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1060248Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1060252Z 2025-09-07T07:18:32.1060361Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1060565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1060639Z return mod(**inputs) 2025-09-07T07:18:32.1060891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1060959Z outputs = self.model( 2025-09-07T07:18:32.1061214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1061287Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1061534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1061613Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1061875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1061973Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1062235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1062388Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1062392Z 2025-09-07T07:18:32.1062502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1062723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1062794Z return mod(**inputs) 2025-09-07T07:18:32.1063057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1063137Z outputs = self.model( 2025-09-07T07:18:32.1063419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1063505Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1063764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1063858Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1064121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1064224Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1064497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1064589Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1064595Z 2025-09-07T07:18:32.1064709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1064922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1064993Z return mod(**inputs) 2025-09-07T07:18:32.1065260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1065332Z outputs = self.model( 2025-09-07T07:18:32.1065598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1065674Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1065979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1066077Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1066343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1066459Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1066749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1066859Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1066863Z 2025-09-07T07:18:32.1066973Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1067192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1067267Z return mod(**inputs) 2025-09-07T07:18:32.1067515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1067590Z outputs = self.model( 2025-09-07T07:18:32.1067843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1067934Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1068165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1068246Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1068501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1068600Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1068856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1068984Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1068987Z 2025-09-07T07:18:32.1069090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1069299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1069367Z return mod(**inputs) 2025-09-07T07:18:32.1069646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1069714Z outputs = self.model( 2025-09-07T07:18:32.1069962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1070059Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1070280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1070364Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1070606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1070702Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1070952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1071035Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1071038Z 2025-09-07T07:18:32.1071149Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1071347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1071419Z return mod(**inputs) 2025-09-07T07:18:32.1071667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1071733Z outputs = self.model( 2025-09-07T07:18:32.1071989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1072064Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1072294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1072375Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1072627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1072773Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1072777Z 2025-09-07T07:18:32.1072880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1073089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1073156Z return mod(**inputs) 2025-09-07T07:18:32.1073418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1073485Z outputs = self.model( 2025-09-07T07:18:32.1073739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1073820Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1074069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1074158Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1074424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1074536Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1074758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1074827Z return self.act(input) 2025-09-07T07:18:32.1074830Z 2025-09-07T07:18:32.1074935Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1075131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1075196Z return mod(**inputs) 2025-09-07T07:18:32.1075446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1075513Z outputs = self.model( 2025-09-07T07:18:32.1075782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1075858Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1076104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1076186Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1076435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1076524Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1076528Z 2025-09-07T07:18:32.1076631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1076836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1076905Z return mod(**inputs) 2025-09-07T07:18:32.1077153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1077230Z outputs = self.model( 2025-09-07T07:18:32.1077481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1077563Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1077787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1077866Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1078121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1078219Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1078477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1078590Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1078611Z 2025-09-07T07:18:32.1078722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1078926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1078993Z return mod(**inputs) 2025-09-07T07:18:32.1079253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1079322Z outputs = self.model( 2025-09-07T07:18:32.1079577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1079649Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1079874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1079980Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1080234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1080339Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1080587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1080675Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1080678Z 2025-09-07T07:18:32.1080778Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1080976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1081051Z return mod(**inputs) 2025-09-07T07:18:32.1081314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1081390Z outputs = self.model( 2025-09-07T07:18:32.1081652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1081747Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1081992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1082075Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1082358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1082463Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1082725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1082849Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1082853Z 2025-09-07T07:18:32.1082959Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1083188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1083259Z return mod(**inputs) 2025-09-07T07:18:32.1083536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1083607Z outputs = self.model( 2025-09-07T07:18:32.1083878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1083958Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1084182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1084268Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1084518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1084618Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1084877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1085033Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1085037Z 2025-09-07T07:18:32.1085146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1085356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1085428Z return mod(**inputs) 2025-09-07T07:18:32.1085689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1085756Z outputs = self.model( 2025-09-07T07:18:32.1086010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1086081Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1086328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1086410Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1086662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1086771Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1087024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1087118Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1087122Z 2025-09-07T07:18:32.1087225Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1087429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1087504Z return mod(**inputs) 2025-09-07T07:18:32.1087755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1087828Z outputs = self.model( 2025-09-07T07:18:32.1088096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1088178Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1088419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1088499Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1088755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1088853Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1089110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1089207Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1089213Z 2025-09-07T07:18:32.1089313Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1089523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1089591Z return mod(**inputs) 2025-09-07T07:18:32.1089851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1089919Z outputs = self.model( 2025-09-07T07:18:32.1090180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1090253Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1090477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1090561Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1090820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1090923Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1091193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1091321Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1091325Z 2025-09-07T07:18:32.1091435Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1091636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1091709Z return mod(**inputs) 2025-09-07T07:18:32.1091985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1092056Z outputs = self.model( 2025-09-07T07:18:32.1092342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1092437Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1092685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1092771Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1093081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1093181Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1093432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1093523Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1093527Z 2025-09-07T07:18:32.1093632Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1093843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1093913Z return mod(**inputs) 2025-09-07T07:18:32.1094194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1094272Z outputs = self.model( 2025-09-07T07:18:32.1094523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1094618Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1094843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1094923Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1095185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1095303Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1095309Z 2025-09-07T07:18:32.1095419Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1095619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1095694Z return mod(**inputs) 2025-09-07T07:18:32.1095945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1096012Z outputs = self.model( 2025-09-07T07:18:32.1096271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1096342Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1096572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1096650Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1096903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1097030Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1097247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1097350Z return self.act(input) 2025-09-07T07:18:32.1097354Z 2025-09-07T07:18:32.1097457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1097668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1097733Z return mod(**inputs) 2025-09-07T07:18:32.1097991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1098067Z outputs = self.model( 2025-09-07T07:18:32.1098316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1098396Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1098640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1098720Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1098978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1099059Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1099062Z 2025-09-07T07:18:32.1099173Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1099375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1099440Z return mod(**inputs) 2025-09-07T07:18:32.1099696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1099764Z outputs = self.model( 2025-09-07T07:18:32.1100018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1100093Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1100341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1100423Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1100701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.1100791Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.1100794Z 2025-09-07T07:18:32.1100897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1101104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1101170Z return mod(**inputs) 2025-09-07T07:18:32.1101428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1101512Z outputs = self.model( 2025-09-07T07:18:32.1101779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1101865Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1102102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1102188Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1102459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1102562Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1102831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1102949Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1102952Z 2025-09-07T07:18:32.1103068Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1103278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1103371Z return mod(**inputs) 2025-09-07T07:18:32.1103647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1103718Z outputs = self.model( 2025-09-07T07:18:32.1103999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1104075Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1104312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1104405Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1104670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1104802Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1105067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1105161Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1105165Z 2025-09-07T07:18:32.1105273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1105499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1105577Z return mod(**inputs) 2025-09-07T07:18:32.1105918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1106004Z outputs = self.model( 2025-09-07T07:18:32.1106279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1106361Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1106607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1106714Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1107006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1107112Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1107446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1107572Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1107576Z 2025-09-07T07:18:32.1107685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1107910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1107982Z return mod(**inputs) 2025-09-07T07:18:32.1108257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1108328Z outputs = self.model( 2025-09-07T07:18:32.1108618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1108703Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1108946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1109037Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1109313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1109418Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1109705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1109850Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1109853Z 2025-09-07T07:18:32.1109969Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1110203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1110280Z return mod(**inputs) 2025-09-07T07:18:32.1110557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1110630Z outputs = self.model( 2025-09-07T07:18:32.1110901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1110977Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1111220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1111303Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1111598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1111712Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1111995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1112094Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1112100Z 2025-09-07T07:18:32.1112207Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1112436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1112504Z return mod(**inputs) 2025-09-07T07:18:32.1112773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1112850Z outputs = self.model( 2025-09-07T07:18:32.1113119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1113204Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1113456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1113542Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1113854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1113959Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1114237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1114337Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1114341Z 2025-09-07T07:18:32.1114448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1114668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1114736Z return mod(**inputs) 2025-09-07T07:18:32.1114995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1115063Z outputs = self.model( 2025-09-07T07:18:32.1115324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1115399Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1115625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1115711Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1115985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1116095Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1116367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1116503Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1116524Z 2025-09-07T07:18:32.1116641Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1116854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1116932Z return mod(**inputs) 2025-09-07T07:18:32.1117200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1117280Z outputs = self.model( 2025-09-07T07:18:32.1117546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1117632Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1117864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1117961Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1118216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1118316Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1118566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1118657Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1118660Z 2025-09-07T07:18:32.1118761Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1118967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1119034Z return mod(**inputs) 2025-09-07T07:18:32.1119281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1119359Z outputs = self.model( 2025-09-07T07:18:32.1119809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1119901Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1120129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1120248Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1120500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1120621Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1120625Z 2025-09-07T07:18:32.1120736Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1120937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1121012Z return mod(**inputs) 2025-09-07T07:18:32.1121259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1121331Z outputs = self.model( 2025-09-07T07:18:32.1121591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1121664Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1121897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1121976Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1122227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1122352Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1122567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1122649Z return self.act(input) 2025-09-07T07:18:32.1122652Z 2025-09-07T07:18:32.1122757Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1122990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1123059Z return mod(**inputs) 2025-09-07T07:18:32.1123309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1123386Z outputs = self.model( 2025-09-07T07:18:32.1123633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1123713Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1123936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1124014Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1124294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1124377Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1124382Z 2025-09-07T07:18:32.1124490Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1124689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1124763Z return mod(**inputs) 2025-09-07T07:18:32.1125015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1125080Z outputs = self.model( 2025-09-07T07:18:32.1125334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1125405Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1125639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1125718Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1125980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1126088Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1126352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1126474Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1126478Z 2025-09-07T07:18:32.1126582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1126783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1126860Z return mod(**inputs) 2025-09-07T07:18:32.1127129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1127210Z outputs = self.model( 2025-09-07T07:18:32.1127493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1127575Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1127804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1127884Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1128145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1128244Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1128503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1128583Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1128589Z 2025-09-07T07:18:32.1128691Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1128902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1128989Z return mod(**inputs) 2025-09-07T07:18:32.1129246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1129312Z outputs = self.model( 2025-09-07T07:18:32.1129563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1129644Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1129867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1129952Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1130203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1130325Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1130578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1130689Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1130692Z 2025-09-07T07:18:32.1130804Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1131006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1131081Z return mod(**inputs) 2025-09-07T07:18:32.1131332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1131399Z outputs = self.model( 2025-09-07T07:18:32.1131657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1131730Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1131961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1132058Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1132318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1132431Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1132682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1132824Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1132828Z 2025-09-07T07:18:32.1132930Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1133140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1133209Z return mod(**inputs) 2025-09-07T07:18:32.1133460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1133541Z outputs = self.model( 2025-09-07T07:18:32.1133791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1133873Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1134110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1134187Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1134437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1134534Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1134786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1134874Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1134877Z 2025-09-07T07:18:32.1134989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1135206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1135271Z return mod(**inputs) 2025-09-07T07:18:32.1135530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1135597Z outputs = self.model( 2025-09-07T07:18:32.1135851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1135924Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1136146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1136232Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1136499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1136605Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1136854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1136960Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1136964Z 2025-09-07T07:18:32.1137064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1137272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1137344Z return mod(**inputs) 2025-09-07T07:18:32.1137589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1137663Z outputs = self.model( 2025-09-07T07:18:32.1137909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1137979Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1138224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1138305Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1138591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1138696Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1138956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1139098Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1139102Z 2025-09-07T07:18:32.1139208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1139430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1139501Z return mod(**inputs) 2025-09-07T07:18:32.1139773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1139844Z outputs = self.model( 2025-09-07T07:18:32.1140110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1140193Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1140429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1140517Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1140778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1140887Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1141143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1141246Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1141250Z 2025-09-07T07:18:32.1141363Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1141577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1141655Z return mod(**inputs) 2025-09-07T07:18:32.1141921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1141992Z outputs = self.model( 2025-09-07T07:18:32.1142267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1142343Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1142587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1142686Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1142948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1143081Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1143084Z 2025-09-07T07:18:32.1143193Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1143413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1143484Z return mod(**inputs) 2025-09-07T07:18:32.1143747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1143825Z outputs = self.model( 2025-09-07T07:18:32.1144093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1144180Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1144439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1144535Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1144800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1144936Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1145174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1145249Z return self.act(input) 2025-09-07T07:18:32.1145252Z 2025-09-07T07:18:32.1145367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1145578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1145649Z return mod(**inputs) 2025-09-07T07:18:32.1145985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1146068Z outputs = self.model( 2025-09-07T07:18:32.1146342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1146418Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1146658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1146751Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1147015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1147110Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1147114Z 2025-09-07T07:18:32.1147222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1147454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1147522Z return mod(**inputs) 2025-09-07T07:18:32.1147779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1147880Z outputs = self.model( 2025-09-07T07:18:32.1148136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1148218Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1148443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1148523Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1148790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.1148872Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.1148891Z 2025-09-07T07:18:32.1149002Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1149213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1149286Z return mod(**inputs) 2025-09-07T07:18:32.1149532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1149599Z outputs = self.model( 2025-09-07T07:18:32.1149852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1149924Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1150154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1150231Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1150483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1150592Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1150861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1150981Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1150985Z 2025-09-07T07:18:32.1151104Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1151312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1151379Z return mod(**inputs) 2025-09-07T07:18:32.1151626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1151705Z outputs = self.model( 2025-09-07T07:18:32.1151959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1152039Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1152265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1152346Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1152604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1152704Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1152961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1153042Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1153045Z 2025-09-07T07:18:32.1153149Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1153359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1153427Z return mod(**inputs) 2025-09-07T07:18:32.1153687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1153773Z outputs = self.model( 2025-09-07T07:18:32.1154032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1154105Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1154331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1154418Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1154668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1154773Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1155028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1155157Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1155162Z 2025-09-07T07:18:32.1155273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1155473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1155550Z return mod(**inputs) 2025-09-07T07:18:32.1155813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1155886Z outputs = self.model( 2025-09-07T07:18:32.1156149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1156227Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1156463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1156548Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1156812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1156933Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1157191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1157352Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1157357Z 2025-09-07T07:18:32.1157462Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1157669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1157736Z return mod(**inputs) 2025-09-07T07:18:32.1157983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1158060Z outputs = self.model( 2025-09-07T07:18:32.1158308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1158392Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1158617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1158703Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1158956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1159055Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1159313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1159401Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1159405Z 2025-09-07T07:18:32.1159513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1159715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1159782Z return mod(**inputs) 2025-09-07T07:18:32.1160059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1160126Z outputs = self.model( 2025-09-07T07:18:32.1160384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1160458Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1160681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1160768Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1161016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1161142Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1161392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1161499Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1161503Z 2025-09-07T07:18:32.1161606Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1161806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1161882Z return mod(**inputs) 2025-09-07T07:18:32.1162130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1162202Z outputs = self.model( 2025-09-07T07:18:32.1162451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1162524Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1162756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1162836Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1163112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1163212Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1163502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1163642Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1163646Z 2025-09-07T07:18:32.1163754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1163980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1164046Z return mod(**inputs) 2025-09-07T07:18:32.1164303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1164371Z outputs = self.model( 2025-09-07T07:18:32.1164622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1164706Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1164932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1165020Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1165274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1165380Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1165631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1165712Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1165717Z 2025-09-07T07:18:32.1165828Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1166032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1166126Z return mod(**inputs) 2025-09-07T07:18:32.1166382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1166450Z outputs = self.model( 2025-09-07T07:18:32.1166714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1166786Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1167024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1167103Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1167361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1167514Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1167519Z 2025-09-07T07:18:32.1167624Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1167834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1167900Z return mod(**inputs) 2025-09-07T07:18:32.1168158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1168235Z outputs = self.model( 2025-09-07T07:18:32.1168476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1168553Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1168770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1168855Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1169111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1169227Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1169445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1169531Z return self.act(input) 2025-09-07T07:18:32.1169535Z 2025-09-07T07:18:32.1169642Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1169835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1169900Z return mod(**inputs) 2025-09-07T07:18:32.1170148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1170215Z outputs = self.model( 2025-09-07T07:18:32.1170466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1170538Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1170766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1170842Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1171088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1171174Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1171178Z 2025-09-07T07:18:32.1171280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1171481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1171546Z return mod(**inputs) 2025-09-07T07:18:32.1171790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1171866Z outputs = self.model( 2025-09-07T07:18:32.1172120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1172218Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1172444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1172524Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1172783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1172882Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1173140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1173252Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1173275Z 2025-09-07T07:18:32.1173386Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1173589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1173662Z return mod(**inputs) 2025-09-07T07:18:32.1173935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1174008Z outputs = self.model( 2025-09-07T07:18:32.1174288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1174366Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1174605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1174699Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1174967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1175082Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1175370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1175466Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1175470Z 2025-09-07T07:18:32.1175595Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1175808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1175887Z return mod(**inputs) 2025-09-07T07:18:32.1176158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1176232Z outputs = self.model( 2025-09-07T07:18:32.1176480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1176554Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1176788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1176870Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1177129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1177229Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1177485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1177610Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1177613Z 2025-09-07T07:18:32.1177723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1177942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1178022Z return mod(**inputs) 2025-09-07T07:18:32.1178281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1178366Z outputs = self.model( 2025-09-07T07:18:32.1178619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1178701Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1178932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1179019Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1179274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1179372Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1179637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1179791Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1179797Z 2025-09-07T07:18:32.1179909Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1180111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1180184Z return mod(**inputs) 2025-09-07T07:18:32.1180437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1180505Z outputs = self.model( 2025-09-07T07:18:32.1180762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1180836Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1181068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1181150Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1181416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1181525Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1181787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1181926Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1181931Z 2025-09-07T07:18:32.1182040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1182264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1182338Z return mod(**inputs) 2025-09-07T07:18:32.1182601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1182682Z outputs = self.model( 2025-09-07T07:18:32.1182947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1183034Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1183279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1183362Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1183638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1183742Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1184022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1184125Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1184129Z 2025-09-07T07:18:32.1184236Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1184460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1184529Z return mod(**inputs) 2025-09-07T07:18:32.1184820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1184891Z outputs = self.model( 2025-09-07T07:18:32.1185163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1185240Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1185477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1185570Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1185926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1186071Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1186349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1186489Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1186493Z 2025-09-07T07:18:32.1186613Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1186844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1186924Z return mod(**inputs) 2025-09-07T07:18:32.1187192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1187270Z outputs = self.model( 2025-09-07T07:18:32.1187537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1187614Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1187864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1187968Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1188241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1188345Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1188626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1188725Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1188729Z 2025-09-07T07:18:32.1188839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1189058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1189129Z return mod(**inputs) 2025-09-07T07:18:32.1189396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1189475Z outputs = self.model( 2025-09-07T07:18:32.1189739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1189826Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1190063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1190154Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1190419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1190543Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1190547Z 2025-09-07T07:18:32.1190664Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1190879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1190959Z return mod(**inputs) 2025-09-07T07:18:32.1191223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1191325Z outputs = self.model( 2025-09-07T07:18:32.1191605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1191684Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1191932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1192015Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1192286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1192420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1192699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1192782Z return self.act(input) 2025-09-07T07:18:32.1192788Z 2025-09-07T07:18:32.1192903Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1193131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1193203Z return mod(**inputs) 2025-09-07T07:18:32.1193478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1193558Z outputs = self.model( 2025-09-07T07:18:32.1193834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1193919Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1194164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1194251Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1194548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1194641Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1194645Z 2025-09-07T07:18:32.1194761Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1194993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1195074Z return mod(**inputs) 2025-09-07T07:18:32.1195351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1195423Z outputs = self.model( 2025-09-07T07:18:32.1195710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1195790Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1196048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1196136Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1196414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-09-07T07:18:32.1196510Z hidden_states = residual + hidden_states 2025-09-07T07:18:32.1196514Z 2025-09-07T07:18:32.1196628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1196860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1196934Z return mod(**inputs) 2025-09-07T07:18:32.1197213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1197292Z outputs = self.model( 2025-09-07T07:18:32.1197556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1197642Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1197885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1197992Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1198255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1198361Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1198633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1198752Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1198756Z 2025-09-07T07:18:32.1198869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1199084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1199172Z return mod(**inputs) 2025-09-07T07:18:32.1199450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1199520Z outputs = self.model( 2025-09-07T07:18:32.1199804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1199884Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1200130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1200221Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1200498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1200612Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1200889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-09-07T07:18:32.1200983Z key_states = self.k_proj(current_states) 2025-09-07T07:18:32.1201007Z 2025-09-07T07:18:32.1201120Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1201342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1201420Z return mod(**inputs) 2025-09-07T07:18:32.1201711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1201793Z outputs = self.model( 2025-09-07T07:18:32.1202065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1202143Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1202394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1202482Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1202762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1202871Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1203148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-09-07T07:18:32.1203268Z query_states = self.q_proj(hidden_states) * self.scaling 2025-09-07T07:18:32.1203272Z 2025-09-07T07:18:32.1203384Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1203611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1203685Z return mod(**inputs) 2025-09-07T07:18:32.1203966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1204041Z outputs = self.model( 2025-09-07T07:18:32.1204312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1204423Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1204668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1204762Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1205035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1205140Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1205421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-09-07T07:18:32.1205569Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-09-07T07:18:32.1205590Z 2025-09-07T07:18:32.1205710Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1205930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1206015Z return mod(**inputs) 2025-09-07T07:18:32.1206289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1206365Z outputs = self.model( 2025-09-07T07:18:32.1206646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1206726Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1206976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1207063Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1207332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1207448Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1207735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-09-07T07:18:32.1207842Z value_states = self.v_proj(current_states) 2025-09-07T07:18:32.1207846Z 2025-09-07T07:18:32.1207958Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1208204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1208279Z return mod(**inputs) 2025-09-07T07:18:32.1208550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1208631Z outputs = self.model( 2025-09-07T07:18:32.1208901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1208990Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1209234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1209324Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1209615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1209722Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1210008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-09-07T07:18:32.1210110Z attn_output = torch.bmm(attn_probs, value_states) 2025-09-07T07:18:32.1210114Z 2025-09-07T07:18:32.1210232Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1210460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1210531Z return mod(**inputs) 2025-09-07T07:18:32.1210822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1210893Z outputs = self.model( 2025-09-07T07:18:32.1211205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1211284Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1211534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1211629Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1211909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1212023Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1212310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-09-07T07:18:32.1212473Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-09-07T07:18:32.1212486Z 2025-09-07T07:18:32.1212597Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1212827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1212907Z return mod(**inputs) 2025-09-07T07:18:32.1213209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1213288Z outputs = self.model( 2025-09-07T07:18:32.1213564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1213643Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1213911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1213995Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1214281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-09-07T07:18:32.1214402Z hidden_states, self_attn_weights = self.self_attn( 2025-09-07T07:18:32.1214687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-09-07T07:18:32.1214782Z attn_output = self.out_proj(attn_output) 2025-09-07T07:18:32.1214811Z 2025-09-07T07:18:32.1214921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1215148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1215217Z return mod(**inputs) 2025-09-07T07:18:32.1215490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1215567Z outputs = self.model( 2025-09-07T07:18:32.1215839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1215924Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1216166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1216256Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1216533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1216658Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1216662Z 2025-09-07T07:18:32.1216781Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1217005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1217082Z return mod(**inputs) 2025-09-07T07:18:32.1217366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1217438Z outputs = self.model( 2025-09-07T07:18:32.1217721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1217817Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1218059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1218144Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1218432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-09-07T07:18:32.1218558Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-09-07T07:18:32.1218792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:18:32.1218877Z return self.act(input) 2025-09-07T07:18:32.1218899Z 2025-09-07T07:18:32.1219012Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1219247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1219324Z return mod(**inputs) 2025-09-07T07:18:32.1219763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-09-07T07:18:32.1219853Z outputs = self.model( 2025-09-07T07:18:32.1220130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-09-07T07:18:32.1220217Z layer_outputs = decoder_layer( 2025-09-07T07:18:32.1220457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:18:32.1220543Z return super().__call__(*args, **kwargs) 2025-09-07T07:18:32.1220820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-09-07T07:18:32.1220909Z hidden_states = self.fc2(hidden_states) 2025-09-07T07:18:32.1220915Z 2025-09-07T07:18:32.1221039Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1221304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1221386Z return mod(**inputs) 2025-09-07T07:18:32.1221686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 681, in forward 2025-09-07T07:18:32.1221776Z logits = self.lm_head(outputs[0]) 2025-09-07T07:18:32.1221780Z 2025-09-07T07:18:32.1221901Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:18:32.1222118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:18:32.1222199Z return mod(**inputs) 2025-09-07T07:18:32.1222475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-09-07T07:18:32.1222558Z loss = self.loss_function( 2025-09-07T07:18:32.1222844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-09-07T07:18:32.1223042Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-09-07T07:18:32.1223334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-09-07T07:18:32.1223561Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-09-07T07:18:32.1223566Z 2025-09-07T07:18:46.6460630Z Compilation time (from dynamo_timed): 27.708255907 2025-09-07T07:18:46.6556431Z pass 2025-09-07T07:18:46.6558619Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:18:46.6559735Z TIMING: _recursive_pre_grad_passes:0.01363 _recursive_joint_graph_passes:0.80011 _recursive_post_grad_passes:0.27905 async_compile.wait:0.84439 code_gen:13.80668 inductor_compile:17.04956 backend_compile:23.01492 gc:0.0006 entire_frame_compile:27.70826 total_wall_time:27.70826 2025-09-07T07:18:46.6564916Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:29106 | FakeTensor.__torch_dispatch__:9977 | ProxyTorchDispatchMode.__torch_dispatch__:10816 2025-09-07T07:18:46.6569950Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-09-07T07:18:49.7430056Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:18:49.7431079Z import pynvml # type: ignore[import] 2025-09-07T07:18:52.4973759Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:18:52.4975081Z from pkg_resources import resource_filename 2025-09-07T07:18:53.1700725Z 2025-09-07T07:18:56.3522455Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:18:56.3523047Z loading model: 0it [00:03, ?it/s] 2025-09-07T07:18:56.3546040Z cpu eval XLNetLMHeadModel 2025-09-07T07:18:58.8405925Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:18:59.7841705Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:19:00.7289848Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:19:22.4183839Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4185608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4186168Z return mod(**inputs) 2025-09-07T07:19:22.4191700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4196612Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4197468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-09-07T07:19:22.4198086Z word_emb_k = self.word_embedding(input_ids) 2025-09-07T07:19:22.4204353Z 2025-09-07T07:19:22.4204895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4205367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4205847Z return mod(**inputs) 2025-09-07T07:19:22.4206292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4206776Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4207228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-09-07T07:19:22.4207745Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-09-07T07:19:22.4208297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-09-07T07:19:22.4208842Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-09-07T07:19:22.4209363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-09-07T07:19:22.4209933Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-09-07T07:19:22.4210185Z 2025-09-07T07:19:22.4210314Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4210728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4211092Z return mod(**inputs) 2025-09-07T07:19:22.4211722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4212186Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4212633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-09-07T07:19:22.4213127Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-09-07T07:19:22.4213667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-09-07T07:19:22.4214207Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-09-07T07:19:22.4214731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-09-07T07:19:22.4215355Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-09-07T07:19:22.4215584Z 2025-09-07T07:19:22.4215711Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4216117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4216489Z return mod(**inputs) 2025-09-07T07:19:22.4216917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4217371Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4217815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4218236Z outputs = layer_module( 2025-09-07T07:19:22.4218649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4219077Z outputs = self.rel_attn( 2025-09-07T07:19:22.4219515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4220288Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4220797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4221311Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4221526Z 2025-09-07T07:19:22.4221653Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4222071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4222440Z return mod(**inputs) 2025-09-07T07:19:22.4222882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4223332Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4223780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4224216Z outputs = layer_module( 2025-09-07T07:19:22.4224621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4225055Z outputs = self.rel_attn( 2025-09-07T07:19:22.4225471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4226141Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4226618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4227122Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4227316Z 2025-09-07T07:19:22.4227431Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4227827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4228226Z return mod(**inputs) 2025-09-07T07:19:22.4228629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4229063Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4229495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4229916Z outputs = layer_module( 2025-09-07T07:19:22.4230321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4230739Z outputs = self.rel_attn( 2025-09-07T07:19:22.4231144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4231655Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4232123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4232605Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4232786Z 2025-09-07T07:19:22.4232906Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4233309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4233669Z return mod(**inputs) 2025-09-07T07:19:22.4234072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4234505Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4234932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4235354Z outputs = layer_module( 2025-09-07T07:19:22.4235785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4236205Z outputs = self.rel_attn( 2025-09-07T07:19:22.4236601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4237071Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4237532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4238011Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4238188Z 2025-09-07T07:19:22.4238309Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4238694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4239049Z return mod(**inputs) 2025-09-07T07:19:22.4239447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4239882Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4240316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4240737Z outputs = layer_module( 2025-09-07T07:19:22.4241138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4241554Z outputs = self.rel_attn( 2025-09-07T07:19:22.4241961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4242398Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4242855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4243340Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4243546Z 2025-09-07T07:19:22.4243661Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4244051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4244395Z return mod(**inputs) 2025-09-07T07:19:22.4244794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4245229Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4245659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4246073Z outputs = layer_module( 2025-09-07T07:19:22.4246474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4246913Z outputs = self.rel_attn( 2025-09-07T07:19:22.4247316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4247758Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4248219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4248691Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4248880Z 2025-09-07T07:19:22.4248997Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4249389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4249744Z return mod(**inputs) 2025-09-07T07:19:22.4250129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4250561Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4250997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4251409Z outputs = layer_module( 2025-09-07T07:19:22.4251826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4252267Z outputs = self.rel_attn( 2025-09-07T07:19:22.4252671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4253106Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4253560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4254038Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4254219Z 2025-09-07T07:19:22.4254333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4254738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4255103Z return mod(**inputs) 2025-09-07T07:19:22.4255516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4255971Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4256407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4256823Z outputs = layer_module( 2025-09-07T07:19:22.4257204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4257608Z outputs = self.rel_attn( 2025-09-07T07:19:22.4258000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4258437Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4258898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4259392Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4259568Z 2025-09-07T07:19:22.4259689Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4260074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4260423Z return mod(**inputs) 2025-09-07T07:19:22.4260815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4261245Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4261693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4262143Z outputs = layer_module( 2025-09-07T07:19:22.4262543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4262961Z outputs = self.rel_attn( 2025-09-07T07:19:22.4263369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4263828Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4264285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4264762Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4264941Z 2025-09-07T07:19:22.4265060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4265454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4265928Z return mod(**inputs) 2025-09-07T07:19:22.4266377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4266836Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4267291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4267733Z outputs = layer_module( 2025-09-07T07:19:22.4268131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4268546Z outputs = self.rel_attn( 2025-09-07T07:19:22.4268956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4269409Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4269856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4270350Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4270539Z 2025-09-07T07:19:22.4270654Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4271055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4271420Z return mod(**inputs) 2025-09-07T07:19:22.4271829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4272269Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4272717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4273162Z outputs = layer_module( 2025-09-07T07:19:22.4273567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4274001Z outputs = self.rel_attn( 2025-09-07T07:19:22.4274419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4274890Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4275361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4275840Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4276044Z 2025-09-07T07:19:22.4276158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4276561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4276919Z return mod(**inputs) 2025-09-07T07:19:22.4277332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4277779Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4278210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4278625Z outputs = layer_module( 2025-09-07T07:19:22.4279023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4279448Z outputs = self.rel_attn( 2025-09-07T07:19:22.4279857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4280308Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4280767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4281249Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4281432Z 2025-09-07T07:19:22.4281554Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4281969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4282351Z return mod(**inputs) 2025-09-07T07:19:22.4282753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4283226Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4283662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4284107Z outputs = layer_module( 2025-09-07T07:19:22.4284513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4284950Z outputs = self.rel_attn( 2025-09-07T07:19:22.4285380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4285817Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4286292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4286789Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4286970Z 2025-09-07T07:19:22.4287097Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4287501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4287854Z return mod(**inputs) 2025-09-07T07:19:22.4288260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4288710Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4289150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4289577Z outputs = layer_module( 2025-09-07T07:19:22.4289994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4290459Z outputs = self.rel_attn( 2025-09-07T07:19:22.4290874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4291328Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4291789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4292258Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4292434Z 2025-09-07T07:19:22.4292544Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4292911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4293268Z return mod(**inputs) 2025-09-07T07:19:22.4293637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4294071Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4294499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4294914Z outputs = layer_module( 2025-09-07T07:19:22.4295305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4295734Z outputs = self.rel_attn( 2025-09-07T07:19:22.4296132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4296562Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4297016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4297489Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4297692Z 2025-09-07T07:19:22.4297807Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4298199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4298573Z return mod(**inputs) 2025-09-07T07:19:22.4298994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4299436Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4299890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4300383Z outputs = layer_module( 2025-09-07T07:19:22.4300872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4301306Z outputs = self.rel_attn( 2025-09-07T07:19:22.4301726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4302177Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4302654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4303151Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4303335Z 2025-09-07T07:19:22.4303450Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4303856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4304223Z return mod(**inputs) 2025-09-07T07:19:22.4304635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4305089Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4305528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4306055Z outputs = layer_module( 2025-09-07T07:19:22.4306473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4306909Z outputs = self.rel_attn( 2025-09-07T07:19:22.4307317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4307752Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4308191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4308650Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4308839Z 2025-09-07T07:19:22.4308955Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4309340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4309678Z return mod(**inputs) 2025-09-07T07:19:22.4310052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4310460Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4310866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4311250Z outputs = layer_module( 2025-09-07T07:19:22.4311626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4312016Z outputs = self.rel_attn( 2025-09-07T07:19:22.4312394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4312800Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4313252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4313707Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4313877Z 2025-09-07T07:19:22.4314011Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4314381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4314724Z return mod(**inputs) 2025-09-07T07:19:22.4315119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4315553Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4315987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4316412Z outputs = layer_module( 2025-09-07T07:19:22.4316782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4317188Z outputs = self.rel_attn( 2025-09-07T07:19:22.4317586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4318022Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4318465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4318941Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4319126Z 2025-09-07T07:19:22.4319240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4319842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4320209Z return mod(**inputs) 2025-09-07T07:19:22.4320605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4321103Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4321537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4321957Z outputs = layer_module( 2025-09-07T07:19:22.4322358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4322775Z outputs = self.rel_attn( 2025-09-07T07:19:22.4323186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4323632Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4324096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4324615Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4324804Z 2025-09-07T07:19:22.4324921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4325313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4325683Z return mod(**inputs) 2025-09-07T07:19:22.4326079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4326520Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4326950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4327379Z outputs = layer_module( 2025-09-07T07:19:22.4327778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4328198Z outputs = self.rel_attn( 2025-09-07T07:19:22.4328621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4329055Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4329529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4330023Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4330220Z 2025-09-07T07:19:22.4330341Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4330739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4331097Z return mod(**inputs) 2025-09-07T07:19:22.4331493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4331941Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4332378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4332812Z outputs = layer_module( 2025-09-07T07:19:22.4333217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4333632Z outputs = self.rel_attn( 2025-09-07T07:19:22.4334034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4334470Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4334938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4335434Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4335614Z 2025-09-07T07:19:22.4335736Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4336130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4336481Z return mod(**inputs) 2025-09-07T07:19:22.4336854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4337266Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4337674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4338062Z outputs = layer_module( 2025-09-07T07:19:22.4338438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4338833Z outputs = self.rel_attn( 2025-09-07T07:19:22.4339214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4339644Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4340067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4340528Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4340700Z 2025-09-07T07:19:22.4340809Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4341175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4341499Z return mod(**inputs) 2025-09-07T07:19:22.4341871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4342281Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4342688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4343091Z outputs = layer_module( 2025-09-07T07:19:22.4343478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4343874Z outputs = self.rel_attn( 2025-09-07T07:19:22.4344253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4344721Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4345182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4345658Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4345915Z 2025-09-07T07:19:22.4346037Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4346439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4346806Z return mod(**inputs) 2025-09-07T07:19:22.4347192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4347602Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4348007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4348402Z outputs = layer_module( 2025-09-07T07:19:22.4348775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4349159Z outputs = self.rel_attn( 2025-09-07T07:19:22.4349539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4349965Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4350118Z 2025-09-07T07:19:22.4350235Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4350602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4350990Z return mod(**inputs) 2025-09-07T07:19:22.4351372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4351774Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4352168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4352543Z outputs = layer_module( 2025-09-07T07:19:22.4352910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4353289Z outputs = self.rel_attn( 2025-09-07T07:19:22.4353656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4354147Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4354304Z 2025-09-07T07:19:22.4354417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4354808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4355160Z return mod(**inputs) 2025-09-07T07:19:22.4355556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4355989Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4356422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4356806Z outputs = layer_module( 2025-09-07T07:19:22.4357177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4357563Z outputs = self.rel_attn( 2025-09-07T07:19:22.4357927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4358342Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4358749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4359244Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4359440Z 2025-09-07T07:19:22.4359556Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4359920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4360256Z return mod(**inputs) 2025-09-07T07:19:22.4360621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4361025Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4361422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-09-07T07:19:22.4361865Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-09-07T07:19:22.4362365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-09-07T07:19:22.4362859Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-09-07T07:19:22.4363339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-09-07T07:19:22.4363845Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-09-07T07:19:22.4364052Z 2025-09-07T07:19:22.4364157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4364550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4364903Z return mod(**inputs) 2025-09-07T07:19:22.4365303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4365753Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4366191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4366585Z outputs = layer_module( 2025-09-07T07:19:22.4366982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4367398Z outputs = self.rel_attn( 2025-09-07T07:19:22.4367794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4368300Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4368536Z 2025-09-07T07:19:22.4368651Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4369051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4369408Z return mod(**inputs) 2025-09-07T07:19:22.4369802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4370227Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4370640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4371036Z outputs = layer_module( 2025-09-07T07:19:22.4372043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4372458Z outputs = self.rel_attn( 2025-09-07T07:19:22.4372865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4373296Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4373762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4374231Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4374428Z 2025-09-07T07:19:22.4374553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4374923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4375257Z return mod(**inputs) 2025-09-07T07:19:22.4375660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4376098Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4376539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4376968Z outputs = layer_module( 2025-09-07T07:19:22.4377369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4377795Z outputs = self.rel_attn( 2025-09-07T07:19:22.4378193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4378620Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4378785Z 2025-09-07T07:19:22.4378891Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4379258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4379612Z return mod(**inputs) 2025-09-07T07:19:22.4380018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4380455Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4380885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4381331Z outputs = layer_module( 2025-09-07T07:19:22.4381720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4382137Z outputs = self.rel_attn( 2025-09-07T07:19:22.4382556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4382987Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4383422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4383904Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4384100Z 2025-09-07T07:19:22.4384237Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4384628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4384995Z return mod(**inputs) 2025-09-07T07:19:22.4385396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4385938Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4386390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4386821Z outputs = layer_module( 2025-09-07T07:19:22.4387228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4387653Z outputs = self.rel_attn( 2025-09-07T07:19:22.4388050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4388484Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4388973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4389463Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4389642Z 2025-09-07T07:19:22.4389760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4390172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4390529Z return mod(**inputs) 2025-09-07T07:19:22.4390920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4391356Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4391792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4392211Z outputs = layer_module( 2025-09-07T07:19:22.4392608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4393019Z outputs = self.rel_attn( 2025-09-07T07:19:22.4393417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4393854Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4394322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4394808Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4394989Z 2025-09-07T07:19:22.4395112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4395493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4395853Z return mod(**inputs) 2025-09-07T07:19:22.4396254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4396748Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4397172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4397589Z outputs = layer_module( 2025-09-07T07:19:22.4397984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4398558Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4399136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4399566Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4400016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4400445Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4400868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4401291Z output = self.layer_1(output) 2025-09-07T07:19:22.4401428Z 2025-09-07T07:19:22.4401542Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4401938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4402299Z return mod(**inputs) 2025-09-07T07:19:22.4402698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4403132Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4403555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4403970Z outputs = layer_module( 2025-09-07T07:19:22.4404403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4404976Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4405558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4405996Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4406425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4406850Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4407261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4407690Z output = self.activation_function(output) 2025-09-07T07:19:22.4408082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4408459Z return self.act(input) 2025-09-07T07:19:22.4408582Z 2025-09-07T07:19:22.4408704Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4409100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4409447Z return mod(**inputs) 2025-09-07T07:19:22.4409842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4410270Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4410701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4411105Z outputs = layer_module( 2025-09-07T07:19:22.4411507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4412073Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4412672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4413108Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4413521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4413940Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4414346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4414762Z output = self.layer_2(output) 2025-09-07T07:19:22.4414897Z 2025-09-07T07:19:22.4415015Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4415434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4415792Z return mod(**inputs) 2025-09-07T07:19:22.4416189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4416622Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4417049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4417471Z outputs = layer_module( 2025-09-07T07:19:22.4417870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4418284Z outputs = self.rel_attn( 2025-09-07T07:19:22.4418689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4419137Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4419309Z 2025-09-07T07:19:22.4419424Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4420101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4420465Z return mod(**inputs) 2025-09-07T07:19:22.4420885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4421314Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4421746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4422166Z outputs = layer_module( 2025-09-07T07:19:22.4422568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4422979Z outputs = self.rel_attn( 2025-09-07T07:19:22.4423387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4423847Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4424016Z 2025-09-07T07:19:22.4424137Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4424528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4424887Z return mod(**inputs) 2025-09-07T07:19:22.4425308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4425807Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4426266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4426694Z outputs = layer_module( 2025-09-07T07:19:22.4427092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4427520Z outputs = self.rel_attn( 2025-09-07T07:19:22.4427942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4428408Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4428845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4429363Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4429583Z 2025-09-07T07:19:22.4429700Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4430101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4430477Z return mod(**inputs) 2025-09-07T07:19:22.4430874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4431343Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4431804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4432232Z outputs = layer_module( 2025-09-07T07:19:22.4432640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4433053Z outputs = self.rel_attn( 2025-09-07T07:19:22.4433462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4433971Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4434181Z 2025-09-07T07:19:22.4434304Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4434707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4435079Z return mod(**inputs) 2025-09-07T07:19:22.4435500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4435949Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4436400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4436853Z outputs = layer_module( 2025-09-07T07:19:22.4437414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4437864Z outputs = self.rel_attn( 2025-09-07T07:19:22.4438282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4438685Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4439111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4439623Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4439839Z 2025-09-07T07:19:22.4439957Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4440346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4440707Z return mod(**inputs) 2025-09-07T07:19:22.4441099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4441506Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4441914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4442311Z outputs = layer_module( 2025-09-07T07:19:22.4442679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4443071Z outputs = self.rel_attn( 2025-09-07T07:19:22.4443450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4443908Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4444066Z 2025-09-07T07:19:22.4444183Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4444564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4444917Z return mod(**inputs) 2025-09-07T07:19:22.4445334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4445776Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4446222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4446652Z outputs = layer_module( 2025-09-07T07:19:22.4447052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4447472Z outputs = self.rel_attn( 2025-09-07T07:19:22.4447871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4448286Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4448721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4449209Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4449402Z 2025-09-07T07:19:22.4449528Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4449899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4450228Z return mod(**inputs) 2025-09-07T07:19:22.4450606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4451039Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4451459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4451889Z outputs = layer_module( 2025-09-07T07:19:22.4452291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4452710Z outputs = self.rel_attn( 2025-09-07T07:19:22.4453110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4453543Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4453979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4454443Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4454634Z 2025-09-07T07:19:22.4454748Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4455137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4455490Z return mod(**inputs) 2025-09-07T07:19:22.4455880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4456311Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4456738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4457156Z outputs = layer_module( 2025-09-07T07:19:22.4457545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4457961Z outputs = self.rel_attn( 2025-09-07T07:19:22.4458359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4458829Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4459282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4459759Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4459945Z 2025-09-07T07:19:22.4460056Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4460443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4460797Z return mod(**inputs) 2025-09-07T07:19:22.4461190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4461644Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4462074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4462493Z outputs = layer_module( 2025-09-07T07:19:22.4462890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4463455Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4464029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4464464Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4464902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4465340Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4465819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4466288Z output = self.layer_1(output) 2025-09-07T07:19:22.4466437Z 2025-09-07T07:19:22.4466553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4466954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4468250Z return mod(**inputs) 2025-09-07T07:19:22.4468656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4469091Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4469526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4469935Z outputs = layer_module( 2025-09-07T07:19:22.4470324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4470890Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4471462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4471894Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4472321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4472710Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4473097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4473510Z output = self.activation_function(output) 2025-09-07T07:19:22.4473881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4474236Z return self.act(input) 2025-09-07T07:19:22.4474351Z 2025-09-07T07:19:22.4474459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4474871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4475229Z return mod(**inputs) 2025-09-07T07:19:22.4475624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4476053Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4476478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4476895Z outputs = layer_module( 2025-09-07T07:19:22.4477302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4477839Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4478394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4478801Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4479200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4479598Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4479988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4480375Z output = self.layer_2(output) 2025-09-07T07:19:22.4480512Z 2025-09-07T07:19:22.4480618Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4480992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4481327Z return mod(**inputs) 2025-09-07T07:19:22.4481703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4482121Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4482531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4482924Z outputs = layer_module( 2025-09-07T07:19:22.4483321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4483708Z outputs = self.rel_attn( 2025-09-07T07:19:22.4484094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4484532Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4484698Z 2025-09-07T07:19:22.4484818Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4485213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4485578Z return mod(**inputs) 2025-09-07T07:19:22.4485978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4486411Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4486845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4487266Z outputs = layer_module( 2025-09-07T07:19:22.4487655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4488058Z outputs = self.rel_attn( 2025-09-07T07:19:22.4488439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4488873Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4489037Z 2025-09-07T07:19:22.4489149Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4489557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4489905Z return mod(**inputs) 2025-09-07T07:19:22.4490300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4490729Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4491145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4491569Z outputs = layer_module( 2025-09-07T07:19:22.4491967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4492393Z outputs = self.rel_attn( 2025-09-07T07:19:22.4492813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4493237Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4493680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4494181Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4494387Z 2025-09-07T07:19:22.4494508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4494894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4495270Z return mod(**inputs) 2025-09-07T07:19:22.4495668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4496111Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4496545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4496955Z outputs = layer_module( 2025-09-07T07:19:22.4497375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4497787Z outputs = self.rel_attn( 2025-09-07T07:19:22.4498213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4498671Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4498862Z 2025-09-07T07:19:22.4498969Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4499343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4499673Z return mod(**inputs) 2025-09-07T07:19:22.4500046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4500448Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4500857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4501253Z outputs = layer_module( 2025-09-07T07:19:22.4501631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4502021Z outputs = self.rel_attn( 2025-09-07T07:19:22.4502392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4502788Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4503200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4503680Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4503884Z 2025-09-07T07:19:22.4504004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4504386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4504760Z return mod(**inputs) 2025-09-07T07:19:22.4505152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4505587Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4506098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4506520Z outputs = layer_module( 2025-09-07T07:19:22.4506919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4507333Z outputs = self.rel_attn( 2025-09-07T07:19:22.4507755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4508177Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4508349Z 2025-09-07T07:19:22.4508458Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4508836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4509212Z return mod(**inputs) 2025-09-07T07:19:22.4509610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4510047Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4510478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4510881Z outputs = layer_module( 2025-09-07T07:19:22.4511258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4511643Z outputs = self.rel_attn( 2025-09-07T07:19:22.4512046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4512449Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4512876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4513339Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4513520Z 2025-09-07T07:19:22.4513625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4513995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4514361Z return mod(**inputs) 2025-09-07T07:19:22.4514771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4515205Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4515628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4516023Z outputs = layer_module( 2025-09-07T07:19:22.4516398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4516805Z outputs = self.rel_attn( 2025-09-07T07:19:22.4517197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4517637Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4518098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4518582Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4518764Z 2025-09-07T07:19:22.4518885Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4519265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4519838Z return mod(**inputs) 2025-09-07T07:19:22.4520243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4520686Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4521122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4521532Z outputs = layer_module( 2025-09-07T07:19:22.4521934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4522356Z outputs = self.rel_attn( 2025-09-07T07:19:22.4522760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4523238Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4523697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4524178Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4524357Z 2025-09-07T07:19:22.4524478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4524864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4525208Z return mod(**inputs) 2025-09-07T07:19:22.4525600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4526032Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4526461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4526877Z outputs = layer_module( 2025-09-07T07:19:22.4527290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4527859Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4528453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4528898Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4529343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4529762Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4530153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4530555Z output = self.layer_1(output) 2025-09-07T07:19:22.4530683Z 2025-09-07T07:19:22.4530798Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4531164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4531500Z return mod(**inputs) 2025-09-07T07:19:22.4531877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4532289Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4532692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4533076Z outputs = layer_module( 2025-09-07T07:19:22.4533457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4533990Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4534544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4534989Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4535390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4535815Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4536223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4536659Z output = self.activation_function(output) 2025-09-07T07:19:22.4537042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4537406Z return self.act(input) 2025-09-07T07:19:22.4537528Z 2025-09-07T07:19:22.4537636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4538024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4538363Z return mod(**inputs) 2025-09-07T07:19:22.4538736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4539159Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4539592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4540008Z outputs = layer_module( 2025-09-07T07:19:22.4540409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4540966Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4541544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4541990Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4542449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4542865Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4543291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4543723Z output = self.layer_2(output) 2025-09-07T07:19:22.4543860Z 2025-09-07T07:19:22.4543985Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4544388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4544742Z return mod(**inputs) 2025-09-07T07:19:22.4545153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4545598Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4546110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4546543Z outputs = layer_module( 2025-09-07T07:19:22.4546942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4547369Z outputs = self.rel_attn( 2025-09-07T07:19:22.4547779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4548246Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4548420Z 2025-09-07T07:19:22.4548537Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4548941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4549303Z return mod(**inputs) 2025-09-07T07:19:22.4549711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4550163Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4550625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4551046Z outputs = layer_module( 2025-09-07T07:19:22.4551455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4551888Z outputs = self.rel_attn( 2025-09-07T07:19:22.4552302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4552768Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4552946Z 2025-09-07T07:19:22.4553060Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4553478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4553841Z return mod(**inputs) 2025-09-07T07:19:22.4554258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4554707Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4555110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4555510Z outputs = layer_module( 2025-09-07T07:19:22.4555902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4556323Z outputs = self.rel_attn( 2025-09-07T07:19:22.4556755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4557198Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4557624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4558117Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4558315Z 2025-09-07T07:19:22.4558421Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4558829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4559180Z return mod(**inputs) 2025-09-07T07:19:22.4559575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4559998Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4560433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4560857Z outputs = layer_module( 2025-09-07T07:19:22.4561269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4561682Z outputs = self.rel_attn( 2025-09-07T07:19:22.4562075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4562553Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4562757Z 2025-09-07T07:19:22.4562866Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4563252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4563602Z return mod(**inputs) 2025-09-07T07:19:22.4563986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4564414Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4564820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4565210Z outputs = layer_module( 2025-09-07T07:19:22.4565616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4566028Z outputs = self.rel_attn( 2025-09-07T07:19:22.4566430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4566850Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4567282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4567770Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4567986Z 2025-09-07T07:19:22.4568093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4568480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4568835Z return mod(**inputs) 2025-09-07T07:19:22.4569230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4569654Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4570087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4570502Z outputs = layer_module( 2025-09-07T07:19:22.4570878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4571262Z outputs = self.rel_attn( 2025-09-07T07:19:22.4571647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4572098Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4572274Z 2025-09-07T07:19:22.4572388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4572774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4573104Z return mod(**inputs) 2025-09-07T07:19:22.4573484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4573943Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4574367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4574780Z outputs = layer_module( 2025-09-07T07:19:22.4575170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4575584Z outputs = self.rel_attn( 2025-09-07T07:19:22.4575993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4576389Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4576803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4577292Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4577492Z 2025-09-07T07:19:22.4577605Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4577993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4578343Z return mod(**inputs) 2025-09-07T07:19:22.4578729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4579158Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4579581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4579993Z outputs = layer_module( 2025-09-07T07:19:22.4580389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4580813Z outputs = self.rel_attn( 2025-09-07T07:19:22.4581210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4581641Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4582092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4582566Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4582752Z 2025-09-07T07:19:22.4582869Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4583251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4583622Z return mod(**inputs) 2025-09-07T07:19:22.4584019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4584449Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4584885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4585305Z outputs = layer_module( 2025-09-07T07:19:22.4585786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4586228Z outputs = self.rel_attn( 2025-09-07T07:19:22.4586632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4587085Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4587559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4588038Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4588243Z 2025-09-07T07:19:22.4588359Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4588751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4589122Z return mod(**inputs) 2025-09-07T07:19:22.4589523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4589950Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4590374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4590788Z outputs = layer_module( 2025-09-07T07:19:22.4591164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4591700Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4592245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4592638Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4593026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4593413Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4593788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4594190Z output = self.layer_1(output) 2025-09-07T07:19:22.4594320Z 2025-09-07T07:19:22.4594425Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4594798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4595146Z return mod(**inputs) 2025-09-07T07:19:22.4595542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4596025Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4596454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4596869Z outputs = layer_module( 2025-09-07T07:19:22.4597245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4597775Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4598303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4598731Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4599141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4599543Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4599934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4600346Z output = self.activation_function(output) 2025-09-07T07:19:22.4600717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4601066Z return self.act(input) 2025-09-07T07:19:22.4601180Z 2025-09-07T07:19:22.4601292Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4601655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4601993Z return mod(**inputs) 2025-09-07T07:19:22.4602379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4602814Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4603225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4603617Z outputs = layer_module( 2025-09-07T07:19:22.4603999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4604529Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4605093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4605526Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4605945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4606363Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4606757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4607162Z output = self.layer_2(output) 2025-09-07T07:19:22.4607290Z 2025-09-07T07:19:22.4607399Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4607777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4608139Z return mod(**inputs) 2025-09-07T07:19:22.4608544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4609010Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4609427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4609820Z outputs = layer_module( 2025-09-07T07:19:22.4610197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4610614Z outputs = self.rel_attn( 2025-09-07T07:19:22.4610994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4611411Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4611579Z 2025-09-07T07:19:22.4611686Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4612057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4612409Z return mod(**inputs) 2025-09-07T07:19:22.4612798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4613254Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4613710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4614150Z outputs = layer_module( 2025-09-07T07:19:22.4614547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4614966Z outputs = self.rel_attn( 2025-09-07T07:19:22.4615395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4615843Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4616011Z 2025-09-07T07:19:22.4616131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4616516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4616860Z return mod(**inputs) 2025-09-07T07:19:22.4617259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4617703Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4618149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4618555Z outputs = layer_module( 2025-09-07T07:19:22.4618962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4619375Z outputs = self.rel_attn( 2025-09-07T07:19:22.4620004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4620413Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4620819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4621317Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4621532Z 2025-09-07T07:19:22.4621648Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4622046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4622399Z return mod(**inputs) 2025-09-07T07:19:22.4622790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4623225Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4623657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4624073Z outputs = layer_module( 2025-09-07T07:19:22.4624460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4624876Z outputs = self.rel_attn( 2025-09-07T07:19:22.4625276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4625893Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4626103Z 2025-09-07T07:19:22.4626227Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4626613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4626963Z return mod(**inputs) 2025-09-07T07:19:22.4627246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4627336Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4627614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4627698Z outputs = layer_module( 2025-09-07T07:19:22.4627999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4628083Z outputs = self.rel_attn( 2025-09-07T07:19:22.4628361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4628440Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4628743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4628883Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4628887Z 2025-09-07T07:19:22.4629008Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4629221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4629297Z return mod(**inputs) 2025-09-07T07:19:22.4629579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4629667Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4629983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4630057Z outputs = layer_module( 2025-09-07T07:19:22.4630359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4630434Z outputs = self.rel_attn( 2025-09-07T07:19:22.4630704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4630823Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4630826Z 2025-09-07T07:19:22.4630936Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4631155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4631225Z return mod(**inputs) 2025-09-07T07:19:22.4631508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4631598Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4631870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4631949Z outputs = layer_module( 2025-09-07T07:19:22.4632220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4632299Z outputs = self.rel_attn( 2025-09-07T07:19:22.4632568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4632644Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4632944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4633078Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4633100Z 2025-09-07T07:19:22.4633220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4633433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4633510Z return mod(**inputs) 2025-09-07T07:19:22.4633783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4633872Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4634172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4634244Z outputs = layer_module( 2025-09-07T07:19:22.4634522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4634612Z outputs = self.rel_attn( 2025-09-07T07:19:22.4634881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4634987Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4635286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4635415Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4635419Z 2025-09-07T07:19:22.4635529Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4635745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4635823Z return mod(**inputs) 2025-09-07T07:19:22.4636096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4636192Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4636478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4636557Z outputs = layer_module( 2025-09-07T07:19:22.4636833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4636903Z outputs = self.rel_attn( 2025-09-07T07:19:22.4637167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4637257Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4637547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4637660Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4637666Z 2025-09-07T07:19:22.4637771Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4637981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4638051Z return mod(**inputs) 2025-09-07T07:19:22.4638324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4638411Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4638676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4638745Z outputs = layer_module( 2025-09-07T07:19:22.4639002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4639222Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4639493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4639601Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4639861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4639936Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4640204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4640280Z output = self.layer_1(output) 2025-09-07T07:19:22.4640284Z 2025-09-07T07:19:22.4640399Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4640604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4640679Z return mod(**inputs) 2025-09-07T07:19:22.4640970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4641056Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4641328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4641398Z outputs = layer_module( 2025-09-07T07:19:22.4641670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4641881Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4642151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4642240Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4642504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4642587Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4642863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4642962Z output = self.activation_function(output) 2025-09-07T07:19:22.4643199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4643271Z return self.act(input) 2025-09-07T07:19:22.4643275Z 2025-09-07T07:19:22.4643388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4643591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4643664Z return mod(**inputs) 2025-09-07T07:19:22.4643928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4644013Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4644283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4644354Z outputs = layer_module( 2025-09-07T07:19:22.4644619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4644831Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4645101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4645180Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4645442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4645527Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4645798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4645884Z output = self.layer_2(output) 2025-09-07T07:19:22.4645906Z 2025-09-07T07:19:22.4646020Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4646244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4646322Z return mod(**inputs) 2025-09-07T07:19:22.4646611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4646707Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4646992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4647065Z outputs = layer_module( 2025-09-07T07:19:22.4647354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4647441Z outputs = self.rel_attn( 2025-09-07T07:19:22.4647706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4647806Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4647810Z 2025-09-07T07:19:22.4647921Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4648124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4648190Z return mod(**inputs) 2025-09-07T07:19:22.4648452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4648534Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4648807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4648876Z outputs = layer_module( 2025-09-07T07:19:22.4649164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4649250Z outputs = self.rel_attn( 2025-09-07T07:19:22.4649525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4649656Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4649660Z 2025-09-07T07:19:22.4649776Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4649986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4650052Z return mod(**inputs) 2025-09-07T07:19:22.4650315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4650405Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4650671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4650749Z outputs = layer_module( 2025-09-07T07:19:22.4651003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4651073Z outputs = self.rel_attn( 2025-09-07T07:19:22.4651340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4651412Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4651698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4651832Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4651836Z 2025-09-07T07:19:22.4651945Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4652148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4652212Z return mod(**inputs) 2025-09-07T07:19:22.4652494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4652577Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4652851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4652922Z outputs = layer_module( 2025-09-07T07:19:22.4653182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4653261Z outputs = self.rel_attn( 2025-09-07T07:19:22.4653517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4653689Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4653692Z 2025-09-07T07:19:22.4653797Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4653999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4654074Z return mod(**inputs) 2025-09-07T07:19:22.4654346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4654444Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4654720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4654802Z outputs = layer_module( 2025-09-07T07:19:22.4655076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4655154Z outputs = self.rel_attn( 2025-09-07T07:19:22.4655436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4655534Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4655834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4656009Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4656012Z 2025-09-07T07:19:22.4656131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4656339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4656406Z return mod(**inputs) 2025-09-07T07:19:22.4656678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4656766Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4657049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4657120Z outputs = layer_module( 2025-09-07T07:19:22.4657392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4657471Z outputs = self.rel_attn( 2025-09-07T07:19:22.4657745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4657860Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4657863Z 2025-09-07T07:19:22.4657971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4658183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4658262Z return mod(**inputs) 2025-09-07T07:19:22.4658531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4658627Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4658903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4658997Z outputs = layer_module( 2025-09-07T07:19:22.4659280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4659355Z outputs = self.rel_attn( 2025-09-07T07:19:22.4659640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4659714Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4660014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4660148Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4660166Z 2025-09-07T07:19:22.4660276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4660493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4660566Z return mod(**inputs) 2025-09-07T07:19:22.4660845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4660933Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4661205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4661284Z outputs = layer_module( 2025-09-07T07:19:22.4661552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4661632Z outputs = self.rel_attn( 2025-09-07T07:19:22.4661904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4662006Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4662324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4662447Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4662451Z 2025-09-07T07:19:22.4662585Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4662800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4662878Z return mod(**inputs) 2025-09-07T07:19:22.4663151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4663239Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4663517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4663592Z outputs = layer_module( 2025-09-07T07:19:22.4663872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4663946Z outputs = self.rel_attn( 2025-09-07T07:19:22.4664224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4664330Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4664634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4664765Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4664769Z 2025-09-07T07:19:22.4664881Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4665108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4665181Z return mod(**inputs) 2025-09-07T07:19:22.4665463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4665580Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4665956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4666044Z outputs = layer_module( 2025-09-07T07:19:22.4666327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4666560Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4666861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4666972Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4667263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4667346Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4667642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4667722Z output = self.layer_1(output) 2025-09-07T07:19:22.4667727Z 2025-09-07T07:19:22.4667838Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4668062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4668133Z return mod(**inputs) 2025-09-07T07:19:22.4668426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4668514Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4668792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4668886Z outputs = layer_module( 2025-09-07T07:19:22.4669161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4669406Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4669688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4669778Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4670053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4670130Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4670409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4670507Z output = self.activation_function(output) 2025-09-07T07:19:22.4670748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4670822Z return self.act(input) 2025-09-07T07:19:22.4670826Z 2025-09-07T07:19:22.4670937Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4671159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4671230Z return mod(**inputs) 2025-09-07T07:19:22.4671511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4671598Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4671878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4671952Z outputs = layer_module( 2025-09-07T07:19:22.4672231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4672465Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4672733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4672819Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4673084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4673158Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4673427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4673502Z output = self.layer_2(output) 2025-09-07T07:19:22.4673520Z 2025-09-07T07:19:22.4673631Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4673834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4673909Z return mod(**inputs) 2025-09-07T07:19:22.4674165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4674251Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4674517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4674583Z outputs = layer_module( 2025-09-07T07:19:22.4674844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4674917Z outputs = self.rel_attn( 2025-09-07T07:19:22.4675171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4675298Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4675303Z 2025-09-07T07:19:22.4675419Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4675632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4675698Z return mod(**inputs) 2025-09-07T07:19:22.4675990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4676088Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4676359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4676437Z outputs = layer_module( 2025-09-07T07:19:22.4676706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4676782Z outputs = self.rel_attn( 2025-09-07T07:19:22.4677038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4677146Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4677150Z 2025-09-07T07:19:22.4677265Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4677481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4677559Z return mod(**inputs) 2025-09-07T07:19:22.4677839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4677924Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4678190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4678259Z outputs = layer_module( 2025-09-07T07:19:22.4678523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4678622Z outputs = self.rel_attn( 2025-09-07T07:19:22.4678883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4678955Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4679231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4679377Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4679381Z 2025-09-07T07:19:22.4679485Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4679694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4679777Z return mod(**inputs) 2025-09-07T07:19:22.4680041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4680135Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4680402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4680475Z outputs = layer_module( 2025-09-07T07:19:22.4680737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4680806Z outputs = self.rel_attn( 2025-09-07T07:19:22.4681075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4681211Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4681215Z 2025-09-07T07:19:22.4681324Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4681531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4681605Z return mod(**inputs) 2025-09-07T07:19:22.4681888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4681974Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4682259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4682329Z outputs = layer_module( 2025-09-07T07:19:22.4682600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4682672Z outputs = self.rel_attn( 2025-09-07T07:19:22.4682942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4683025Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4683324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4683470Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4683473Z 2025-09-07T07:19:22.4683582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4683800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4683867Z return mod(**inputs) 2025-09-07T07:19:22.4684139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4684234Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4684505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4684584Z outputs = layer_module( 2025-09-07T07:19:22.4684854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4684948Z outputs = self.rel_attn( 2025-09-07T07:19:22.4685228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4685336Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4685341Z 2025-09-07T07:19:22.4685460Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4685679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4685757Z return mod(**inputs) 2025-09-07T07:19:22.4686056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4686143Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4686433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4686503Z outputs = layer_module( 2025-09-07T07:19:22.4686776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4686847Z outputs = self.rel_attn( 2025-09-07T07:19:22.4687122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4687202Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4687501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4687638Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4687642Z 2025-09-07T07:19:22.4687749Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4687966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4688036Z return mod(**inputs) 2025-09-07T07:19:22.4688342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4688441Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4688738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4688818Z outputs = layer_module( 2025-09-07T07:19:22.4689091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4689162Z outputs = self.rel_attn( 2025-09-07T07:19:22.4689452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4689547Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4689847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4689965Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4689969Z 2025-09-07T07:19:22.4690077Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4690341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4690435Z return mod(**inputs) 2025-09-07T07:19:22.4690741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4691130Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4691456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4691587Z outputs = layer_module( 2025-09-07T07:19:22.4691876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4692000Z outputs = self.rel_attn( 2025-09-07T07:19:22.4692295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4692465Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4692782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4692912Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4692949Z 2025-09-07T07:19:22.4693070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4693294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4693420Z return mod(**inputs) 2025-09-07T07:19:22.4693711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4693854Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4694135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4694229Z outputs = layer_module( 2025-09-07T07:19:22.4694524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4694782Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4695117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4695224Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4695560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4695661Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4695961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4696142Z output = self.layer_1(output) 2025-09-07T07:19:22.4696146Z 2025-09-07T07:19:22.4696280Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4713510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4713710Z return mod(**inputs) 2025-09-07T07:19:22.4714065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4714181Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4714478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4714560Z outputs = layer_module( 2025-09-07T07:19:22.4714825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4715059Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4715330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4715414Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4715699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4715783Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4716064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4716164Z output = self.activation_function(output) 2025-09-07T07:19:22.4716400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4716485Z return self.act(input) 2025-09-07T07:19:22.4716547Z 2025-09-07T07:19:22.4716678Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4716920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4717005Z return mod(**inputs) 2025-09-07T07:19:22.4717278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4717368Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4717628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4717707Z outputs = layer_module( 2025-09-07T07:19:22.4717962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4718219Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4718490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4718568Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4718825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4718898Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4719152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4719223Z output = self.layer_2(output) 2025-09-07T07:19:22.4719228Z 2025-09-07T07:19:22.4719340Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4719749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4719831Z return mod(**inputs) 2025-09-07T07:19:22.4720163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4720251Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4720531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4720600Z outputs = layer_module( 2025-09-07T07:19:22.4720853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4720933Z outputs = self.rel_attn( 2025-09-07T07:19:22.4721183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4721293Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4721298Z 2025-09-07T07:19:22.4721404Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4721615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4721683Z return mod(**inputs) 2025-09-07T07:19:22.4721945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4722037Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4722284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4722357Z outputs = layer_module( 2025-09-07T07:19:22.4722601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4722667Z outputs = self.rel_attn( 2025-09-07T07:19:22.4722918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4723023Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4723028Z 2025-09-07T07:19:22.4723166Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4723366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4723430Z return mod(**inputs) 2025-09-07T07:19:22.4723690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4723773Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4724032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4724098Z outputs = layer_module( 2025-09-07T07:19:22.4724355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4724450Z outputs = self.rel_attn( 2025-09-07T07:19:22.4724705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4724788Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4725063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4725210Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4725213Z 2025-09-07T07:19:22.4725319Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4725517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4725590Z return mod(**inputs) 2025-09-07T07:19:22.4725847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4725937Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4726203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4726278Z outputs = layer_module( 2025-09-07T07:19:22.4726529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4726595Z outputs = self.rel_attn( 2025-09-07T07:19:22.4726865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4727004Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4727008Z 2025-09-07T07:19:22.4727117Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4727321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4727387Z return mod(**inputs) 2025-09-07T07:19:22.4727644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4727727Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4727980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4728045Z outputs = layer_module( 2025-09-07T07:19:22.4728290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4728364Z outputs = self.rel_attn( 2025-09-07T07:19:22.4728619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4728701Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4728980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4729125Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4729129Z 2025-09-07T07:19:22.4729236Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4729452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4729527Z return mod(**inputs) 2025-09-07T07:19:22.4729789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4729879Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4730137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4730206Z outputs = layer_module( 2025-09-07T07:19:22.4730471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4730555Z outputs = self.rel_attn( 2025-09-07T07:19:22.4730818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4730926Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4730929Z 2025-09-07T07:19:22.4731050Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4731247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4731312Z return mod(**inputs) 2025-09-07T07:19:22.4731577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4731655Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4731904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4731968Z outputs = layer_module( 2025-09-07T07:19:22.4732220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4732311Z outputs = self.rel_attn( 2025-09-07T07:19:22.4732571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4732652Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4740688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4741050Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4741058Z 2025-09-07T07:19:22.4741204Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4741458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4741545Z return mod(**inputs) 2025-09-07T07:19:22.4741879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4741980Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4742284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4742364Z outputs = layer_module( 2025-09-07T07:19:22.4742667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4742779Z outputs = self.rel_attn( 2025-09-07T07:19:22.4743063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4743174Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4743478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4743656Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4743662Z 2025-09-07T07:19:22.4743783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4744108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4744192Z return mod(**inputs) 2025-09-07T07:19:22.4744483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4744588Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4744873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4744948Z outputs = layer_module( 2025-09-07T07:19:22.4745228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4745304Z outputs = self.rel_attn( 2025-09-07T07:19:22.4745630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4745906Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4746237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4746363Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4746367Z 2025-09-07T07:19:22.4746482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4746712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4746784Z return mod(**inputs) 2025-09-07T07:19:22.4747075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4747167Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4747455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4747537Z outputs = layer_module( 2025-09-07T07:19:22.4747811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4748094Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4748468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4748567Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4748851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4748931Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4749210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4749293Z output = self.layer_1(output) 2025-09-07T07:19:22.4749298Z 2025-09-07T07:19:22.4749426Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4749647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4749719Z return mod(**inputs) 2025-09-07T07:19:22.4750012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4750106Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4750401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4750474Z outputs = layer_module( 2025-09-07T07:19:22.4750744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4750984Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4751281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4751395Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4751672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4751758Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4752031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4752127Z output = self.activation_function(output) 2025-09-07T07:19:22.4752366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4752440Z return self.act(input) 2025-09-07T07:19:22.4752461Z 2025-09-07T07:19:22.4752583Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4752799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4752872Z return mod(**inputs) 2025-09-07T07:19:22.4753155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4753245Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4753527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4753600Z outputs = layer_module( 2025-09-07T07:19:22.4753879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4754103Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4754400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4754489Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4754766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4754850Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4755191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4755271Z output = self.layer_2(output) 2025-09-07T07:19:22.4755283Z 2025-09-07T07:19:22.4755396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4755613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4755690Z return mod(**inputs) 2025-09-07T07:19:22.4755965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4756062Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4756336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4756409Z outputs = layer_module( 2025-09-07T07:19:22.4756690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4756761Z outputs = self.rel_attn( 2025-09-07T07:19:22.4757027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4757130Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4757133Z 2025-09-07T07:19:22.4757240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4757454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4757523Z return mod(**inputs) 2025-09-07T07:19:22.4757791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4757894Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4758167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4758238Z outputs = layer_module( 2025-09-07T07:19:22.4758512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4758592Z outputs = self.rel_attn( 2025-09-07T07:19:22.4758870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4758996Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4759015Z 2025-09-07T07:19:22.4759122Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4759322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4759399Z return mod(**inputs) 2025-09-07T07:19:22.4759660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4759753Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4760013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4760080Z outputs = layer_module( 2025-09-07T07:19:22.4760342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4760412Z outputs = self.rel_attn( 2025-09-07T07:19:22.4760675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4760753Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4761036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4761176Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4761180Z 2025-09-07T07:19:22.4761303Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4761534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4761603Z return mod(**inputs) 2025-09-07T07:19:22.4761869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4761951Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4762214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4762289Z outputs = layer_module( 2025-09-07T07:19:22.4762544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4762619Z outputs = self.rel_attn( 2025-09-07T07:19:22.4762875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4763023Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4763026Z 2025-09-07T07:19:22.4763131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4763334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4763408Z return mod(**inputs) 2025-09-07T07:19:22.4763667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4763758Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4764016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4764102Z outputs = layer_module( 2025-09-07T07:19:22.4764368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4764437Z outputs = self.rel_attn( 2025-09-07T07:19:22.4764705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4764778Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4765058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4765202Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4765205Z 2025-09-07T07:19:22.4765328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4765540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4765610Z return mod(**inputs) 2025-09-07T07:19:22.4765881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4765964Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4766232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4766312Z outputs = layer_module( 2025-09-07T07:19:22.4766572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4766646Z outputs = self.rel_attn( 2025-09-07T07:19:22.4766906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4767014Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4767026Z 2025-09-07T07:19:22.4767131Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4767340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4767415Z return mod(**inputs) 2025-09-07T07:19:22.4767694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4767803Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4768064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4768133Z outputs = layer_module( 2025-09-07T07:19:22.4768400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4768473Z outputs = self.rel_attn( 2025-09-07T07:19:22.4768745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4768824Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4769175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4769348Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4769353Z 2025-09-07T07:19:22.4769464Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4769687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4769756Z return mod(**inputs) 2025-09-07T07:19:22.4770080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4770169Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4770452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4770532Z outputs = layer_module( 2025-09-07T07:19:22.4770857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4770934Z outputs = self.rel_attn( 2025-09-07T07:19:22.4771196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4771298Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4771614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4771739Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4771743Z 2025-09-07T07:19:22.4771862Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4772097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4772175Z return mod(**inputs) 2025-09-07T07:19:22.4772480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4772589Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4772891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4772967Z outputs = layer_module( 2025-09-07T07:19:22.4773256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4773331Z outputs = self.rel_attn( 2025-09-07T07:19:22.4773612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4773721Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4774028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4774157Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4774162Z 2025-09-07T07:19:22.4774275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4774520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4774612Z return mod(**inputs) 2025-09-07T07:19:22.4774894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4774992Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4775294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4775375Z outputs = layer_module( 2025-09-07T07:19:22.4775655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4775889Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4776187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4776275Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4776566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4776646Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4776934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4777014Z output = self.layer_1(output) 2025-09-07T07:19:22.4777019Z 2025-09-07T07:19:22.4777134Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4777362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4777457Z return mod(**inputs) 2025-09-07T07:19:22.4777745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4777834Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4778119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4778199Z outputs = layer_module( 2025-09-07T07:19:22.4778477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4778714Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4779022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4779112Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4779397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4779481Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4779748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4779839Z output = self.activation_function(output) 2025-09-07T07:19:22.4780062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4780132Z return self.act(input) 2025-09-07T07:19:22.4780135Z 2025-09-07T07:19:22.4780242Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4780451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4780520Z return mod(**inputs) 2025-09-07T07:19:22.4780783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4780868Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4781125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4781216Z outputs = layer_module( 2025-09-07T07:19:22.4781497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4781713Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4781980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4782067Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4782327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4782404Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4782685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4782761Z output = self.layer_2(output) 2025-09-07T07:19:22.4782767Z 2025-09-07T07:19:22.4782886Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4783104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4783175Z return mod(**inputs) 2025-09-07T07:19:22.4783456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4783544Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4783827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4783899Z outputs = layer_module( 2025-09-07T07:19:22.4784201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4784272Z outputs = self.rel_attn( 2025-09-07T07:19:22.4784532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4784646Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4784650Z 2025-09-07T07:19:22.4784758Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4784972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4785042Z return mod(**inputs) 2025-09-07T07:19:22.4785305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4785417Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4785690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4785855Z outputs = layer_module( 2025-09-07T07:19:22.4786133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4786204Z outputs = self.rel_attn( 2025-09-07T07:19:22.4786485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4786595Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4786599Z 2025-09-07T07:19:22.4786722Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4786951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4787037Z return mod(**inputs) 2025-09-07T07:19:22.4787329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4787416Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4787684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4787772Z outputs = layer_module( 2025-09-07T07:19:22.4788061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4788131Z outputs = self.rel_attn( 2025-09-07T07:19:22.4788386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4788469Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4788743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4788890Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4788895Z 2025-09-07T07:19:22.4789000Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4789211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4789279Z return mod(**inputs) 2025-09-07T07:19:22.4789539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4789628Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4789882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4789958Z outputs = layer_module( 2025-09-07T07:19:22.4790213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4790284Z outputs = self.rel_attn( 2025-09-07T07:19:22.4790545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4790700Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4790703Z 2025-09-07T07:19:22.4790817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4791020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4791092Z return mod(**inputs) 2025-09-07T07:19:22.4791349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4791432Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4791697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4791781Z outputs = layer_module( 2025-09-07T07:19:22.4792055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4792125Z outputs = self.rel_attn( 2025-09-07T07:19:22.4792391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4792471Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4792759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4792901Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4792904Z 2025-09-07T07:19:22.4793010Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4793225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4793292Z return mod(**inputs) 2025-09-07T07:19:22.4793558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4793650Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4793917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4794007Z outputs = layer_module( 2025-09-07T07:19:22.4794295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4794365Z outputs = self.rel_attn( 2025-09-07T07:19:22.4794629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4794729Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4794733Z 2025-09-07T07:19:22.4794842Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4795043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4795110Z return mod(**inputs) 2025-09-07T07:19:22.4795374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4795456Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4795729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4795795Z outputs = layer_module( 2025-09-07T07:19:22.4796050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4796116Z outputs = self.rel_attn( 2025-09-07T07:19:22.4796366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4796447Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4796715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4796863Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4796867Z 2025-09-07T07:19:22.4796968Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4797166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4797241Z return mod(**inputs) 2025-09-07T07:19:22.4797490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4797579Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4797828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4797920Z outputs = layer_module( 2025-09-07T07:19:22.4798171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4798238Z outputs = self.rel_attn( 2025-09-07T07:19:22.4798494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4798583Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4798863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4798972Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4798975Z 2025-09-07T07:19:22.4799075Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4799279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4799342Z return mod(**inputs) 2025-09-07T07:19:22.4799603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4799684Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4799943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4800015Z outputs = layer_module( 2025-09-07T07:19:22.4800298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4800375Z outputs = self.rel_attn( 2025-09-07T07:19:22.4800627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4800721Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4800996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4801107Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4801111Z 2025-09-07T07:19:22.4801219Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4801422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4801493Z return mod(**inputs) 2025-09-07T07:19:22.4801747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4801832Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4802093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4802159Z outputs = layer_module( 2025-09-07T07:19:22.4802418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4802626Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4802896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4802993Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4803245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4803323Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4803571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4803649Z output = self.layer_1(output) 2025-09-07T07:19:22.4803652Z 2025-09-07T07:19:22.4803754Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4803950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4804039Z return mod(**inputs) 2025-09-07T07:19:22.4804289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4804379Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4804629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4804701Z outputs = layer_module( 2025-09-07T07:19:22.4804955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4805161Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4805429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4805504Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4805766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4805838Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4806089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4806185Z output = self.activation_function(output) 2025-09-07T07:19:22.4806416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4806517Z return self.act(input) 2025-09-07T07:19:22.4806521Z 2025-09-07T07:19:22.4806625Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4806827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4806892Z return mod(**inputs) 2025-09-07T07:19:22.4807141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4807231Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4807484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4807558Z outputs = layer_module( 2025-09-07T07:19:22.4807813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4808023Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4808298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4808373Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4808635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4808707Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4808967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4809057Z output = self.layer_2(output) 2025-09-07T07:19:22.4809061Z 2025-09-07T07:19:22.4809163Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4809367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4809431Z return mod(**inputs) 2025-09-07T07:19:22.4809689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4809769Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4810019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4810091Z outputs = layer_module( 2025-09-07T07:19:22.4810357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4810431Z outputs = self.rel_attn( 2025-09-07T07:19:22.4810681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4810778Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4810788Z 2025-09-07T07:19:22.4810892Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4811091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4811162Z return mod(**inputs) 2025-09-07T07:19:22.4811413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4811502Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4811753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4811818Z outputs = layer_module( 2025-09-07T07:19:22.4812075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4812142Z outputs = self.rel_attn( 2025-09-07T07:19:22.4812413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4812531Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4812535Z 2025-09-07T07:19:22.4812635Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4812838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4812903Z return mod(**inputs) 2025-09-07T07:19:22.4813158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4813242Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4813499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4813566Z outputs = layer_module( 2025-09-07T07:19:22.4813816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4813893Z outputs = self.rel_attn( 2025-09-07T07:19:22.4814153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4814243Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4814502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4814630Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4814636Z 2025-09-07T07:19:22.4814744Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4814943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4815036Z return mod(**inputs) 2025-09-07T07:19:22.4815294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4815378Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4815651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4815717Z outputs = layer_module( 2025-09-07T07:19:22.4815973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4816042Z outputs = self.rel_attn( 2025-09-07T07:19:22.4816299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4816485Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4816490Z 2025-09-07T07:19:22.4816589Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4816800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4816861Z return mod(**inputs) 2025-09-07T07:19:22.4817115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4817194Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4817435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4817507Z outputs = layer_module( 2025-09-07T07:19:22.4817748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4817823Z outputs = self.rel_attn( 2025-09-07T07:19:22.4818064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4818142Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4818467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4818596Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4818615Z 2025-09-07T07:19:22.4818723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4818916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4818986Z return mod(**inputs) 2025-09-07T07:19:22.4819232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4819313Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4819754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4819829Z outputs = layer_module( 2025-09-07T07:19:22.4820086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4820155Z outputs = self.rel_attn( 2025-09-07T07:19:22.4820421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4820522Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4820525Z 2025-09-07T07:19:22.4820628Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4820835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4820902Z return mod(**inputs) 2025-09-07T07:19:22.4821162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4821345Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4821597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4821672Z outputs = layer_module( 2025-09-07T07:19:22.4821925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4821998Z outputs = self.rel_attn( 2025-09-07T07:19:22.4822252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4822322Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4822601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4822762Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4822765Z 2025-09-07T07:19:22.4822880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4823076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4823148Z return mod(**inputs) 2025-09-07T07:19:22.4823401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4823483Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4823745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4823810Z outputs = layer_module( 2025-09-07T07:19:22.4824064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4824131Z outputs = self.rel_attn( 2025-09-07T07:19:22.4824383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4824482Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4824760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4824907Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4824913Z 2025-09-07T07:19:22.4825057Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4825285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4825353Z return mod(**inputs) 2025-09-07T07:19:22.4825636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4825793Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4826075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4826155Z outputs = layer_module( 2025-09-07T07:19:22.4826424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4826495Z outputs = self.rel_attn( 2025-09-07T07:19:22.4826784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4826873Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4827151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4827259Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4827263Z 2025-09-07T07:19:22.4827372Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4827567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4827651Z return mod(**inputs) 2025-09-07T07:19:22.4827964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4828048Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4828321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4828390Z outputs = layer_module( 2025-09-07T07:19:22.4828658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4828879Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4829145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4829252Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4829517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4829595Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4829842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4829913Z output = self.layer_1(output) 2025-09-07T07:19:22.4829916Z 2025-09-07T07:19:22.4830022Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4830216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4830286Z return mod(**inputs) 2025-09-07T07:19:22.4830535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4830617Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4830870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4830935Z outputs = layer_module( 2025-09-07T07:19:22.4831185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4831427Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4831688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4831763Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4832006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4832085Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4832328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4832428Z output = self.activation_function(output) 2025-09-07T07:19:22.4832636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4832703Z return self.act(input) 2025-09-07T07:19:22.4832707Z 2025-09-07T07:19:22.4832816Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4833012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4833082Z return mod(**inputs) 2025-09-07T07:19:22.4833331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4833410Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4833672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4833735Z outputs = layer_module( 2025-09-07T07:19:22.4834002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4834208Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4834485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4834564Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4834822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4834902Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4835160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4835583Z output = self.layer_2(output) 2025-09-07T07:19:22.4835587Z 2025-09-07T07:19:22.4835692Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4835908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4835981Z return mod(**inputs) 2025-09-07T07:19:22.4836239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4836331Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4836591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4836666Z outputs = layer_module( 2025-09-07T07:19:22.4836929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4837000Z outputs = self.rel_attn( 2025-09-07T07:19:22.4837269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4837370Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4837373Z 2025-09-07T07:19:22.4837484Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4837708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4837788Z return mod(**inputs) 2025-09-07T07:19:22.4838071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4838158Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4838417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4838484Z outputs = layer_module( 2025-09-07T07:19:22.4838744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4838810Z outputs = self.rel_attn( 2025-09-07T07:19:22.4839061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4839165Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4839168Z 2025-09-07T07:19:22.4839269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4839471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4839537Z return mod(**inputs) 2025-09-07T07:19:22.4839787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4839874Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4840124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4840198Z outputs = layer_module( 2025-09-07T07:19:22.4840447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4840535Z outputs = self.rel_attn( 2025-09-07T07:19:22.4840789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4840860Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4841134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4841268Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4841271Z 2025-09-07T07:19:22.4841380Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4841577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4841665Z return mod(**inputs) 2025-09-07T07:19:22.4841925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4842007Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4842266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4842332Z outputs = layer_module( 2025-09-07T07:19:22.4842582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4842656Z outputs = self.rel_attn( 2025-09-07T07:19:22.4842904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4843046Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4843051Z 2025-09-07T07:19:22.4843151Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4843354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4843421Z return mod(**inputs) 2025-09-07T07:19:22.4843671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4843777Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4844048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4844125Z outputs = layer_module( 2025-09-07T07:19:22.4844379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4844447Z outputs = self.rel_attn( 2025-09-07T07:19:22.4844712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4844786Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4845068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4845202Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4845205Z 2025-09-07T07:19:22.4845320Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4845525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4845592Z return mod(**inputs) 2025-09-07T07:19:22.4845861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4845944Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4846210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4846281Z outputs = layer_module( 2025-09-07T07:19:22.4846544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4846639Z outputs = self.rel_attn( 2025-09-07T07:19:22.4846899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4847006Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4847011Z 2025-09-07T07:19:22.4847113Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4847312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4847383Z return mod(**inputs) 2025-09-07T07:19:22.4847638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4847743Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4847993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4848067Z outputs = layer_module( 2025-09-07T07:19:22.4848316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4848385Z outputs = self.rel_attn( 2025-09-07T07:19:22.4848644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4848715Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4848989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4849112Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4849118Z 2025-09-07T07:19:22.4849220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4849423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4849488Z return mod(**inputs) 2025-09-07T07:19:22.4849749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4849831Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4850149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4850218Z outputs = layer_module( 2025-09-07T07:19:22.4850478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4850554Z outputs = self.rel_attn( 2025-09-07T07:19:22.4850804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4850903Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4851175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4851295Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4851298Z 2025-09-07T07:19:22.4851408Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4851613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4851687Z return mod(**inputs) 2025-09-07T07:19:22.4851947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4852032Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4852295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4852365Z outputs = layer_module( 2025-09-07T07:19:22.4852639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4852722Z outputs = self.rel_attn( 2025-09-07T07:19:22.4852979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4853068Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4853341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4853458Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4853461Z 2025-09-07T07:19:22.4853563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4853765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4853850Z return mod(**inputs) 2025-09-07T07:19:22.4854101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4854191Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4854441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4854513Z outputs = layer_module( 2025-09-07T07:19:22.4854764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4854977Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4855238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4855313Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4855575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4855646Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4855904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4855975Z output = self.layer_1(output) 2025-09-07T07:19:22.4855978Z 2025-09-07T07:19:22.4856093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4856312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4856378Z return mod(**inputs) 2025-09-07T07:19:22.4856640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4856723Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4856983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4857051Z outputs = layer_module( 2025-09-07T07:19:22.4857304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4857525Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4857793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4857881Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4858145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4858217Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4858477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4858566Z output = self.activation_function(output) 2025-09-07T07:19:22.4858795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4858887Z return self.act(input) 2025-09-07T07:19:22.4858891Z 2025-09-07T07:19:22.4859002Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4859203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4859279Z return mod(**inputs) 2025-09-07T07:19:22.4859536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4859616Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4859875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4859944Z outputs = layer_module( 2025-09-07T07:19:22.4860222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4860441Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4860713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4860801Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4861064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4861145Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4861405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4861478Z output = self.layer_2(output) 2025-09-07T07:19:22.4861483Z 2025-09-07T07:19:22.4861599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4861801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4861875Z return mod(**inputs) 2025-09-07T07:19:22.4862136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4862218Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4862515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4862583Z outputs = layer_module( 2025-09-07T07:19:22.4862846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4862915Z outputs = self.rel_attn( 2025-09-07T07:19:22.4863169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4863280Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4863283Z 2025-09-07T07:19:22.4863386Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4863600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4863666Z return mod(**inputs) 2025-09-07T07:19:22.4863933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4864018Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4864291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4864370Z outputs = layer_module( 2025-09-07T07:19:22.4864644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4864725Z outputs = self.rel_attn( 2025-09-07T07:19:22.4864995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4865102Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4865142Z 2025-09-07T07:19:22.4865254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4865471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4865547Z return mod(**inputs) 2025-09-07T07:19:22.4865907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4866012Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4866295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4866370Z outputs = layer_module( 2025-09-07T07:19:22.4866683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4866760Z outputs = self.rel_attn( 2025-09-07T07:19:22.4867050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4867131Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4867435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4867598Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4867602Z 2025-09-07T07:19:22.4867705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4867914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4867980Z return mod(**inputs) 2025-09-07T07:19:22.4868245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4868331Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4868591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4868666Z outputs = layer_module( 2025-09-07T07:19:22.4868944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4869020Z outputs = self.rel_attn( 2025-09-07T07:19:22.4869295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4869431Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4869434Z 2025-09-07T07:19:22.4869543Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4869746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4869822Z return mod(**inputs) 2025-09-07T07:19:22.4870090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4870174Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4870438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4870506Z outputs = layer_module( 2025-09-07T07:19:22.4870771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4870840Z outputs = self.rel_attn( 2025-09-07T07:19:22.4871106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4871178Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4871457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4871595Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4871617Z 2025-09-07T07:19:22.4871723Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4871931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4871999Z return mod(**inputs) 2025-09-07T07:19:22.4872261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4872350Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4872609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4872682Z outputs = layer_module( 2025-09-07T07:19:22.4872940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4873032Z outputs = self.rel_attn( 2025-09-07T07:19:22.4873288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4873391Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4873394Z 2025-09-07T07:19:22.4873510Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4873713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4873787Z return mod(**inputs) 2025-09-07T07:19:22.4874046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4874128Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4874391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4874462Z outputs = layer_module( 2025-09-07T07:19:22.4874724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4874796Z outputs = self.rel_attn( 2025-09-07T07:19:22.4875051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4875146Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4875438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4875575Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4875579Z 2025-09-07T07:19:22.4875681Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4875891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4875959Z return mod(**inputs) 2025-09-07T07:19:22.4876223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4876317Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4876590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4876666Z outputs = layer_module( 2025-09-07T07:19:22.4876931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4876998Z outputs = self.rel_attn( 2025-09-07T07:19:22.4877267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4877358Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4877646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4877762Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4877786Z 2025-09-07T07:19:22.4877895Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4878097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4878164Z return mod(**inputs) 2025-09-07T07:19:22.4878434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4878517Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4878773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4878841Z outputs = layer_module( 2025-09-07T07:19:22.4879097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4879193Z outputs = self.rel_attn( 2025-09-07T07:19:22.4879458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4879553Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4879827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4879941Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4879946Z 2025-09-07T07:19:22.4880048Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4880243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4880317Z return mod(**inputs) 2025-09-07T07:19:22.4880569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4880658Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4880909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4880977Z outputs = layer_module( 2025-09-07T07:19:22.4881237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4881478Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4881747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4881825Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4882087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4882163Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4882413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4882496Z output = self.layer_1(output) 2025-09-07T07:19:22.4882499Z 2025-09-07T07:19:22.4882600Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4882802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4882869Z return mod(**inputs) 2025-09-07T07:19:22.4883128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4883218Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4883467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4883540Z outputs = layer_module( 2025-09-07T07:19:22.4883786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4884001Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4884282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4884362Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4884630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4884703Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4884971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4885056Z output = self.activation_function(output) 2025-09-07T07:19:22.4885268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4885376Z return self.act(input) 2025-09-07T07:19:22.4885379Z 2025-09-07T07:19:22.4885480Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4885706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4885775Z return mod(**inputs) 2025-09-07T07:19:22.4886051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4886147Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4886417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4886491Z outputs = layer_module( 2025-09-07T07:19:22.4886745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4886965Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4887229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4887309Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4887583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4887670Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4887955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4888027Z output = self.layer_2(output) 2025-09-07T07:19:22.4888030Z 2025-09-07T07:19:22.4888133Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4888336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4888403Z return mod(**inputs) 2025-09-07T07:19:22.4888664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4888748Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4889018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4889088Z outputs = layer_module( 2025-09-07T07:19:22.4889346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4889423Z outputs = self.rel_attn( 2025-09-07T07:19:22.4889680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4889787Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4889790Z 2025-09-07T07:19:22.4889893Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4890096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4890171Z return mod(**inputs) 2025-09-07T07:19:22.4890459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4890547Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4890797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4890872Z outputs = layer_module( 2025-09-07T07:19:22.4891121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4891188Z outputs = self.rel_attn( 2025-09-07T07:19:22.4891450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4891569Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4891573Z 2025-09-07T07:19:22.4891680Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4891877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4891942Z return mod(**inputs) 2025-09-07T07:19:22.4892203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4892284Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4892539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4892604Z outputs = layer_module( 2025-09-07T07:19:22.4892855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4892930Z outputs = self.rel_attn( 2025-09-07T07:19:22.4893189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4893267Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4893537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4893675Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4893696Z 2025-09-07T07:19:22.4893801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4894021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4894096Z return mod(**inputs) 2025-09-07T07:19:22.4894349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4894436Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4894688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4894756Z outputs = layer_module( 2025-09-07T07:19:22.4895022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4895090Z outputs = self.rel_attn( 2025-09-07T07:19:22.4895363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4895496Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4895500Z 2025-09-07T07:19:22.4895607Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4895803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4895867Z return mod(**inputs) 2025-09-07T07:19:22.4896145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4896231Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4896498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4896596Z outputs = layer_module( 2025-09-07T07:19:22.4896857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4896932Z outputs = self.rel_attn( 2025-09-07T07:19:22.4897184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4897261Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4897528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4897664Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4897685Z 2025-09-07T07:19:22.4897786Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4897984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4898059Z return mod(**inputs) 2025-09-07T07:19:22.4898315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4898404Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4898659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4898725Z outputs = layer_module( 2025-09-07T07:19:22.4898985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4899048Z outputs = self.rel_attn( 2025-09-07T07:19:22.4899309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4899406Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4899412Z 2025-09-07T07:19:22.4899518Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4899721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4899801Z return mod(**inputs) 2025-09-07T07:19:22.4900082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4900166Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4900429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4900495Z outputs = layer_module( 2025-09-07T07:19:22.4900754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4900829Z outputs = self.rel_attn( 2025-09-07T07:19:22.4901087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4901168Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4901444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4901572Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4901582Z 2025-09-07T07:19:22.4901686Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4901889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4901962Z return mod(**inputs) 2025-09-07T07:19:22.4902221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4902312Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4902571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4902657Z outputs = layer_module( 2025-09-07T07:19:22.4902920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4902991Z outputs = self.rel_attn( 2025-09-07T07:19:22.4903255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4903346Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4903630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4903752Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4903775Z 2025-09-07T07:19:22.4903880Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4904089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4904159Z return mod(**inputs) 2025-09-07T07:19:22.4904422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4904507Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4904767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4904842Z outputs = layer_module( 2025-09-07T07:19:22.4905105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4905185Z outputs = self.rel_attn( 2025-09-07T07:19:22.4905465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4905561Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4905937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4906066Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4906070Z 2025-09-07T07:19:22.4906211Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4906447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4906520Z return mod(**inputs) 2025-09-07T07:19:22.4906815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4906903Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4907189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4907259Z outputs = layer_module( 2025-09-07T07:19:22.4907523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4907737Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4908011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4908104Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4908363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4908446Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4908699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4908775Z output = self.layer_1(output) 2025-09-07T07:19:22.4908786Z 2025-09-07T07:19:22.4908891Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4909097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4909191Z return mod(**inputs) 2025-09-07T07:19:22.4909454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4909548Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4909808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4909876Z outputs = layer_module( 2025-09-07T07:19:22.4910142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4910353Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4910648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4910729Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4910989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4911070Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4911326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4911422Z output = self.activation_function(output) 2025-09-07T07:19:22.4911638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4911716Z return self.act(input) 2025-09-07T07:19:22.4911719Z 2025-09-07T07:19:22.4911823Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4912026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4912103Z return mod(**inputs) 2025-09-07T07:19:22.4912359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4912450Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4912731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4912820Z outputs = layer_module( 2025-09-07T07:19:22.4913088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4913298Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4913566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4913645Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4913910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4913983Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4914241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4914321Z output = self.layer_2(output) 2025-09-07T07:19:22.4914327Z 2025-09-07T07:19:22.4914432Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4914639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4914705Z return mod(**inputs) 2025-09-07T07:19:22.4914971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4915062Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4915318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4915446Z outputs = layer_module( 2025-09-07T07:19:22.4915710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4915780Z outputs = self.rel_attn( 2025-09-07T07:19:22.4916049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4916149Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4916154Z 2025-09-07T07:19:22.4916265Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4916470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4916560Z return mod(**inputs) 2025-09-07T07:19:22.4916820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4916905Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4917170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4917237Z outputs = layer_module( 2025-09-07T07:19:22.4917501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4917571Z outputs = self.rel_attn( 2025-09-07T07:19:22.4917827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4917935Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4917939Z 2025-09-07T07:19:22.4918042Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4918252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4918318Z return mod(**inputs) 2025-09-07T07:19:22.4918585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4918669Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4918945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4919039Z outputs = layer_module( 2025-09-07T07:19:22.4919299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4919376Z outputs = self.rel_attn( 2025-09-07T07:19:22.4919819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4919902Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4920245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4920385Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4920389Z 2025-09-07T07:19:22.4920502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4920708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4920781Z return mod(**inputs) 2025-09-07T07:19:22.4921044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4921129Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4921395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4921463Z outputs = layer_module( 2025-09-07T07:19:22.4921738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4921808Z outputs = self.rel_attn( 2025-09-07T07:19:22.4922130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4922271Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4922276Z 2025-09-07T07:19:22.4922379Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4922586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4922652Z return mod(**inputs) 2025-09-07T07:19:22.4922910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4923001Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4923291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4923366Z outputs = layer_module( 2025-09-07T07:19:22.4923623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4923701Z outputs = self.rel_attn( 2025-09-07T07:19:22.4923992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4924073Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4924386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4924532Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4924535Z 2025-09-07T07:19:22.4924657Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4924890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4924961Z return mod(**inputs) 2025-09-07T07:19:22.4925263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4925355Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4925671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4925746Z outputs = layer_module( 2025-09-07T07:19:22.4926059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4926132Z outputs = self.rel_attn( 2025-09-07T07:19:22.4926382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4926485Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4926490Z 2025-09-07T07:19:22.4926591Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4926799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4926864Z return mod(**inputs) 2025-09-07T07:19:22.4927115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4927205Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4927459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4927530Z outputs = layer_module( 2025-09-07T07:19:22.4927779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4927845Z outputs = self.rel_attn( 2025-09-07T07:19:22.4928101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4928174Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4928473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4928595Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4928599Z 2025-09-07T07:19:22.4928707Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4928903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4928968Z return mod(**inputs) 2025-09-07T07:19:22.4929226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4929307Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4929563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4929651Z outputs = layer_module( 2025-09-07T07:19:22.4929902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4929975Z outputs = self.rel_attn( 2025-09-07T07:19:22.4930226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4930321Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4930595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4930712Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4930716Z 2025-09-07T07:19:22.4930816Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4931016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4931090Z return mod(**inputs) 2025-09-07T07:19:22.4931343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4931434Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4931699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4931768Z outputs = layer_module( 2025-09-07T07:19:22.4932043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4932112Z outputs = self.rel_attn( 2025-09-07T07:19:22.4932376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4932462Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4932741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4932850Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4932855Z 2025-09-07T07:19:22.4932956Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4933159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4933226Z return mod(**inputs) 2025-09-07T07:19:22.4933487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4933566Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4933817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4933890Z outputs = layer_module( 2025-09-07T07:19:22.4934141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4934355Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4934639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4934726Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4934986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4935058Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4935317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4935389Z output = self.layer_1(output) 2025-09-07T07:19:22.4935392Z 2025-09-07T07:19:22.4935502Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4935725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4935789Z return mod(**inputs) 2025-09-07T07:19:22.4936048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4936131Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4936391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4936458Z outputs = layer_module( 2025-09-07T07:19:22.4936708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4936921Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4937178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4937264Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4937515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4937595Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4937842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4937945Z output = self.activation_function(output) 2025-09-07T07:19:22.4938182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4938254Z return self.act(input) 2025-09-07T07:19:22.4938257Z 2025-09-07T07:19:22.4938367Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4938566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4938632Z return mod(**inputs) 2025-09-07T07:19:22.4938889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4938971Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4939231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4939296Z outputs = layer_module( 2025-09-07T07:19:22.4939556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4939759Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4940015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4940098Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4940351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4940427Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4940692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4940763Z output = self.layer_2(output) 2025-09-07T07:19:22.4940773Z 2025-09-07T07:19:22.4940875Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4941072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4941144Z return mod(**inputs) 2025-09-07T07:19:22.4941393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4941482Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4941731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4941814Z outputs = layer_module( 2025-09-07T07:19:22.4942079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4942148Z outputs = self.rel_attn( 2025-09-07T07:19:22.4942416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4942514Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4942517Z 2025-09-07T07:19:22.4942619Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4942829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4942895Z return mod(**inputs) 2025-09-07T07:19:22.4943164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4943247Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4943510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4943578Z outputs = layer_module( 2025-09-07T07:19:22.4943834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4943910Z outputs = self.rel_attn( 2025-09-07T07:19:22.4944198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4944310Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4944313Z 2025-09-07T07:19:22.4944417Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4944618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4944697Z return mod(**inputs) 2025-09-07T07:19:22.4944958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4945049Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4945312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4945383Z outputs = layer_module( 2025-09-07T07:19:22.4945650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4945780Z outputs = self.rel_attn( 2025-09-07T07:19:22.4946076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4946157Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4946459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4946606Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4946610Z 2025-09-07T07:19:22.4946724Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4946992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4947068Z return mod(**inputs) 2025-09-07T07:19:22.4947361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4947456Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4947747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4947830Z outputs = layer_module( 2025-09-07T07:19:22.4948111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4948213Z outputs = self.rel_attn( 2025-09-07T07:19:22.4948494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4948651Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4948655Z 2025-09-07T07:19:22.4948767Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4948991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4949071Z return mod(**inputs) 2025-09-07T07:19:22.4949380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4949476Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4949756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4949828Z outputs = layer_module( 2025-09-07T07:19:22.4950120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4950194Z outputs = self.rel_attn( 2025-09-07T07:19:22.4950481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4950560Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4950877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4951055Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4951059Z 2025-09-07T07:19:22.4951176Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4951406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4951480Z return mod(**inputs) 2025-09-07T07:19:22.4951786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4951880Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4952177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4952260Z outputs = layer_module( 2025-09-07T07:19:22.4952545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4952631Z outputs = self.rel_attn( 2025-09-07T07:19:22.4952915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4953027Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4953038Z 2025-09-07T07:19:22.4953152Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4953377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4953460Z return mod(**inputs) 2025-09-07T07:19:22.4953761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4953883Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4954177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4954242Z outputs = layer_module( 2025-09-07T07:19:22.4954495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4954559Z outputs = self.rel_attn( 2025-09-07T07:19:22.4954816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4954886Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4955163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4955289Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4955294Z 2025-09-07T07:19:22.4955392Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4955596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4955662Z return mod(**inputs) 2025-09-07T07:19:22.4955919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4956002Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4956256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4956329Z outputs = layer_module( 2025-09-07T07:19:22.4956579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4956663Z outputs = self.rel_attn( 2025-09-07T07:19:22.4956906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4956993Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4957283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4957408Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4957411Z 2025-09-07T07:19:22.4957518Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4957712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4957787Z return mod(**inputs) 2025-09-07T07:19:22.4958035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4958119Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4958371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4958439Z outputs = layer_module( 2025-09-07T07:19:22.4958699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4958766Z outputs = self.rel_attn( 2025-09-07T07:19:22.4959023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4959119Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4959390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4959506Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4959510Z 2025-09-07T07:19:22.4959608Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4959808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4959897Z return mod(**inputs) 2025-09-07T07:19:22.4960146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4960235Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4960488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4960558Z outputs = layer_module( 2025-09-07T07:19:22.4960807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4961009Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4961290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4961367Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4961621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4961692Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4961940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4962015Z output = self.layer_1(output) 2025-09-07T07:19:22.4962017Z 2025-09-07T07:19:22.4962116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4962315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4962377Z return mod(**inputs) 2025-09-07T07:19:22.4962630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4962708Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4962958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4963028Z outputs = layer_module( 2025-09-07T07:19:22.4963292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4963514Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4963767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4963841Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4964091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4964163Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4964413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4964499Z output = self.activation_function(output) 2025-09-07T07:19:22.4964714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4964781Z return self.act(input) 2025-09-07T07:19:22.4964784Z 2025-09-07T07:19:22.4964884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4965082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4965144Z return mod(**inputs) 2025-09-07T07:19:22.4965398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4965480Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4965730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4965823Z outputs = layer_module( 2025-09-07T07:19:22.4966074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4966285Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4966544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4966628Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4966886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4966955Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4967222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4967292Z output = self.layer_2(output) 2025-09-07T07:19:22.4967296Z 2025-09-07T07:19:22.4967402Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4967595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4967659Z return mod(**inputs) 2025-09-07T07:19:22.4967911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4967990Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4968242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4968305Z outputs = layer_module( 2025-09-07T07:19:22.4968553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4968621Z outputs = self.rel_attn( 2025-09-07T07:19:22.4968863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4968965Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4968968Z 2025-09-07T07:19:22.4969065Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4969294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4969358Z return mod(**inputs) 2025-09-07T07:19:22.4969600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4969687Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4969930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4970003Z outputs = layer_module( 2025-09-07T07:19:22.4970247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4970315Z outputs = self.rel_attn( 2025-09-07T07:19:22.4970564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4970661Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4970665Z 2025-09-07T07:19:22.4970772Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4970963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4971033Z return mod(**inputs) 2025-09-07T07:19:22.4971278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4971360Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4971618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4971702Z outputs = layer_module( 2025-09-07T07:19:22.4971964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4972029Z outputs = self.rel_attn( 2025-09-07T07:19:22.4972275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4972353Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4972619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4972757Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4972760Z 2025-09-07T07:19:22.4972861Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4973082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4973146Z return mod(**inputs) 2025-09-07T07:19:22.4973398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4973488Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4973743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4973816Z outputs = layer_module( 2025-09-07T07:19:22.4974070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4974137Z outputs = self.rel_attn( 2025-09-07T07:19:22.4974395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.4974528Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.4974531Z 2025-09-07T07:19:22.4974637Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4974836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4974908Z return mod(**inputs) 2025-09-07T07:19:22.4975184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4975285Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4975559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4975624Z outputs = layer_module( 2025-09-07T07:19:22.4975875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4975942Z outputs = self.rel_attn( 2025-09-07T07:19:22.4976191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4976272Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4976549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.4976686Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.4976690Z 2025-09-07T07:19:22.4976804Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4976998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4977069Z return mod(**inputs) 2025-09-07T07:19:22.4977322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4977408Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4977660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4977732Z outputs = layer_module( 2025-09-07T07:19:22.4977997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4978064Z outputs = self.rel_attn( 2025-09-07T07:19:22.4978322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.4978422Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.4978426Z 2025-09-07T07:19:22.4978531Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4978726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4978790Z return mod(**inputs) 2025-09-07T07:19:22.4979049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4979148Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4979407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4979472Z outputs = layer_module( 2025-09-07T07:19:22.4979729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4979798Z outputs = self.rel_attn( 2025-09-07T07:19:22.4980048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4980126Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4980390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.4980522Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.4980525Z 2025-09-07T07:19:22.4980627Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4980826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4980899Z return mod(**inputs) 2025-09-07T07:19:22.4981164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4981255Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4981535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4981604Z outputs = layer_module( 2025-09-07T07:19:22.4981859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4981927Z outputs = self.rel_attn( 2025-09-07T07:19:22.4982191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4982278Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4982557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4982668Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4982673Z 2025-09-07T07:19:22.4982773Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4982979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4983044Z return mod(**inputs) 2025-09-07T07:19:22.4983300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4983381Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4983631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4983704Z outputs = layer_module( 2025-09-07T07:19:22.4983969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4984042Z outputs = self.rel_attn( 2025-09-07T07:19:22.4984292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.4984388Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.4984661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.4984771Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.4984775Z 2025-09-07T07:19:22.4984885Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4985107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4985181Z return mod(**inputs) 2025-09-07T07:19:22.4985440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4985525Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4985885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4985962Z outputs = layer_module( 2025-09-07T07:19:22.4986245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4986472Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4986761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4986848Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4987122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4987212Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4987483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.4987593Z output = self.layer_1(output) 2025-09-07T07:19:22.4987598Z 2025-09-07T07:19:22.4987735Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4987933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4988007Z return mod(**inputs) 2025-09-07T07:19:22.4988262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4988354Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4988612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4988690Z outputs = layer_module( 2025-09-07T07:19:22.4988945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4989161Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4989440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4989518Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4989782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4989855Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4990112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.4990211Z output = self.activation_function(output) 2025-09-07T07:19:22.4990442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.4990517Z return self.act(input) 2025-09-07T07:19:22.4990520Z 2025-09-07T07:19:22.4990622Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4990825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4990889Z return mod(**inputs) 2025-09-07T07:19:22.4991140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4991229Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4991482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4991569Z outputs = layer_module( 2025-09-07T07:19:22.4991816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.4992019Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.4992281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.4992356Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.4992611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.4992679Z output_x = self.ff(output_x) 2025-09-07T07:19:22.4992925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.4993003Z output = self.layer_2(output) 2025-09-07T07:19:22.4993006Z 2025-09-07T07:19:22.4993106Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4993308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4993373Z return mod(**inputs) 2025-09-07T07:19:22.4993629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4993722Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4993982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4994058Z outputs = layer_module( 2025-09-07T07:19:22.4994308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4994387Z outputs = self.rel_attn( 2025-09-07T07:19:22.4994651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.4994754Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.4994766Z 2025-09-07T07:19:22.4994870Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4995073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4995146Z return mod(**inputs) 2025-09-07T07:19:22.4995409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4995507Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4995759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4995826Z outputs = layer_module( 2025-09-07T07:19:22.4996085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4996154Z outputs = self.rel_attn( 2025-09-07T07:19:22.4996410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.4996529Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.4996532Z 2025-09-07T07:19:22.4996632Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4996840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4996908Z return mod(**inputs) 2025-09-07T07:19:22.4997165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4997245Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4997497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4997588Z outputs = layer_module( 2025-09-07T07:19:22.4997844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.4997926Z outputs = self.rel_attn( 2025-09-07T07:19:22.4998181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.4998262Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.4998551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.4998678Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.4998681Z 2025-09-07T07:19:22.4998787Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.4998976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.4999048Z return mod(**inputs) 2025-09-07T07:19:22.4999299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.4999378Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.4999629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.4999692Z outputs = layer_module( 2025-09-07T07:19:22.4999975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5000041Z outputs = self.rel_attn( 2025-09-07T07:19:22.5000289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.5000416Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.5000419Z 2025-09-07T07:19:22.5000516Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5000718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5000780Z return mod(**inputs) 2025-09-07T07:19:22.5001032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5001110Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5001353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5001425Z outputs = layer_module( 2025-09-07T07:19:22.5001668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5001740Z outputs = self.rel_attn( 2025-09-07T07:19:22.5001979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5002058Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5002318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.5002461Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.5002464Z 2025-09-07T07:19:22.5002570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5002763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5002835Z return mod(**inputs) 2025-09-07T07:19:22.5003077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5003156Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5003408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5003489Z outputs = layer_module( 2025-09-07T07:19:22.5003742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5003809Z outputs = self.rel_attn( 2025-09-07T07:19:22.5004055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.5004160Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.5004163Z 2025-09-07T07:19:22.5004264Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5004465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5004527Z return mod(**inputs) 2025-09-07T07:19:22.5004791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5004874Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5005139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5005210Z outputs = layer_module( 2025-09-07T07:19:22.5005465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5005534Z outputs = self.rel_attn( 2025-09-07T07:19:22.5005811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5005901Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5006176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.5006299Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.5006302Z 2025-09-07T07:19:22.5006419Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5006612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5006681Z return mod(**inputs) 2025-09-07T07:19:22.5006927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5007008Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5007260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5007325Z outputs = layer_module( 2025-09-07T07:19:22.5007576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5007640Z outputs = self.rel_attn( 2025-09-07T07:19:22.5007883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5007977Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5008243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5008381Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5008385Z 2025-09-07T07:19:22.5008482Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5008684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5008746Z return mod(**inputs) 2025-09-07T07:19:22.5008997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5009087Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5009334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5009404Z outputs = layer_module( 2025-09-07T07:19:22.5009668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5009733Z outputs = self.rel_attn( 2025-09-07T07:19:22.5009982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5010068Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5010340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5010448Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5010451Z 2025-09-07T07:19:22.5010557Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5010747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5010811Z return mod(**inputs) 2025-09-07T07:19:22.5011063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5011141Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5011395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5011460Z outputs = layer_module( 2025-09-07T07:19:22.5011718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5011979Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5012234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5012318Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5012570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5012639Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5012885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.5012955Z output = self.layer_1(output) 2025-09-07T07:19:22.5012958Z 2025-09-07T07:19:22.5013064Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5013251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5013321Z return mod(**inputs) 2025-09-07T07:19:22.5013559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5013642Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5013888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5013950Z outputs = layer_module( 2025-09-07T07:19:22.5014194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5014404Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5014653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5014733Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5014978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5015056Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5015303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.5015398Z output = self.activation_function(output) 2025-09-07T07:19:22.5015622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.5015688Z return self.act(input) 2025-09-07T07:19:22.5015693Z 2025-09-07T07:19:22.5015799Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5015991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5016061Z return mod(**inputs) 2025-09-07T07:19:22.5016308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5016389Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5016641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5016747Z outputs = layer_module( 2025-09-07T07:19:22.5017078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5017298Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5017566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5017639Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5017907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5018004Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5018256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.5018336Z output = self.layer_2(output) 2025-09-07T07:19:22.5018339Z 2025-09-07T07:19:22.5018442Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5018645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5018718Z return mod(**inputs) 2025-09-07T07:19:22.5018961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5019050Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5019297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5019370Z outputs = layer_module( 2025-09-07T07:19:22.5019790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5019863Z outputs = self.rel_attn( 2025-09-07T07:19:22.5020123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.5020222Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.5020228Z 2025-09-07T07:19:22.5020336Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5020532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5020650Z return mod(**inputs) 2025-09-07T07:19:22.5020911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5020992Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5021250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5021315Z outputs = layer_module( 2025-09-07T07:19:22.5021578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5021653Z outputs = self.rel_attn( 2025-09-07T07:19:22.5021906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.5022041Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.5022046Z 2025-09-07T07:19:22.5022146Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5022353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5022418Z return mod(**inputs) 2025-09-07T07:19:22.5022673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5022762Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5023013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5023084Z outputs = layer_module( 2025-09-07T07:19:22.5023333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5023401Z outputs = self.rel_attn( 2025-09-07T07:19:22.5023660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5023734Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5024008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.5024163Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.5024193Z 2025-09-07T07:19:22.5024305Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5024507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5024574Z return mod(**inputs) 2025-09-07T07:19:22.5024848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5024933Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5025205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5025276Z outputs = layer_module( 2025-09-07T07:19:22.5025538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5025617Z outputs = self.rel_attn( 2025-09-07T07:19:22.5025920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.5026071Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.5026075Z 2025-09-07T07:19:22.5026179Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5026392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5026461Z return mod(**inputs) 2025-09-07T07:19:22.5026719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5026834Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5027090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5027165Z outputs = layer_module( 2025-09-07T07:19:22.5027425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5027493Z outputs = self.rel_attn( 2025-09-07T07:19:22.5027770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5027844Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5028128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.5028287Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.5028290Z 2025-09-07T07:19:22.5028396Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5028609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5028677Z return mod(**inputs) 2025-09-07T07:19:22.5028943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5029034Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5029284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5029347Z outputs = layer_module( 2025-09-07T07:19:22.5029592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5029669Z outputs = self.rel_attn( 2025-09-07T07:19:22.5029925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.5030033Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.5030036Z 2025-09-07T07:19:22.5030138Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5030356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5030455Z return mod(**inputs) 2025-09-07T07:19:22.5030712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5030803Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5031062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5031137Z outputs = layer_module( 2025-09-07T07:19:22.5031393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5031463Z outputs = self.rel_attn( 2025-09-07T07:19:22.5031727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5031799Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5032084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.5032211Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.5032215Z 2025-09-07T07:19:22.5032318Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5032529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5032595Z return mod(**inputs) 2025-09-07T07:19:22.5032861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5032945Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5033220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5033294Z outputs = layer_module( 2025-09-07T07:19:22.5033551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5033629Z outputs = self.rel_attn( 2025-09-07T07:19:22.5033886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5033981Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5034258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5034393Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5034398Z 2025-09-07T07:19:22.5034508Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5034709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5034782Z return mod(**inputs) 2025-09-07T07:19:22.5035042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5035126Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5035389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5035455Z outputs = layer_module( 2025-09-07T07:19:22.5035715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5035786Z outputs = self.rel_attn( 2025-09-07T07:19:22.5036045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5036133Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5036410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5036543Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5036548Z 2025-09-07T07:19:22.5036667Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5036876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5036941Z return mod(**inputs) 2025-09-07T07:19:22.5037198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5037287Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5037546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5037619Z outputs = layer_module( 2025-09-07T07:19:22.5037876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5038093Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5038359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5038438Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5038706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5038780Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5039042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.5039116Z output = self.layer_1(output) 2025-09-07T07:19:22.5039120Z 2025-09-07T07:19:22.5039240Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5039450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5039516Z return mod(**inputs) 2025-09-07T07:19:22.5039792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5039871Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5040125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5040189Z outputs = layer_module( 2025-09-07T07:19:22.5040438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5040671Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5040933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5041020Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5041278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5041351Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5041611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.5041697Z output = self.activation_function(output) 2025-09-07T07:19:22.5041919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.5041987Z return self.act(input) 2025-09-07T07:19:22.5041993Z 2025-09-07T07:19:22.5042100Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5042301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5042367Z return mod(**inputs) 2025-09-07T07:19:22.5042627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5042729Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5043018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5043089Z outputs = layer_module( 2025-09-07T07:19:22.5043349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5043567Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5043844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5043931Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5044185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5044256Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5044518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.5044589Z output = self.layer_2(output) 2025-09-07T07:19:22.5044592Z 2025-09-07T07:19:22.5044702Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5044905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5044975Z return mod(**inputs) 2025-09-07T07:19:22.5045228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5045311Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5045588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5045653Z outputs = layer_module( 2025-09-07T07:19:22.5045914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5045984Z outputs = self.rel_attn( 2025-09-07T07:19:22.5046236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.5046342Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.5046345Z 2025-09-07T07:19:22.5046448Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5046661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5046746Z return mod(**inputs) 2025-09-07T07:19:22.5047014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5047099Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5047362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5047438Z outputs = layer_module( 2025-09-07T07:19:22.5047707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5047781Z outputs = self.rel_attn( 2025-09-07T07:19:22.5048034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.5048132Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.5048137Z 2025-09-07T07:19:22.5048246Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5048449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5048522Z return mod(**inputs) 2025-09-07T07:19:22.5048777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5048880Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5049155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5049223Z outputs = layer_module( 2025-09-07T07:19:22.5049485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5049551Z outputs = self.rel_attn( 2025-09-07T07:19:22.5049809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5049882Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5050150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.5050289Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.5050293Z 2025-09-07T07:19:22.5050393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5050599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5050663Z return mod(**inputs) 2025-09-07T07:19:22.5050918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5051004Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5051258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5051334Z outputs = layer_module( 2025-09-07T07:19:22.5051587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5051695Z outputs = self.rel_attn( 2025-09-07T07:19:22.5051937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.5052066Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.5052070Z 2025-09-07T07:19:22.5052175Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5052369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5052438Z return mod(**inputs) 2025-09-07T07:19:22.5052683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5052779Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5053032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5053098Z outputs = layer_module( 2025-09-07T07:19:22.5053344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5053411Z outputs = self.rel_attn( 2025-09-07T07:19:22.5053666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5053737Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5054008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.5054145Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.5054151Z 2025-09-07T07:19:22.5054252Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5054452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5054519Z return mod(**inputs) 2025-09-07T07:19:22.5054770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5054856Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5055136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5055213Z outputs = layer_module( 2025-09-07T07:19:22.5055467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5055536Z outputs = self.rel_attn( 2025-09-07T07:19:22.5055853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.5055952Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.5055955Z 2025-09-07T07:19:22.5056063Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5056257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5056328Z return mod(**inputs) 2025-09-07T07:19:22.5056578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5056661Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5056922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5056987Z outputs = layer_module( 2025-09-07T07:19:22.5057241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5057310Z outputs = self.rel_attn( 2025-09-07T07:19:22.5057557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5057651Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5057918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.5058049Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.5058052Z 2025-09-07T07:19:22.5058154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5058359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5058424Z return mod(**inputs) 2025-09-07T07:19:22.5058679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5058770Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5059042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5059117Z outputs = layer_module( 2025-09-07T07:19:22.5059368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5059434Z outputs = self.rel_attn( 2025-09-07T07:19:22.5059692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5059779Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5060057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5060167Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5060170Z 2025-09-07T07:19:22.5060276Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5060474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5060540Z return mod(**inputs) 2025-09-07T07:19:22.5060801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5060882Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5061158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5061241Z outputs = layer_module( 2025-09-07T07:19:22.5061495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5061570Z outputs = self.rel_attn( 2025-09-07T07:19:22.5061818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5061914Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5062184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5062294Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5062306Z 2025-09-07T07:19:22.5062407Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5062604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5062677Z return mod(**inputs) 2025-09-07T07:19:22.5062927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5063015Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5063266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5063333Z outputs = layer_module( 2025-09-07T07:19:22.5063594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5063823Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5064099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5064178Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5064440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5064523Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5064780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.5064863Z output = self.layer_1(output) 2025-09-07T07:19:22.5064883Z 2025-09-07T07:19:22.5064988Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5065197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5065265Z return mod(**inputs) 2025-09-07T07:19:22.5065539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5065634Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5066005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5066091Z outputs = layer_module( 2025-09-07T07:19:22.5066363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5066587Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5066885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5066966Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5067236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5067308Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5067592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.5067698Z output = self.activation_function(output) 2025-09-07T07:19:22.5067930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.5068009Z return self.act(input) 2025-09-07T07:19:22.5068014Z 2025-09-07T07:19:22.5068116Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5068321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5068389Z return mod(**inputs) 2025-09-07T07:19:22.5068639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5068728Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5068979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5069051Z outputs = layer_module( 2025-09-07T07:19:22.5069302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5069510Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5069771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5069847Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5070105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5070197Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5070455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.5070529Z output = self.layer_2(output) 2025-09-07T07:19:22.5070532Z 2025-09-07T07:19:22.5070638Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5070842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5070907Z return mod(**inputs) 2025-09-07T07:19:22.5071169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5071251Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5071528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5071595Z outputs = layer_module( 2025-09-07T07:19:22.5071849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5071926Z outputs = self.rel_attn( 2025-09-07T07:19:22.5072179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.5072284Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.5072287Z 2025-09-07T07:19:22.5072388Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5072586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5072660Z return mod(**inputs) 2025-09-07T07:19:22.5072910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5073001Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5073252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5073320Z outputs = layer_module( 2025-09-07T07:19:22.5073593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5073663Z outputs = self.rel_attn( 2025-09-07T07:19:22.5073953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.5074054Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.5074058Z 2025-09-07T07:19:22.5074166Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5074367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5074435Z return mod(**inputs) 2025-09-07T07:19:22.5074698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5074782Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5075048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5075118Z outputs = layer_module( 2025-09-07T07:19:22.5075371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5075448Z outputs = self.rel_attn( 2025-09-07T07:19:22.5075705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5075789Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5076067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.5076215Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.5076247Z 2025-09-07T07:19:22.5076351Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5076555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5076632Z return mod(**inputs) 2025-09-07T07:19:22.5076904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5076993Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5077250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5077314Z outputs = layer_module( 2025-09-07T07:19:22.5077576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5077659Z outputs = self.rel_attn( 2025-09-07T07:19:22.5077921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.5078057Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.5078061Z 2025-09-07T07:19:22.5078174Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5078378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5078445Z return mod(**inputs) 2025-09-07T07:19:22.5078713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5078795Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5079061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5079129Z outputs = layer_module( 2025-09-07T07:19:22.5079383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5079463Z outputs = self.rel_attn( 2025-09-07T07:19:22.5079718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5079813Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5080107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.5080243Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.5080253Z 2025-09-07T07:19:22.5080362Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5080578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5080658Z return mod(**inputs) 2025-09-07T07:19:22.5080939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5081046Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5081321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5081394Z outputs = layer_module( 2025-09-07T07:19:22.5081677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5081750Z outputs = self.rel_attn( 2025-09-07T07:19:22.5082029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.5082138Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.5082142Z 2025-09-07T07:19:22.5082255Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5082479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5082566Z return mod(**inputs) 2025-09-07T07:19:22.5082829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5082914Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5083183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5083250Z outputs = layer_module( 2025-09-07T07:19:22.5083506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5083583Z outputs = self.rel_attn( 2025-09-07T07:19:22.5083841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5083941Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5084218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.5084347Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.5084350Z 2025-09-07T07:19:22.5084464Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5084672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5084749Z return mod(**inputs) 2025-09-07T07:19:22.5085015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5085100Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5085377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5085449Z outputs = layer_module( 2025-09-07T07:19:22.5085718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5085789Z outputs = self.rel_attn( 2025-09-07T07:19:22.5086058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5086187Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5086499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5086626Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5086630Z 2025-09-07T07:19:22.5086737Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5086957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5087029Z return mod(**inputs) 2025-09-07T07:19:22.5087303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5087402Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5087682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5087761Z outputs = layer_module( 2025-09-07T07:19:22.5088045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5088127Z outputs = self.rel_attn( 2025-09-07T07:19:22.5088406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5088502Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5088815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5088939Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5088942Z 2025-09-07T07:19:22.5089061Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5089305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5089378Z return mod(**inputs) 2025-09-07T07:19:22.5089674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5089766Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5090057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5090130Z outputs = layer_module( 2025-09-07T07:19:22.5090416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5090667Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5090958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5091052Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5091338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5091425Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5091709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.5091788Z output = self.layer_1(output) 2025-09-07T07:19:22.5091792Z 2025-09-07T07:19:22.5091911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5092140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5092217Z return mod(**inputs) 2025-09-07T07:19:22.5092490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5092587Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5092863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5092952Z outputs = layer_module( 2025-09-07T07:19:22.5093263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5093475Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5093750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5093829Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5094090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5094172Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5094431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.5094525Z output = self.activation_function(output) 2025-09-07T07:19:22.5094745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.5094823Z return self.act(input) 2025-09-07T07:19:22.5094826Z 2025-09-07T07:19:22.5094932Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5095139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5095217Z return mod(**inputs) 2025-09-07T07:19:22.5095496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5095593Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5095866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5095956Z outputs = layer_module( 2025-09-07T07:19:22.5096235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5096455Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5096739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5096822Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5097101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5097196Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5097466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.5097551Z output = self.layer_2(output) 2025-09-07T07:19:22.5097555Z 2025-09-07T07:19:22.5097665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5097887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5097957Z return mod(**inputs) 2025-09-07T07:19:22.5098233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5098329Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5098599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5098678Z outputs = layer_module( 2025-09-07T07:19:22.5098951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5099027Z outputs = self.rel_attn( 2025-09-07T07:19:22.5099308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.5099415Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.5099448Z 2025-09-07T07:19:22.5099570Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5099806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5099890Z return mod(**inputs) 2025-09-07T07:19:22.5100172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5100262Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5100560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5100633Z outputs = layer_module( 2025-09-07T07:19:22.5100924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5100999Z outputs = self.rel_attn( 2025-09-07T07:19:22.5101278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.5101401Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.5101406Z 2025-09-07T07:19:22.5101517Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5101747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5101819Z return mod(**inputs) 2025-09-07T07:19:22.5102105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5102199Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5102482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5102583Z outputs = layer_module( 2025-09-07T07:19:22.5102866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5102949Z outputs = self.rel_attn( 2025-09-07T07:19:22.5103230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5103310Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5103616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.5103763Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.5103785Z 2025-09-07T07:19:22.5103907Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5104124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5104198Z return mod(**inputs) 2025-09-07T07:19:22.5104490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5104583Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5104874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5104947Z outputs = layer_module( 2025-09-07T07:19:22.5105234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5105310Z outputs = self.rel_attn( 2025-09-07T07:19:22.5105590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.5105815Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.5105825Z 2025-09-07T07:19:22.5105944Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5106175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5106271Z return mod(**inputs) 2025-09-07T07:19:22.5106580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5106684Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5106968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5107052Z outputs = layer_module( 2025-09-07T07:19:22.5107333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5107419Z outputs = self.rel_attn( 2025-09-07T07:19:22.5107695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5107777Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5108086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.5108232Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.5108236Z 2025-09-07T07:19:22.5108357Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5108579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5108653Z return mod(**inputs) 2025-09-07T07:19:22.5108945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5109039Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5109328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5109422Z outputs = layer_module( 2025-09-07T07:19:22.5109708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5109791Z outputs = self.rel_attn( 2025-09-07T07:19:22.5110080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.5110202Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.5110205Z 2025-09-07T07:19:22.5110317Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5110550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5110641Z return mod(**inputs) 2025-09-07T07:19:22.5110923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5111024Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5111309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5111392Z outputs = layer_module( 2025-09-07T07:19:22.5111679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5111754Z outputs = self.rel_attn( 2025-09-07T07:19:22.5112045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5112126Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5112432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.5112572Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.5112575Z 2025-09-07T07:19:22.5112697Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5112915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5112988Z return mod(**inputs) 2025-09-07T07:19:22.5113336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5113429Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5113722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5113794Z outputs = layer_module( 2025-09-07T07:19:22.5114071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5114154Z outputs = self.rel_attn( 2025-09-07T07:19:22.5114434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5114538Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5114844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5114975Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5114979Z 2025-09-07T07:19:22.5115093Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5115313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5115395Z return mod(**inputs) 2025-09-07T07:19:22.5115692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5115789Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5116072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5116164Z outputs = layer_module( 2025-09-07T07:19:22.5116451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5116530Z outputs = self.rel_attn( 2025-09-07T07:19:22.5116819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5116917Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5117219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5117348Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5117352Z 2025-09-07T07:19:22.5117483Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5117711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5117786Z return mod(**inputs) 2025-09-07T07:19:22.5118085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5118176Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5118472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5118552Z outputs = layer_module( 2025-09-07T07:19:22.5118831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5119071Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5119364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5119459Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5119914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5120003Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5120341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.5120450Z output = self.layer_1(output) 2025-09-07T07:19:22.5120454Z 2025-09-07T07:19:22.5120579Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5120799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5120871Z return mod(**inputs) 2025-09-07T07:19:22.5121217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5121312Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5121604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5121680Z outputs = layer_module( 2025-09-07T07:19:22.5121965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5122209Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5122511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5122610Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5122907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5122997Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5123279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.5123402Z output = self.activation_function(output) 2025-09-07T07:19:22.5123647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.5123723Z return self.act(input) 2025-09-07T07:19:22.5123728Z 2025-09-07T07:19:22.5123849Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5124083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5124155Z return mod(**inputs) 2025-09-07T07:19:22.5124456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5124546Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5124887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5124960Z outputs = layer_module( 2025-09-07T07:19:22.5125252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5125482Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5125775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5125868Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5126156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5126241Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5126522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.5126602Z output = self.layer_2(output) 2025-09-07T07:19:22.5126606Z 2025-09-07T07:19:22.5126728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5126956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5127027Z return mod(**inputs) 2025-09-07T07:19:22.5127294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5127399Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5127658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5127727Z outputs = layer_module( 2025-09-07T07:19:22.5127991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5128062Z outputs = self.rel_attn( 2025-09-07T07:19:22.5128326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.5128427Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.5128431Z 2025-09-07T07:19:22.5128534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5128744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5128821Z return mod(**inputs) 2025-09-07T07:19:22.5129080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5129160Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5129407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5129479Z outputs = layer_module( 2025-09-07T07:19:22.5129729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5129803Z outputs = self.rel_attn( 2025-09-07T07:19:22.5130068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.5130171Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.5130175Z 2025-09-07T07:19:22.5130275Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5130470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5130542Z return mod(**inputs) 2025-09-07T07:19:22.5130790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5130877Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5131125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5131211Z outputs = layer_module( 2025-09-07T07:19:22.5131473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5131543Z outputs = self.rel_attn( 2025-09-07T07:19:22.5131804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5131876Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5132159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.5132292Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.5132296Z 2025-09-07T07:19:22.5132398Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5132606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5132672Z return mod(**inputs) 2025-09-07T07:19:22.5132937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5133021Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5133303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5133379Z outputs = layer_module( 2025-09-07T07:19:22.5133648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5133724Z outputs = self.rel_attn( 2025-09-07T07:19:22.5133977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.5134111Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.5134124Z 2025-09-07T07:19:22.5134228Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5134430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5134509Z return mod(**inputs) 2025-09-07T07:19:22.5134769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5134861Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5135121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5135189Z outputs = layer_module( 2025-09-07T07:19:22.5135461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5135530Z outputs = self.rel_attn( 2025-09-07T07:19:22.5135795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5135869Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5136143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.5136302Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.5136305Z 2025-09-07T07:19:22.5136411Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5136622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5136687Z return mod(**inputs) 2025-09-07T07:19:22.5136953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5137036Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5137295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5137385Z outputs = layer_module( 2025-09-07T07:19:22.5137644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5137719Z outputs = self.rel_attn( 2025-09-07T07:19:22.5137978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.5138086Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.5138098Z 2025-09-07T07:19:22.5138203Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5138406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5138478Z return mod(**inputs) 2025-09-07T07:19:22.5138748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5138836Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5139087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5139153Z outputs = layer_module( 2025-09-07T07:19:22.5139412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5139495Z outputs = self.rel_attn( 2025-09-07T07:19:22.5139768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5139841Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5140106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.5140237Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.5140242Z 2025-09-07T07:19:22.5140344Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5140545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5140611Z return mod(**inputs) 2025-09-07T07:19:22.5140869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5140951Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5141204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5141279Z outputs = layer_module( 2025-09-07T07:19:22.5141529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5141602Z outputs = self.rel_attn( 2025-09-07T07:19:22.5141854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5141943Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5142223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5142353Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5142356Z 2025-09-07T07:19:22.5142469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5142672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5142740Z return mod(**inputs) 2025-09-07T07:19:22.5143004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5143088Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5143348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5143432Z outputs = layer_module( 2025-09-07T07:19:22.5143693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5143762Z outputs = self.rel_attn( 2025-09-07T07:19:22.5144016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5144113Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5144407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5144532Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5144535Z 2025-09-07T07:19:22.5144643Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5144855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5144932Z return mod(**inputs) 2025-09-07T07:19:22.5145204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5145299Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5145571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5145667Z outputs = layer_module( 2025-09-07T07:19:22.5146027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5146259Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5146551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5146636Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5146924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5147002Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5147277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.5147363Z output = self.layer_1(output) 2025-09-07T07:19:22.5147367Z 2025-09-07T07:19:22.5147478Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5147714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5147783Z return mod(**inputs) 2025-09-07T07:19:22.5148066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5148154Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5148428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5148513Z outputs = layer_module( 2025-09-07T07:19:22.5148784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5149045Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5149332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5149416Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5149698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5149775Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5150053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.5150165Z output = self.activation_function(output) 2025-09-07T07:19:22.5150407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.5150485Z return self.act(input) 2025-09-07T07:19:22.5150488Z 2025-09-07T07:19:22.5150598Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5150823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5150892Z return mod(**inputs) 2025-09-07T07:19:22.5151180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5151268Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5151541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5151619Z outputs = layer_module( 2025-09-07T07:19:22.5151896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5152125Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5152411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5152519Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5152837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5152914Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5153194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.5153272Z output = self.layer_2(output) 2025-09-07T07:19:22.5153275Z 2025-09-07T07:19:22.5153393Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5153608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5153679Z return mod(**inputs) 2025-09-07T07:19:22.5153965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5154055Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5154339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5154409Z outputs = layer_module( 2025-09-07T07:19:22.5154684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5154765Z outputs = self.rel_attn( 2025-09-07T07:19:22.5155040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-09-07T07:19:22.5155155Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-09-07T07:19:22.5155159Z 2025-09-07T07:19:22.5155269Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5155509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5155577Z return mod(**inputs) 2025-09-07T07:19:22.5155850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5155963Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5156222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5156296Z outputs = layer_module( 2025-09-07T07:19:22.5156565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5156654Z outputs = self.rel_attn( 2025-09-07T07:19:22.5156916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-09-07T07:19:22.5157018Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-09-07T07:19:22.5157021Z 2025-09-07T07:19:22.5157128Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5157326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5157399Z return mod(**inputs) 2025-09-07T07:19:22.5157650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5157730Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5157991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5158058Z outputs = layer_module( 2025-09-07T07:19:22.5158325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5158394Z outputs = self.rel_attn( 2025-09-07T07:19:22.5158651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5158734Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5159026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-09-07T07:19:22.5159193Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-09-07T07:19:22.5159197Z 2025-09-07T07:19:22.5159298Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5159500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5159564Z return mod(**inputs) 2025-09-07T07:19:22.5159816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5159904Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5160156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5160228Z outputs = layer_module( 2025-09-07T07:19:22.5160478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5160543Z outputs = self.rel_attn( 2025-09-07T07:19:22.5160796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-09-07T07:19:22.5160928Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-09-07T07:19:22.5160932Z 2025-09-07T07:19:22.5161038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5161231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5161296Z return mod(**inputs) 2025-09-07T07:19:22.5161549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5161651Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5161910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5161977Z outputs = layer_module( 2025-09-07T07:19:22.5162241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5162309Z outputs = self.rel_attn( 2025-09-07T07:19:22.5162566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5162646Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5162939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-09-07T07:19:22.5163074Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-09-07T07:19:22.5163079Z 2025-09-07T07:19:22.5163184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5163389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5163464Z return mod(**inputs) 2025-09-07T07:19:22.5163717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5163805Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5164053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5164126Z outputs = layer_module( 2025-09-07T07:19:22.5164375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5164442Z outputs = self.rel_attn( 2025-09-07T07:19:22.5164703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-09-07T07:19:22.5164806Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-09-07T07:19:22.5164809Z 2025-09-07T07:19:22.5164934Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5165154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5165222Z return mod(**inputs) 2025-09-07T07:19:22.5165488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5165570Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5165833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5165902Z outputs = layer_module( 2025-09-07T07:19:22.5166157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5166235Z outputs = self.rel_attn( 2025-09-07T07:19:22.5166489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-09-07T07:19:22.5166569Z attn_vec = self.rel_attn_core( 2025-09-07T07:19:22.5166847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-09-07T07:19:22.5166979Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-09-07T07:19:22.5166982Z 2025-09-07T07:19:22.5167086Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5167288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5167365Z return mod(**inputs) 2025-09-07T07:19:22.5167625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5167734Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5167992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5168070Z outputs = layer_module( 2025-09-07T07:19:22.5168333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5168402Z outputs = self.rel_attn( 2025-09-07T07:19:22.5168656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5168743Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5169024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5169151Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5169156Z 2025-09-07T07:19:22.5169259Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5169465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5169532Z return mod(**inputs) 2025-09-07T07:19:22.5169792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5169877Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5170129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5170205Z outputs = layer_module( 2025-09-07T07:19:22.5170454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-09-07T07:19:22.5170533Z outputs = self.rel_attn( 2025-09-07T07:19:22.5170784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-09-07T07:19:22.5170874Z output_h = self.post_attention(h, attn_vec) 2025-09-07T07:19:22.5171170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-09-07T07:19:22.5171305Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-09-07T07:19:22.5171309Z 2025-09-07T07:19:22.5171418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5171616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5171688Z return mod(**inputs) 2025-09-07T07:19:22.5171939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5172021Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5172281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5172348Z outputs = layer_module( 2025-09-07T07:19:22.5172609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5172818Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5173084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5173161Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5173413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5173495Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5173752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-09-07T07:19:22.5173853Z output = self.layer_1(output) 2025-09-07T07:19:22.5173856Z 2025-09-07T07:19:22.5173960Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5174167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5174243Z return mod(**inputs) 2025-09-07T07:19:22.5174520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5174614Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5174886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5174955Z outputs = layer_module( 2025-09-07T07:19:22.5175246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5175475Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5175749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5175825Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5176093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5176165Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5176419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-09-07T07:19:22.5176517Z output = self.activation_function(output) 2025-09-07T07:19:22.5176735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:19:22.5176816Z return self.act(input) 2025-09-07T07:19:22.5176820Z 2025-09-07T07:19:22.5176923Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5177125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5177196Z return mod(**inputs) 2025-09-07T07:19:22.5177487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-09-07T07:19:22.5177600Z transformer_outputs = self.transformer( 2025-09-07T07:19:22.5177876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-09-07T07:19:22.5177953Z outputs = layer_module( 2025-09-07T07:19:22.5178231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-09-07T07:19:22.5178457Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-09-07T07:19:22.5178745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:19:22.5178829Z return forward_fn(*input_tensors) 2025-09-07T07:19:22.5179117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-09-07T07:19:22.5179194Z output_x = self.ff(output_x) 2025-09-07T07:19:22.5179471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-09-07T07:19:22.5179557Z output = self.layer_2(output) 2025-09-07T07:19:22.5179561Z 2025-09-07T07:19:22.5179672Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5179894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5179965Z return mod(**inputs) 2025-09-07T07:19:22.5180246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1624, in forward 2025-09-07T07:19:22.5180364Z logits = self.lm_loss(transformer_outputs[0]) 2025-09-07T07:19:22.5180368Z 2025-09-07T07:19:22.5180476Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:19:22.5180696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:19:22.5180767Z return mod(**inputs) 2025-09-07T07:19:22.5181050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1630, in forward 2025-09-07T07:19:22.5181193Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-09-07T07:19:22.5181197Z 2025-09-07T07:19:38.2182083Z Compilation time (from dynamo_timed): 35.509433076 2025-09-07T07:19:38.2227080Z pass 2025-09-07T07:19:38.2227563Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:19:38.2228838Z TIMING: _recursive_pre_grad_passes:0.0133 _recursive_joint_graph_passes:1.36187 _recursive_post_grad_passes:0.23794 async_compile.wait:0.82337 code_gen:14.34746 inductor_compile:19.45005 backend_compile:29.24751 gc:0.00175 entire_frame_compile:35.50943 total_wall_time:35.50943 2025-09-07T07:19:38.2229969Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:56659 | FakeTensor.__torch_dispatch__:15989 | ProxyTorchDispatchMode.__torch_dispatch__:18623 2025-09-07T07:19:38.2230605Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-09-07T07:19:41.6565920Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-09-07T07:19:41.6567816Z import pynvml # type: ignore[import] 2025-09-07T07:19:44.4333872Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/librosa/util/files.py:10: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-09-07T07:19:44.4335018Z from pkg_resources import resource_filename 2025-09-07T07:19:45.0938219Z 2025-09-07T07:19:46.3835970Z loading model: 0it [00:00, ?it/s] 2025-09-07T07:19:46.3842582Z loading model: 0it [00:01, ?it/s] 2025-09-07T07:19:46.3860849Z cpu eval YituTechConvBert 2025-09-07T07:19:47.3607569Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:19:47.6367407Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:19:47.9210581Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:20:00.5569733Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5570460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5570939Z return mod(**inputs) 2025-09-07T07:20:00.5572112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5572768Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5573276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5573732Z hidden_states = self.encoder( 2025-09-07T07:20:00.5574191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5574642Z layer_outputs = layer_module( 2025-09-07T07:20:00.5579332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5579880Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5580728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5581186Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5581630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5582077Z self_outputs = self.self( 2025-09-07T07:20:00.5582493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.5582936Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.5583099Z 2025-09-07T07:20:00.5583222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5583675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5584044Z return mod(**inputs) 2025-09-07T07:20:00.5584470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5584963Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5585420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5586067Z hidden_states = self.encoder( 2025-09-07T07:20:00.5586517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5586959Z layer_outputs = layer_module( 2025-09-07T07:20:00.5587345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5587747Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5588205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5588654Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5589145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5589580Z self_outputs = self.self( 2025-09-07T07:20:00.5590324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.5590801Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.5590970Z 2025-09-07T07:20:00.5591094Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5591509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5591892Z return mod(**inputs) 2025-09-07T07:20:00.5592322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5592778Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5593246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5593701Z hidden_states = self.encoder( 2025-09-07T07:20:00.5594149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5594603Z layer_outputs = layer_module( 2025-09-07T07:20:00.5594989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5595396Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5595847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5596310Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5596745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5597227Z self_outputs = self.self( 2025-09-07T07:20:00.5597615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.5598037Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.5598185Z 2025-09-07T07:20:00.5598271Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5598489Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5598728Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5599088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5599414Z return mod(**inputs) 2025-09-07T07:20:00.5599823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5600247Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5600666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5601070Z hidden_states = self.encoder( 2025-09-07T07:20:00.5601467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5601869Z layer_outputs = layer_module( 2025-09-07T07:20:00.5602231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5602610Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5603036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5603451Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5603867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5604277Z self_outputs = self.self( 2025-09-07T07:20:00.5604675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.5605132Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.5605303Z 2025-09-07T07:20:00.5605407Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5605655Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5606031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5606359Z return mod(**inputs) 2025-09-07T07:20:00.5606732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5607144Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5607550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5607980Z hidden_states = self.encoder( 2025-09-07T07:20:00.5608401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5608809Z layer_outputs = layer_module( 2025-09-07T07:20:00.5609165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5609548Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5609995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5610410Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5610833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5611244Z self_outputs = self.self( 2025-09-07T07:20:00.5611666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5612167Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5612669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.5613085Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.5613226Z 2025-09-07T07:20:00.5613332Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5613725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5614071Z return mod(**inputs) 2025-09-07T07:20:00.5615327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5615758Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5616181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5616589Z hidden_states = self.encoder( 2025-09-07T07:20:00.5616990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5617401Z layer_outputs = layer_module( 2025-09-07T07:20:00.5617759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5618135Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5618552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5618970Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5619385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5620147Z self_outputs = self.self( 2025-09-07T07:20:00.5620577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5621152Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5621704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.5622150Z x = self.pointwise(x) 2025-09-07T07:20:00.5622283Z 2025-09-07T07:20:00.5622403Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5622812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5623170Z return mod(**inputs) 2025-09-07T07:20:00.5623573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5624023Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5624469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5624908Z hidden_states = self.encoder( 2025-09-07T07:20:00.5625339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5625862Z layer_outputs = layer_module( 2025-09-07T07:20:00.5626259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5626670Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5627131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5627581Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5628083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5628537Z self_outputs = self.self( 2025-09-07T07:20:00.5629012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.5629621Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.5629854Z 2025-09-07T07:20:00.5629971Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5630379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5630746Z return mod(**inputs) 2025-09-07T07:20:00.5631201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5631660Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5632110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5632556Z hidden_states = self.encoder( 2025-09-07T07:20:00.5632997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5633454Z layer_outputs = layer_module( 2025-09-07T07:20:00.5633824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5634222Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5634657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5635089Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5635504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5635903Z self_outputs = self.self( 2025-09-07T07:20:00.5636313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.5636825Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.5637005Z 2025-09-07T07:20:00.5637154Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5637539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5637877Z return mod(**inputs) 2025-09-07T07:20:00.5638274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5638716Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5639146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5639564Z hidden_states = self.encoder( 2025-09-07T07:20:00.5639979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5640406Z layer_outputs = layer_module( 2025-09-07T07:20:00.5640778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5641159Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5641574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5642005Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5642431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5642855Z self_outputs = self.self( 2025-09-07T07:20:00.5643289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.5643797Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.5643999Z 2025-09-07T07:20:00.5644089Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5644324Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5644582Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5644968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5645343Z return mod(**inputs) 2025-09-07T07:20:00.5645762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5646238Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5646693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5647131Z hidden_states = self.encoder( 2025-09-07T07:20:00.5647572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5648011Z layer_outputs = layer_module( 2025-09-07T07:20:00.5648391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5648783Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5649227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5649670Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5650127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5650556Z self_outputs = self.self( 2025-09-07T07:20:00.5650981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.5651461Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.5651648Z 2025-09-07T07:20:00.5651784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5652199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5652555Z return mod(**inputs) 2025-09-07T07:20:00.5652975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5653435Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5653895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5654347Z hidden_states = self.encoder( 2025-09-07T07:20:00.5654783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5655218Z layer_outputs = layer_module( 2025-09-07T07:20:00.5655606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5656017Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5656465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5656909Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5657358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.5657861Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.5658366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.5658840Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5658991Z 2025-09-07T07:20:00.5659104Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5659500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5659857Z return mod(**inputs) 2025-09-07T07:20:00.5660275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5660721Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5661154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5661566Z hidden_states = self.encoder( 2025-09-07T07:20:00.5661987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5662397Z layer_outputs = layer_module( 2025-09-07T07:20:00.5662746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5663116Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5663531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5663959Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5664404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5664826Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5665298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5665985Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5666510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.5666987Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5667151Z 2025-09-07T07:20:00.5667262Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5667713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5668054Z return mod(**inputs) 2025-09-07T07:20:00.5668456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5668894Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5669323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5669750Z hidden_states = self.encoder( 2025-09-07T07:20:00.5670171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5670603Z layer_outputs = layer_module( 2025-09-07T07:20:00.5670970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5671364Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5671796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5672236Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5672668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5673079Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5673548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5674060Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5674546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.5675001Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.5675391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.5675753Z return self.act(input) 2025-09-07T07:20:00.5675883Z 2025-09-07T07:20:00.5675996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5676389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5676743Z return mod(**inputs) 2025-09-07T07:20:00.5677159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5677635Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5678074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5678489Z hidden_states = self.encoder( 2025-09-07T07:20:00.5678897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5679346Z layer_outputs = layer_module( 2025-09-07T07:20:00.5679725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5680131Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5680580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5681033Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5681477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5681913Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5682384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.5682940Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.5683471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.5683929Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5684086Z 2025-09-07T07:20:00.5684200Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5684606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5684958Z return mod(**inputs) 2025-09-07T07:20:00.5685378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5685830Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5686285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5686727Z hidden_states = self.encoder( 2025-09-07T07:20:00.5687146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5687576Z layer_outputs = layer_module( 2025-09-07T07:20:00.5687951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5688356Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5688807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5689246Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5689688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5690103Z self_outputs = self.self( 2025-09-07T07:20:00.5690507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.5690930Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.5691088Z 2025-09-07T07:20:00.5691202Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5691596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5691966Z return mod(**inputs) 2025-09-07T07:20:00.5692373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5692821Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5693273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5693692Z hidden_states = self.encoder( 2025-09-07T07:20:00.5694102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5694534Z layer_outputs = layer_module( 2025-09-07T07:20:00.5694904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5695300Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5695753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5696209Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5696637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5697051Z self_outputs = self.self( 2025-09-07T07:20:00.5697455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.5697898Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.5698086Z 2025-09-07T07:20:00.5698208Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5698610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5698963Z return mod(**inputs) 2025-09-07T07:20:00.5699371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5699812Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5700251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5700678Z hidden_states = self.encoder( 2025-09-07T07:20:00.5701110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5701542Z layer_outputs = layer_module( 2025-09-07T07:20:00.5701947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5702333Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5702775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5703213Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5703651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5704086Z self_outputs = self.self( 2025-09-07T07:20:00.5704502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.5704982Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.5705153Z 2025-09-07T07:20:00.5705243Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5705484Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5705812Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5706218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5706584Z return mod(**inputs) 2025-09-07T07:20:00.5707004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5707454Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5707893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5708352Z hidden_states = self.encoder( 2025-09-07T07:20:00.5708781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5709214Z layer_outputs = layer_module( 2025-09-07T07:20:00.5709591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5709983Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5710425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5710871Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5711310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5711744Z self_outputs = self.self( 2025-09-07T07:20:00.5712155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.5712627Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.5712805Z 2025-09-07T07:20:00.5712895Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5713157Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5713565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5713951Z return mod(**inputs) 2025-09-07T07:20:00.5714378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5714833Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5715276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5715710Z hidden_states = self.encoder( 2025-09-07T07:20:00.5716145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5716585Z layer_outputs = layer_module( 2025-09-07T07:20:00.5716961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5717329Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5717741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5718153Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5718572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5718988Z self_outputs = self.self( 2025-09-07T07:20:00.5719383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5720087Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5720655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.5721061Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.5721196Z 2025-09-07T07:20:00.5721310Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5721669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5722006Z return mod(**inputs) 2025-09-07T07:20:00.5722392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5722807Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5723238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5723630Z hidden_states = self.encoder( 2025-09-07T07:20:00.5724024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5724426Z layer_outputs = layer_module( 2025-09-07T07:20:00.5724776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5725141Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5725557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5725976Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5726406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5726845Z self_outputs = self.self( 2025-09-07T07:20:00.5727231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5727722Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5728253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.5728656Z x = self.pointwise(x) 2025-09-07T07:20:00.5728772Z 2025-09-07T07:20:00.5728916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5729290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5729616Z return mod(**inputs) 2025-09-07T07:20:00.5729996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5730415Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5730817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5731228Z hidden_states = self.encoder( 2025-09-07T07:20:00.5731620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5732024Z layer_outputs = layer_module( 2025-09-07T07:20:00.5732374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5732724Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5733127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5733539Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5733947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5734356Z self_outputs = self.self( 2025-09-07T07:20:00.5734742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.5735241Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.5735457Z 2025-09-07T07:20:00.5735562Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5735929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5736261Z return mod(**inputs) 2025-09-07T07:20:00.5736637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5737059Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5737472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5737894Z hidden_states = self.encoder( 2025-09-07T07:20:00.5738274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5738670Z layer_outputs = layer_module( 2025-09-07T07:20:00.5739014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5739373Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5739772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5740167Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5740572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5740974Z self_outputs = self.self( 2025-09-07T07:20:00.5741364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.5741819Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.5741997Z 2025-09-07T07:20:00.5742102Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5742488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5742819Z return mod(**inputs) 2025-09-07T07:20:00.5743257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5743674Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5744091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5744499Z hidden_states = self.encoder( 2025-09-07T07:20:00.5744909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5745337Z layer_outputs = layer_module( 2025-09-07T07:20:00.5745743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5746164Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5746613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5747091Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5747509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5747960Z self_outputs = self.self( 2025-09-07T07:20:00.5748353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.5748811Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.5748989Z 2025-09-07T07:20:00.5749104Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5749313Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5749553Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5749913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5750241Z return mod(**inputs) 2025-09-07T07:20:00.5750625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5751028Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5751440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5751841Z hidden_states = self.encoder( 2025-09-07T07:20:00.5752278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5752676Z layer_outputs = layer_module( 2025-09-07T07:20:00.5753024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5753386Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5753796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5754212Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5754611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5755007Z self_outputs = self.self( 2025-09-07T07:20:00.5755406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.5755867Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.5756037Z 2025-09-07T07:20:00.5756153Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5756523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5756849Z return mod(**inputs) 2025-09-07T07:20:00.5757247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5757688Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5758085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5758482Z hidden_states = self.encoder( 2025-09-07T07:20:00.5758878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5759277Z layer_outputs = layer_module( 2025-09-07T07:20:00.5759624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5759975Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5760376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5760786Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5761200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.5761656Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.5762099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.5762506Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5762655Z 2025-09-07T07:20:00.5762759Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5763119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5763479Z return mod(**inputs) 2025-09-07T07:20:00.5763847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5764263Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5764668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5765067Z hidden_states = self.encoder( 2025-09-07T07:20:00.5765452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5765864Z layer_outputs = layer_module( 2025-09-07T07:20:00.5766243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5766617Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5767019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5767424Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5767831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5768227Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5768661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5769147Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5769595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.5770011Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5770157Z 2025-09-07T07:20:00.5770265Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5770631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5770947Z return mod(**inputs) 2025-09-07T07:20:00.5771359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5771792Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5772202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5772608Z hidden_states = self.encoder( 2025-09-07T07:20:00.5772994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5773392Z layer_outputs = layer_module( 2025-09-07T07:20:00.5773741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5774105Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5774524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5774940Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5775354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5775762Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5776206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5776691Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5777129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.5777562Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.5777969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.5778316Z return self.act(input) 2025-09-07T07:20:00.5778429Z 2025-09-07T07:20:00.5778534Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5778893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5779217Z return mod(**inputs) 2025-09-07T07:20:00.5779592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5780003Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5780416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5780845Z hidden_states = self.encoder( 2025-09-07T07:20:00.5781246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5781662Z layer_outputs = layer_module( 2025-09-07T07:20:00.5782010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5782382Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5782793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5783235Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5783669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5784113Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5784582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.5785166Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.5785673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.5786232Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5786394Z 2025-09-07T07:20:00.5786533Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5786954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5787320Z return mod(**inputs) 2025-09-07T07:20:00.5787738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5788228Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5788643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5789059Z hidden_states = self.encoder( 2025-09-07T07:20:00.5789459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5789870Z layer_outputs = layer_module( 2025-09-07T07:20:00.5790219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5790591Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5791029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5791482Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5791931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5792338Z self_outputs = self.self( 2025-09-07T07:20:00.5792743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.5793196Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.5793347Z 2025-09-07T07:20:00.5793468Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5793851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5794198Z return mod(**inputs) 2025-09-07T07:20:00.5794623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5795072Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5795512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5795957Z hidden_states = self.encoder( 2025-09-07T07:20:00.5796398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5796836Z layer_outputs = layer_module( 2025-09-07T07:20:00.5797217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5797615Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5798024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5798447Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5798864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5799330Z self_outputs = self.self( 2025-09-07T07:20:00.5799750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.5800200Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.5800360Z 2025-09-07T07:20:00.5800476Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5800849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5801204Z return mod(**inputs) 2025-09-07T07:20:00.5801600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5802019Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5802447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5802883Z hidden_states = self.encoder( 2025-09-07T07:20:00.5803305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5803734Z layer_outputs = layer_module( 2025-09-07T07:20:00.5804115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5804511Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5804949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5805396Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5805833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5806270Z self_outputs = self.self( 2025-09-07T07:20:00.5806665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.5807098Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.5807248Z 2025-09-07T07:20:00.5807343Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5807558Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5807844Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5808216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5808552Z return mod(**inputs) 2025-09-07T07:20:00.5808932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5809361Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5809780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5810189Z hidden_states = self.encoder( 2025-09-07T07:20:00.5810584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5811015Z layer_outputs = layer_module( 2025-09-07T07:20:00.5811379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5811758Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5812177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5812591Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5813000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5813401Z self_outputs = self.self( 2025-09-07T07:20:00.5813790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.5814227Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.5814385Z 2025-09-07T07:20:00.5814467Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5814705Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5815069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5815395Z return mod(**inputs) 2025-09-07T07:20:00.5815802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5816251Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5816673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5817087Z hidden_states = self.encoder( 2025-09-07T07:20:00.5817501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5817907Z layer_outputs = layer_module( 2025-09-07T07:20:00.5818263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5818638Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5819054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5819475Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5820079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5820493Z self_outputs = self.self( 2025-09-07T07:20:00.5820891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5821396Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5821901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.5822312Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.5822502Z 2025-09-07T07:20:00.5822609Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5822976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5823305Z return mod(**inputs) 2025-09-07T07:20:00.5823681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5824101Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5824537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5824974Z hidden_states = self.encoder( 2025-09-07T07:20:00.5825402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5825932Z layer_outputs = layer_module( 2025-09-07T07:20:00.5826320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5826716Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5827159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5827607Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5828051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5828490Z self_outputs = self.self( 2025-09-07T07:20:00.5828909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5829422Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5829922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.5830358Z x = self.pointwise(x) 2025-09-07T07:20:00.5830488Z 2025-09-07T07:20:00.5830599Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5831027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5831381Z return mod(**inputs) 2025-09-07T07:20:00.5831820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5832268Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5832721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5833161Z hidden_states = self.encoder( 2025-09-07T07:20:00.5833601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5834036Z layer_outputs = layer_module( 2025-09-07T07:20:00.5834415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5834813Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5835262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5835701Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5836153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5836588Z self_outputs = self.self( 2025-09-07T07:20:00.5837023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.5837552Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.5837801Z 2025-09-07T07:20:00.5837916Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5838307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5838675Z return mod(**inputs) 2025-09-07T07:20:00.5839086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5839525Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5839969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5840447Z hidden_states = self.encoder( 2025-09-07T07:20:00.5840851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5841275Z layer_outputs = layer_module( 2025-09-07T07:20:00.5841621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5841995Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5842410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5842830Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5843247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5843648Z self_outputs = self.self( 2025-09-07T07:20:00.5844047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.5844509Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.5844692Z 2025-09-07T07:20:00.5844805Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5845178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5845508Z return mod(**inputs) 2025-09-07T07:20:00.5845901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5846376Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5846832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5847274Z hidden_states = self.encoder( 2025-09-07T07:20:00.5847698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5848128Z layer_outputs = layer_module( 2025-09-07T07:20:00.5848488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5848863Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5849273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5849707Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5850131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5850545Z self_outputs = self.self( 2025-09-07T07:20:00.5850946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.5851403Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.5851597Z 2025-09-07T07:20:00.5851686Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5851914Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5852158Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5852522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5852889Z return mod(**inputs) 2025-09-07T07:20:00.5853317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5853784Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5854224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5854648Z hidden_states = self.encoder( 2025-09-07T07:20:00.5855082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5855580Z layer_outputs = layer_module( 2025-09-07T07:20:00.5855959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5856348Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5856793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5857247Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5857699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5858137Z self_outputs = self.self( 2025-09-07T07:20:00.5858545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.5859030Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.5859215Z 2025-09-07T07:20:00.5859327Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5859717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5860070Z return mod(**inputs) 2025-09-07T07:20:00.5860478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5860932Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5861400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5861863Z hidden_states = self.encoder( 2025-09-07T07:20:00.5862308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5862742Z layer_outputs = layer_module( 2025-09-07T07:20:00.5863120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5863519Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5863969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5864406Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5864845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.5865339Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.5865905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.5866358Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5866509Z 2025-09-07T07:20:00.5866623Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5867019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5867375Z return mod(**inputs) 2025-09-07T07:20:00.5867784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5868268Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5868666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5869071Z hidden_states = self.encoder( 2025-09-07T07:20:00.5869463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5869859Z layer_outputs = layer_module( 2025-09-07T07:20:00.5870211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5870569Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5870974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5871404Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5871809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5872202Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5872639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5873120Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5873568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.5873980Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5874118Z 2025-09-07T07:20:00.5874222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5874581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5874911Z return mod(**inputs) 2025-09-07T07:20:00.5875300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5875721Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5876154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5876586Z hidden_states = self.encoder( 2025-09-07T07:20:00.5876989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5877401Z layer_outputs = layer_module( 2025-09-07T07:20:00.5877738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5878096Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5878499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5878914Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5879316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5879702Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5880133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5880614Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5881064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.5881502Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.5881880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.5882224Z return self.act(input) 2025-09-07T07:20:00.5882365Z 2025-09-07T07:20:00.5882470Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5882836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5883168Z return mod(**inputs) 2025-09-07T07:20:00.5883542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5883960Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5884390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5884788Z hidden_states = self.encoder( 2025-09-07T07:20:00.5885179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5885600Z layer_outputs = layer_module( 2025-09-07T07:20:00.5885956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5886329Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5886754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5887166Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5887581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5887985Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5888424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.5888929Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.5889391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.5889814Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5889963Z 2025-09-07T07:20:00.5890070Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5890460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5890791Z return mod(**inputs) 2025-09-07T07:20:00.5891239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5891672Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5892098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5892521Z hidden_states = self.encoder( 2025-09-07T07:20:00.5892924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5893343Z layer_outputs = layer_module( 2025-09-07T07:20:00.5893708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5894088Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5894512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5894937Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5895367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5895791Z self_outputs = self.self( 2025-09-07T07:20:00.5896200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.5896640Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.5896791Z 2025-09-07T07:20:00.5896897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5897298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5897631Z return mod(**inputs) 2025-09-07T07:20:00.5898025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5898442Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5898858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5899271Z hidden_states = self.encoder( 2025-09-07T07:20:00.5899675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5900103Z layer_outputs = layer_module( 2025-09-07T07:20:00.5900459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5900841Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5901269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5901700Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5902118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5902533Z self_outputs = self.self( 2025-09-07T07:20:00.5902939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.5903403Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.5903559Z 2025-09-07T07:20:00.5903685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5904077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5904436Z return mod(**inputs) 2025-09-07T07:20:00.5904864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5905326Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5905901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5906351Z hidden_states = self.encoder( 2025-09-07T07:20:00.5906791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5907239Z layer_outputs = layer_module( 2025-09-07T07:20:00.5907617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5907994Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5908405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5908821Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5909251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5909662Z self_outputs = self.self( 2025-09-07T07:20:00.5910049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.5910480Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.5910639Z 2025-09-07T07:20:00.5910724Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5910950Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5911207Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5911564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5911911Z return mod(**inputs) 2025-09-07T07:20:00.5912286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5912694Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5913097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5913495Z hidden_states = self.encoder( 2025-09-07T07:20:00.5913886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5914285Z layer_outputs = layer_module( 2025-09-07T07:20:00.5914631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5915007Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5915410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5915824Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5916245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5916651Z self_outputs = self.self( 2025-09-07T07:20:00.5917051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.5917493Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.5917659Z 2025-09-07T07:20:00.5917742Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5917996Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5918351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5918685Z return mod(**inputs) 2025-09-07T07:20:00.5919084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5919499Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5920202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5920628Z hidden_states = self.encoder( 2025-09-07T07:20:00.5921024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5921427Z layer_outputs = layer_module( 2025-09-07T07:20:00.5921778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5922136Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5922548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5922968Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5923374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5923773Z self_outputs = self.self( 2025-09-07T07:20:00.5924155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5924643Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5925135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.5925553Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.5925690Z 2025-09-07T07:20:00.5925805Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5926173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5926541Z return mod(**inputs) 2025-09-07T07:20:00.5926926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5927327Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5927722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5928105Z hidden_states = self.encoder( 2025-09-07T07:20:00.5928496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5928898Z layer_outputs = layer_module( 2025-09-07T07:20:00.5929239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5929611Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5929999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5930399Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5930794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5931180Z self_outputs = self.self( 2025-09-07T07:20:00.5931552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.5932020Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.5932494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.5932891Z x = self.pointwise(x) 2025-09-07T07:20:00.5933001Z 2025-09-07T07:20:00.5933112Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5933468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5933803Z return mod(**inputs) 2025-09-07T07:20:00.5934203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5934613Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5935024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5935422Z hidden_states = self.encoder( 2025-09-07T07:20:00.5935822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5936220Z layer_outputs = layer_module( 2025-09-07T07:20:00.5936568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5936926Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5937337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5937747Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5938156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5938553Z self_outputs = self.self( 2025-09-07T07:20:00.5938938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.5939436Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.5939657Z 2025-09-07T07:20:00.5939766Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5940148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5940472Z return mod(**inputs) 2025-09-07T07:20:00.5940865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5941274Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5941691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5942095Z hidden_states = self.encoder( 2025-09-07T07:20:00.5942548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5942955Z layer_outputs = layer_module( 2025-09-07T07:20:00.5943314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5943709Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5944131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5944549Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5944972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5945423Z self_outputs = self.self( 2025-09-07T07:20:00.5946009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.5946582Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.5947039Z 2025-09-07T07:20:00.5947220Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5947661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5948082Z return mod(**inputs) 2025-09-07T07:20:00.5964741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5965462Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5965906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5966426Z hidden_states = self.encoder( 2025-09-07T07:20:00.5966918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5967381Z layer_outputs = layer_module( 2025-09-07T07:20:00.5967765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5968179Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5968629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5969061Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5969488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5969897Z self_outputs = self.self( 2025-09-07T07:20:00.5970318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.5970785Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.5970975Z 2025-09-07T07:20:00.5971071Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5971297Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.5971537Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5971910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5972246Z return mod(**inputs) 2025-09-07T07:20:00.5972637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5973090Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5973502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5973905Z hidden_states = self.encoder( 2025-09-07T07:20:00.5974302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5974700Z layer_outputs = layer_module( 2025-09-07T07:20:00.5975046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5975422Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5975839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5976297Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5976714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.5977129Z self_outputs = self.self( 2025-09-07T07:20:00.5977531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.5977988Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.5978172Z 2025-09-07T07:20:00.5978294Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5978672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5979021Z return mod(**inputs) 2025-09-07T07:20:00.5979421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5979856Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5980286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5980701Z hidden_states = self.encoder( 2025-09-07T07:20:00.5981134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5981541Z layer_outputs = layer_module( 2025-09-07T07:20:00.5981914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5982276Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5982692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.5983110Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.5983528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.5983995Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.5984451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.5984875Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5985025Z 2025-09-07T07:20:00.5985136Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5985519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5985978Z return mod(**inputs) 2025-09-07T07:20:00.5986391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5986840Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5987294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5987750Z hidden_states = self.encoder( 2025-09-07T07:20:00.5988160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5988560Z layer_outputs = layer_module( 2025-09-07T07:20:00.5988918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5989292Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5989712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5990130Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5990549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5990966Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5991402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5991887Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5992334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.5992751Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.5992895Z 2025-09-07T07:20:00.5992999Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.5993361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.5993684Z return mod(**inputs) 2025-09-07T07:20:00.5994057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.5994468Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.5994877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.5995286Z hidden_states = self.encoder( 2025-09-07T07:20:00.5995750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.5996151Z layer_outputs = layer_module( 2025-09-07T07:20:00.5996529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.5996891Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.5997293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.5997700Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.5998107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.5998503Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.5998938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.5999420Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.5999861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6000306Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6000689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6001031Z return self.act(input) 2025-09-07T07:20:00.6001146Z 2025-09-07T07:20:00.6001256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6001615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6001949Z return mod(**inputs) 2025-09-07T07:20:00.6002372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6002786Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6003209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6003640Z hidden_states = self.encoder( 2025-09-07T07:20:00.6004081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6004526Z layer_outputs = layer_module( 2025-09-07T07:20:00.6004901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6005294Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6005705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6006132Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6006542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6006953Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6007413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6007942Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6008441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6008885Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6009037Z 2025-09-07T07:20:00.6009150Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6009514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6009852Z return mod(**inputs) 2025-09-07T07:20:00.6010245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6010690Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6011012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6011091Z hidden_states = self.encoder( 2025-09-07T07:20:00.6011371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6011445Z layer_outputs = layer_module( 2025-09-07T07:20:00.6011689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6011770Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6012060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6012146Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6012426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6012510Z self_outputs = self.self( 2025-09-07T07:20:00.6012789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6012892Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6012896Z 2025-09-07T07:20:00.6013004Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6013217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6013295Z return mod(**inputs) 2025-09-07T07:20:00.6013574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6013680Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6013954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6014036Z hidden_states = self.encoder( 2025-09-07T07:20:00.6014308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6014379Z layer_outputs = layer_module( 2025-09-07T07:20:00.6014613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6014710Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6014987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6015071Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6015341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6015423Z self_outputs = self.self( 2025-09-07T07:20:00.6015693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6015787Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6015791Z 2025-09-07T07:20:00.6015897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6016100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6016175Z return mod(**inputs) 2025-09-07T07:20:00.6016445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6016537Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6016808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6016887Z hidden_states = self.encoder( 2025-09-07T07:20:00.6017194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6017269Z layer_outputs = layer_module( 2025-09-07T07:20:00.6017504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6017582Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6017861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6017943Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6018214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6018294Z self_outputs = self.self( 2025-09-07T07:20:00.6018564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6018664Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6018669Z 2025-09-07T07:20:00.6018753Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6019225Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6019333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6019694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6019778Z return mod(**inputs) 2025-09-07T07:20:00.6020045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6020130Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6020456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6020543Z hidden_states = self.encoder( 2025-09-07T07:20:00.6020803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6020885Z layer_outputs = layer_module( 2025-09-07T07:20:00.6021106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6021185Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6021458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6021568Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6021843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6021915Z self_outputs = self.self( 2025-09-07T07:20:00.6022179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6022294Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6022298Z 2025-09-07T07:20:00.6022379Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6022489Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6022697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6022773Z return mod(**inputs) 2025-09-07T07:20:00.6023050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6023136Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6023415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6023488Z hidden_states = self.encoder( 2025-09-07T07:20:00.6023787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6023860Z layer_outputs = layer_module( 2025-09-07T07:20:00.6024111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6024201Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6024471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6024560Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6024832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6024901Z self_outputs = self.self( 2025-09-07T07:20:00.6025196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6025378Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6025677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6025815Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6025822Z 2025-09-07T07:20:00.6025949Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6026164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6026236Z return mod(**inputs) 2025-09-07T07:20:00.6026532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6026618Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6026940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6027010Z hidden_states = self.encoder( 2025-09-07T07:20:00.6027276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6027352Z layer_outputs = layer_module( 2025-09-07T07:20:00.6027571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6027655Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6027919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6028025Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6028292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6028361Z self_outputs = self.self( 2025-09-07T07:20:00.6028626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6028785Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6029050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6029119Z x = self.pointwise(x) 2025-09-07T07:20:00.6029122Z 2025-09-07T07:20:00.6029222Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6029421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6029487Z return mod(**inputs) 2025-09-07T07:20:00.6029751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6029831Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6030094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6030178Z hidden_states = self.encoder( 2025-09-07T07:20:00.6030447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6030525Z layer_outputs = layer_module( 2025-09-07T07:20:00.6030742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6030825Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6031082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6031160Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6031425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6031491Z self_outputs = self.self( 2025-09-07T07:20:00.6031755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6031906Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6031910Z 2025-09-07T07:20:00.6032017Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6032211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6032275Z return mod(**inputs) 2025-09-07T07:20:00.6032548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6032626Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6032906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6032975Z hidden_states = self.encoder( 2025-09-07T07:20:00.6033232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6033310Z layer_outputs = layer_module( 2025-09-07T07:20:00.6033522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6033602Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6033861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6033970Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6034235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6034308Z self_outputs = self.self( 2025-09-07T07:20:00.6034582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6034703Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6034707Z 2025-09-07T07:20:00.6034817Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6035018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6035084Z return mod(**inputs) 2025-09-07T07:20:00.6035363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6035447Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6035725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6035797Z hidden_states = self.encoder( 2025-09-07T07:20:00.6036068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6036164Z layer_outputs = layer_module( 2025-09-07T07:20:00.6036411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6036497Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6036761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6036850Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6037123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6037195Z self_outputs = self.self( 2025-09-07T07:20:00.6037475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6037607Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6037610Z 2025-09-07T07:20:00.6037701Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6037783Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6037888Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6038096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6038162Z return mod(**inputs) 2025-09-07T07:20:00.6038442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6038525Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6038805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6038894Z hidden_states = self.encoder( 2025-09-07T07:20:00.6039167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6039247Z layer_outputs = layer_module( 2025-09-07T07:20:00.6039471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6039557Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6039830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6039913Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6040194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6040281Z self_outputs = self.self( 2025-09-07T07:20:00.6040562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6040682Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6040685Z 2025-09-07T07:20:00.6040792Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6041005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6041071Z return mod(**inputs) 2025-09-07T07:20:00.6041351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6041434Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6041716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6041790Z hidden_states = self.encoder( 2025-09-07T07:20:00.6042060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6042141Z layer_outputs = layer_module( 2025-09-07T07:20:00.6042368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6042477Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6042762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6042845Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6043126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6043258Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6043538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6043625Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6043629Z 2025-09-07T07:20:00.6043741Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6043943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6044009Z return mod(**inputs) 2025-09-07T07:20:00.6044284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6044366Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6044641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6044713Z hidden_states = self.encoder( 2025-09-07T07:20:00.6044983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6045061Z layer_outputs = layer_module( 2025-09-07T07:20:00.6045313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6045401Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6045688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6045793Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6046060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6046142Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6046458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6046603Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6046883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6046968Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6046972Z 2025-09-07T07:20:00.6047076Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6047288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6047355Z return mod(**inputs) 2025-09-07T07:20:00.6047631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6047714Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6047989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6048065Z hidden_states = self.encoder( 2025-09-07T07:20:00.6048334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6048416Z layer_outputs = layer_module( 2025-09-07T07:20:00.6048639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6048746Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6049037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6049124Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6049400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6049480Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6049794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6049916Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6050190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6050317Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6050536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6050617Z return self.act(input) 2025-09-07T07:20:00.6050621Z 2025-09-07T07:20:00.6050725Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6050937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6051005Z return mod(**inputs) 2025-09-07T07:20:00.6051281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6051368Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6051639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6051716Z hidden_states = self.encoder( 2025-09-07T07:20:00.6051976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6052044Z layer_outputs = layer_module( 2025-09-07T07:20:00.6052264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6052340Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6052610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6052708Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6052973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6053051Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6053347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6053491Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6053760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6053852Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6053856Z 2025-09-07T07:20:00.6053958Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6054159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6054239Z return mod(**inputs) 2025-09-07T07:20:00.6054513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6054604Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6054876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6054972Z hidden_states = self.encoder( 2025-09-07T07:20:00.6055262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6055334Z layer_outputs = layer_module( 2025-09-07T07:20:00.6055579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6055661Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6055957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6056055Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6056325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6056405Z self_outputs = self.self( 2025-09-07T07:20:00.6056675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6056778Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6056781Z 2025-09-07T07:20:00.6056884Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6057091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6057156Z return mod(**inputs) 2025-09-07T07:20:00.6057432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6057521Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6057780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6057877Z hidden_states = self.encoder( 2025-09-07T07:20:00.6058140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6058209Z layer_outputs = layer_module( 2025-09-07T07:20:00.6058434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6058510Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6058778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6058855Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6059130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6059208Z self_outputs = self.self( 2025-09-07T07:20:00.6059472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6059560Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6059565Z 2025-09-07T07:20:00.6059668Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6059871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6059937Z return mod(**inputs) 2025-09-07T07:20:00.6060202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6060289Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6060554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6060632Z hidden_states = self.encoder( 2025-09-07T07:20:00.6060895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6060964Z layer_outputs = layer_module( 2025-09-07T07:20:00.6061206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6061312Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6061585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6061666Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6061932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6062002Z self_outputs = self.self( 2025-09-07T07:20:00.6062263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6062363Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6062366Z 2025-09-07T07:20:00.6062447Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6062533Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6062636Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6062832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6062906Z return mod(**inputs) 2025-09-07T07:20:00.6063170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6063261Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6063535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6063608Z hidden_states = self.encoder( 2025-09-07T07:20:00.6063905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6063977Z layer_outputs = layer_module( 2025-09-07T07:20:00.6064210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6064289Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6064566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6064648Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6064918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6065029Z self_outputs = self.self( 2025-09-07T07:20:00.6065322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6065444Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6065448Z 2025-09-07T07:20:00.6065534Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6065644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6065967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6066044Z return mod(**inputs) 2025-09-07T07:20:00.6066343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6066431Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6066735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6066834Z hidden_states = self.encoder( 2025-09-07T07:20:00.6067104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6067187Z layer_outputs = layer_module( 2025-09-07T07:20:00.6067430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6067531Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6067812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6067893Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6068187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6068262Z self_outputs = self.self( 2025-09-07T07:20:00.6068557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6068732Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6069021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6069113Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6069118Z 2025-09-07T07:20:00.6069230Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6069451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6069522Z return mod(**inputs) 2025-09-07T07:20:00.6069827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6069915Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6070205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6070289Z hidden_states = self.encoder( 2025-09-07T07:20:00.6070606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6070689Z layer_outputs = layer_module( 2025-09-07T07:20:00.6070929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6071013Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6071309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6071396Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6071699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6071794Z self_outputs = self.self( 2025-09-07T07:20:00.6072086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6072256Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6072560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6072643Z x = self.pointwise(x) 2025-09-07T07:20:00.6072647Z 2025-09-07T07:20:00.6072760Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6072980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6073050Z return mod(**inputs) 2025-09-07T07:20:00.6073351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6073445Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6073729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6073814Z hidden_states = self.encoder( 2025-09-07T07:20:00.6074098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6074198Z layer_outputs = layer_module( 2025-09-07T07:20:00.6074454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6074535Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6074837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6074915Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6075175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6075245Z self_outputs = self.self( 2025-09-07T07:20:00.6075507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6075674Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6075677Z 2025-09-07T07:20:00.6075783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6075991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6076069Z return mod(**inputs) 2025-09-07T07:20:00.6076339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6076418Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6076678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6076757Z hidden_states = self.encoder( 2025-09-07T07:20:00.6077019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6077112Z layer_outputs = layer_module( 2025-09-07T07:20:00.6077334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6077412Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6077685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6077763Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6078034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6078122Z self_outputs = self.self( 2025-09-07T07:20:00.6078382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6078511Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6078515Z 2025-09-07T07:20:00.6078617Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6078823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6078889Z return mod(**inputs) 2025-09-07T07:20:00.6079164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6079246Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6079516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6079599Z hidden_states = self.encoder( 2025-09-07T07:20:00.6079867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6079949Z layer_outputs = layer_module( 2025-09-07T07:20:00.6080174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6080253Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6080562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6080645Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6080938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6081006Z self_outputs = self.self( 2025-09-07T07:20:00.6081281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6081409Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6081412Z 2025-09-07T07:20:00.6081493Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6081578Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6081682Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6081885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6081957Z return mod(**inputs) 2025-09-07T07:20:00.6082221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6082312Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6082576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6082655Z hidden_states = self.encoder( 2025-09-07T07:20:00.6082918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6083008Z layer_outputs = layer_module( 2025-09-07T07:20:00.6083235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6083311Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6083588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6083667Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6083942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6084013Z self_outputs = self.self( 2025-09-07T07:20:00.6084280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6084421Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6084424Z 2025-09-07T07:20:00.6084525Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6084733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6084802Z return mod(**inputs) 2025-09-07T07:20:00.6085076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6085168Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6085442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6085524Z hidden_states = self.encoder( 2025-09-07T07:20:00.6085792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6085874Z layer_outputs = layer_module( 2025-09-07T07:20:00.6086099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6086180Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6086463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6086577Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6086867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6086998Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6087260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6087351Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6087356Z 2025-09-07T07:20:00.6087457Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6087664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6087731Z return mod(**inputs) 2025-09-07T07:20:00.6087998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6088078Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6088341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6088419Z hidden_states = self.encoder( 2025-09-07T07:20:00.6088678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6088753Z layer_outputs = layer_module( 2025-09-07T07:20:00.6088980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6089053Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6089342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6089423Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6089691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6089770Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6090067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6090204Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6090461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6090563Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6090566Z 2025-09-07T07:20:00.6090665Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6090863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6090926Z return mod(**inputs) 2025-09-07T07:20:00.6091188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6091275Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6091532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6091608Z hidden_states = self.encoder( 2025-09-07T07:20:00.6091863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6091932Z layer_outputs = layer_module( 2025-09-07T07:20:00.6092154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6092231Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6092498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6092596Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6092879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6092957Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6093257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6093384Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6093669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6093787Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6093994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6094064Z return self.act(input) 2025-09-07T07:20:00.6094074Z 2025-09-07T07:20:00.6094176Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6094373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6094445Z return mod(**inputs) 2025-09-07T07:20:00.6094705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6094795Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6095063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6095137Z hidden_states = self.encoder( 2025-09-07T07:20:00.6095410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6096161Z layer_outputs = layer_module( 2025-09-07T07:20:00.6096397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6096476Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6096751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6096845Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6097115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6097220Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6097532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6097674Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6097942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6098025Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6098028Z 2025-09-07T07:20:00.6098139Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6098337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6098411Z return mod(**inputs) 2025-09-07T07:20:00.6098677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6098758Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6099033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6099106Z hidden_states = self.encoder( 2025-09-07T07:20:00.6099379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6099467Z layer_outputs = layer_module( 2025-09-07T07:20:00.6099710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6099789Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6100057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6100147Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6100416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6100496Z self_outputs = self.self( 2025-09-07T07:20:00.6100764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6100860Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6100863Z 2025-09-07T07:20:00.6100978Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6101181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6101256Z return mod(**inputs) 2025-09-07T07:20:00.6101523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6101608Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6101885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6101961Z hidden_states = self.encoder( 2025-09-07T07:20:00.6102240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6102329Z layer_outputs = layer_module( 2025-09-07T07:20:00.6102565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6102646Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6102921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6103011Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6103285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6103362Z self_outputs = self.self( 2025-09-07T07:20:00.6103649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6103732Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6103744Z 2025-09-07T07:20:00.6103848Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6104049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6104126Z return mod(**inputs) 2025-09-07T07:20:00.6104401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6104489Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6104766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6104839Z hidden_states = self.encoder( 2025-09-07T07:20:00.6105127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6105203Z layer_outputs = layer_module( 2025-09-07T07:20:00.6105450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6105531Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6105936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6106064Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6106349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6106433Z self_outputs = self.self( 2025-09-07T07:20:00.6106735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6106836Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6106848Z 2025-09-07T07:20:00.6106936Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6107022Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6107142Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6107363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6107438Z return mod(**inputs) 2025-09-07T07:20:00.6107713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6107797Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6108081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6108152Z hidden_states = self.encoder( 2025-09-07T07:20:00.6108422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6108495Z layer_outputs = layer_module( 2025-09-07T07:20:00.6108716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6108821Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6109085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6109170Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6109435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6109505Z self_outputs = self.self( 2025-09-07T07:20:00.6109781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6109899Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6109903Z 2025-09-07T07:20:00.6109988Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6110090Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6110295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6110361Z return mod(**inputs) 2025-09-07T07:20:00.6110626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6110715Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6110980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6111057Z hidden_states = self.encoder( 2025-09-07T07:20:00.6111321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6111391Z layer_outputs = layer_module( 2025-09-07T07:20:00.6111617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6111696Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6111965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6112101Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6112399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6112470Z self_outputs = self.self( 2025-09-07T07:20:00.6112733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6112908Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6113173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6113256Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6113261Z 2025-09-07T07:20:00.6113363Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6113565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6113639Z return mod(**inputs) 2025-09-07T07:20:00.6113910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6113998Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6114261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6114338Z hidden_states = self.encoder( 2025-09-07T07:20:00.6114607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6114681Z layer_outputs = layer_module( 2025-09-07T07:20:00.6114920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6115013Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6115293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6115375Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6115646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6115725Z self_outputs = self.self( 2025-09-07T07:20:00.6116012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6116205Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6116497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6116577Z x = self.pointwise(x) 2025-09-07T07:20:00.6116580Z 2025-09-07T07:20:00.6116687Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6116892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6116971Z return mod(**inputs) 2025-09-07T07:20:00.6117259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6117353Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6117644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6117723Z hidden_states = self.encoder( 2025-09-07T07:20:00.6118018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6118096Z layer_outputs = layer_module( 2025-09-07T07:20:00.6118343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6118427Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6118744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6118840Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6119125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6119208Z self_outputs = self.self( 2025-09-07T07:20:00.6119497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6119861Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6119870Z 2025-09-07T07:20:00.6119984Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6120200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6120280Z return mod(**inputs) 2025-09-07T07:20:00.6120571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6120668Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6120955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6121030Z hidden_states = self.encoder( 2025-09-07T07:20:00.6121325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6121402Z layer_outputs = layer_module( 2025-09-07T07:20:00.6121647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6121790Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6122080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6122165Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6122454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6122536Z self_outputs = self.self( 2025-09-07T07:20:00.6122818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6122954Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6122984Z 2025-09-07T07:20:00.6123097Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6123312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6123394Z return mod(**inputs) 2025-09-07T07:20:00.6123686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6123785Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6124076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6124160Z hidden_states = self.encoder( 2025-09-07T07:20:00.6124448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6124521Z layer_outputs = layer_module( 2025-09-07T07:20:00.6124770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6124854Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6125154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6125233Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6125526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6125630Z self_outputs = self.self( 2025-09-07T07:20:00.6125905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6126046Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6126050Z 2025-09-07T07:20:00.6126131Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6126222Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6126328Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6126531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6126608Z return mod(**inputs) 2025-09-07T07:20:00.6126883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6126973Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6127249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6127322Z hidden_states = self.encoder( 2025-09-07T07:20:00.6127604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6127684Z layer_outputs = layer_module( 2025-09-07T07:20:00.6127910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6127987Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6128266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6128354Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6128619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6128695Z self_outputs = self.self( 2025-09-07T07:20:00.6128959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6129078Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6129082Z 2025-09-07T07:20:00.6129184Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6129396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6129470Z return mod(**inputs) 2025-09-07T07:20:00.6129735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6129822Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6130087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6130158Z hidden_states = self.encoder( 2025-09-07T07:20:00.6130430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6130498Z layer_outputs = layer_module( 2025-09-07T07:20:00.6130723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6130801Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6131063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6131150Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6131415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6131566Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6131847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6131941Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6131945Z 2025-09-07T07:20:00.6132046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6132240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6132316Z return mod(**inputs) 2025-09-07T07:20:00.6132581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6132671Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6132931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6133004Z hidden_states = self.encoder( 2025-09-07T07:20:00.6133278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6133347Z layer_outputs = layer_module( 2025-09-07T07:20:00.6133570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6133647Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6133918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6134003Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6134265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6134366Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6134666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6134796Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6135063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6135146Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6135156Z 2025-09-07T07:20:00.6135258Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6135477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6135547Z return mod(**inputs) 2025-09-07T07:20:00.6135819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6135910Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6136182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6136254Z hidden_states = self.encoder( 2025-09-07T07:20:00.6136530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6136598Z layer_outputs = layer_module( 2025-09-07T07:20:00.6136827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6136904Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6137172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6137263Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6137534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6137642Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6137965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6138094Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6138365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6138481Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6138708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6138781Z return self.act(input) 2025-09-07T07:20:00.6138786Z 2025-09-07T07:20:00.6138897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6139100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6139165Z return mod(**inputs) 2025-09-07T07:20:00.6139450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6139532Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6139813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6139886Z hidden_states = self.encoder( 2025-09-07T07:20:00.6140164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6140240Z layer_outputs = layer_module( 2025-09-07T07:20:00.6140472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6140583Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6140874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6140974Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6141252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6141329Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6141639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6141776Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6142072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6142158Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6142162Z 2025-09-07T07:20:00.6142274Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6142479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6142547Z return mod(**inputs) 2025-09-07T07:20:00.6142829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6142912Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6143190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6143263Z hidden_states = self.encoder( 2025-09-07T07:20:00.6143536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6143616Z layer_outputs = layer_module( 2025-09-07T07:20:00.6143841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6143928Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6144249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6144332Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6144614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6144688Z self_outputs = self.self( 2025-09-07T07:20:00.6144991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6145091Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6145095Z 2025-09-07T07:20:00.6145210Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6145425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6145495Z return mod(**inputs) 2025-09-07T07:20:00.6145867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6145962Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6146270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6146350Z hidden_states = self.encoder( 2025-09-07T07:20:00.6146655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6146741Z layer_outputs = layer_module( 2025-09-07T07:20:00.6146986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6147101Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6147394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6147487Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6147762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6147834Z self_outputs = self.self( 2025-09-07T07:20:00.6148112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6148196Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6148219Z 2025-09-07T07:20:00.6148336Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6148536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6148604Z return mod(**inputs) 2025-09-07T07:20:00.6148885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6148970Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6149257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6149330Z hidden_states = self.encoder( 2025-09-07T07:20:00.6149615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6149687Z layer_outputs = layer_module( 2025-09-07T07:20:00.6149911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6150000Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6150280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6150370Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6150660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6150731Z self_outputs = self.self( 2025-09-07T07:20:00.6151027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6151120Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6151123Z 2025-09-07T07:20:00.6151211Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6151290Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6151399Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6151607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6151673Z return mod(**inputs) 2025-09-07T07:20:00.6151951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6152033Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6152311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6152384Z hidden_states = self.encoder( 2025-09-07T07:20:00.6152654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6152731Z layer_outputs = layer_module( 2025-09-07T07:20:00.6152955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6153042Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6153312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6153409Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6153686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6153756Z self_outputs = self.self( 2025-09-07T07:20:00.6154032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6154135Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6154139Z 2025-09-07T07:20:00.6154227Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6154333Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6154547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6154620Z return mod(**inputs) 2025-09-07T07:20:00.6154891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6154981Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6155252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6155326Z hidden_states = self.encoder( 2025-09-07T07:20:00.6155604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6155675Z layer_outputs = layer_module( 2025-09-07T07:20:00.6155904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6155982Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6156255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6156349Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6156617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6156696Z self_outputs = self.self( 2025-09-07T07:20:00.6157000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6157172Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6157446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6157524Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6157530Z 2025-09-07T07:20:00.6157644Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6157847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6157926Z return mod(**inputs) 2025-09-07T07:20:00.6158212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6158299Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6158598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6158675Z hidden_states = self.encoder( 2025-09-07T07:20:00.6158970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6159045Z layer_outputs = layer_module( 2025-09-07T07:20:00.6159283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6159376Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6159662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6159780Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6160071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6160154Z self_outputs = self.self( 2025-09-07T07:20:00.6160454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6160628Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6160930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6161039Z x = self.pointwise(x) 2025-09-07T07:20:00.6161043Z 2025-09-07T07:20:00.6161160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6161375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6161448Z return mod(**inputs) 2025-09-07T07:20:00.6161746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6161833Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6162125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6162202Z hidden_states = self.encoder( 2025-09-07T07:20:00.6162500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6162576Z layer_outputs = layer_module( 2025-09-07T07:20:00.6162816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6162908Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6163199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6163297Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6163635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6163712Z self_outputs = self.self( 2025-09-07T07:20:00.6164018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6164192Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6164196Z 2025-09-07T07:20:00.6164320Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6164543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6164621Z return mod(**inputs) 2025-09-07T07:20:00.6164920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6165009Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6165317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6165404Z hidden_states = self.encoder( 2025-09-07T07:20:00.6165702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6165777Z layer_outputs = layer_module( 2025-09-07T07:20:00.6166015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6166108Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6166394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6166508Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6166797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6166879Z self_outputs = self.self( 2025-09-07T07:20:00.6167169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6167296Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6167300Z 2025-09-07T07:20:00.6167418Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6167633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6167732Z return mod(**inputs) 2025-09-07T07:20:00.6168027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6168117Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6168422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6168503Z hidden_states = self.encoder( 2025-09-07T07:20:00.6168804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6168880Z layer_outputs = layer_module( 2025-09-07T07:20:00.6169122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6169213Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6169520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6169617Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6169914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6169997Z self_outputs = self.self( 2025-09-07T07:20:00.6170326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6170481Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6170485Z 2025-09-07T07:20:00.6170580Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6170664Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6170783Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6170995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6171066Z return mod(**inputs) 2025-09-07T07:20:00.6171363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6171452Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6171748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6171834Z hidden_states = self.encoder( 2025-09-07T07:20:00.6172113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6172185Z layer_outputs = layer_module( 2025-09-07T07:20:00.6172412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6172508Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6172772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6172860Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6173139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6173208Z self_outputs = self.self( 2025-09-07T07:20:00.6173483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6173596Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6173599Z 2025-09-07T07:20:00.6173709Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6173905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6173972Z return mod(**inputs) 2025-09-07T07:20:00.6174244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6174347Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6174624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6174698Z hidden_states = self.encoder( 2025-09-07T07:20:00.6175000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6175077Z layer_outputs = layer_module( 2025-09-07T07:20:00.6175318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6175410Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6175701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6175799Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6176090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6176235Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6176549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6176660Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6176664Z 2025-09-07T07:20:00.6176801Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6177022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6177101Z return mod(**inputs) 2025-09-07T07:20:00.6177409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6177499Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6177814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6177885Z hidden_states = self.encoder( 2025-09-07T07:20:00.6178158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6178228Z layer_outputs = layer_module( 2025-09-07T07:20:00.6178449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6178533Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6178798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6178888Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6179154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6179243Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6179553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6179698Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6179982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6180068Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6180071Z 2025-09-07T07:20:00.6180190Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6180404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6180472Z return mod(**inputs) 2025-09-07T07:20:00.6180773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6180878Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6181175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6181254Z hidden_states = self.encoder( 2025-09-07T07:20:00.6181549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6181625Z layer_outputs = layer_module( 2025-09-07T07:20:00.6181864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6181955Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6182242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6182339Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6182623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6182708Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6183040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6183188Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6183496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6183618Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6183855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6183932Z return self.act(input) 2025-09-07T07:20:00.6183936Z 2025-09-07T07:20:00.6184046Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6184268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6184338Z return mod(**inputs) 2025-09-07T07:20:00.6184630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6184717Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6185008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6185097Z hidden_states = self.encoder( 2025-09-07T07:20:00.6185390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6185473Z layer_outputs = layer_module( 2025-09-07T07:20:00.6185930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6186032Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6186342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6186473Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6186769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6186855Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6187203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6187348Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6187638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6187758Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6187762Z 2025-09-07T07:20:00.6187872Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6188095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6188169Z return mod(**inputs) 2025-09-07T07:20:00.6188472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6188572Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6188863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6188949Z hidden_states = self.encoder( 2025-09-07T07:20:00.6189236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6189319Z layer_outputs = layer_module( 2025-09-07T07:20:00.6189561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6189645Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6189944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6190030Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6190357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6190452Z self_outputs = self.self( 2025-09-07T07:20:00.6190744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6190853Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6190857Z 2025-09-07T07:20:00.6190969Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6191195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6191268Z return mod(**inputs) 2025-09-07T07:20:00.6191579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6191667Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6191974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6192061Z hidden_states = self.encoder( 2025-09-07T07:20:00.6192348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6192431Z layer_outputs = layer_module( 2025-09-07T07:20:00.6192671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6192755Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6193053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6193157Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6193461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6193538Z self_outputs = self.self( 2025-09-07T07:20:00.6193830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6193926Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6193929Z 2025-09-07T07:20:00.6194038Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6194260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6194330Z return mod(**inputs) 2025-09-07T07:20:00.6194641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6194728Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6195034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6195121Z hidden_states = self.encoder( 2025-09-07T07:20:00.6195426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6195510Z layer_outputs = layer_module( 2025-09-07T07:20:00.6195751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6195833Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6196141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6196229Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6196524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6196600Z self_outputs = self.self( 2025-09-07T07:20:00.6196932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6197032Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6197050Z 2025-09-07T07:20:00.6197139Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6197234Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6197347Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6197571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6197642Z return mod(**inputs) 2025-09-07T07:20:00.6197936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6198032Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6198326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6198408Z hidden_states = self.encoder( 2025-09-07T07:20:00.6198700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6198777Z layer_outputs = layer_module( 2025-09-07T07:20:00.6199023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6199106Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6199401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6199489Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6199783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6199873Z self_outputs = self.self( 2025-09-07T07:20:00.6200161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6200281Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6200285Z 2025-09-07T07:20:00.6200371Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6200487Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6200700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6200768Z return mod(**inputs) 2025-09-07T07:20:00.6201065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6201169Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6201460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6201539Z hidden_states = self.encoder( 2025-09-07T07:20:00.6201834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6201908Z layer_outputs = layer_module( 2025-09-07T07:20:00.6202147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6202235Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6202521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6202616Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6202904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6202978Z self_outputs = self.self( 2025-09-07T07:20:00.6203276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6203469Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6203781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6203866Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6203870Z 2025-09-07T07:20:00.6203989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6204201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6204271Z return mod(**inputs) 2025-09-07T07:20:00.6204572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6204658Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6204955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6205031Z hidden_states = self.encoder( 2025-09-07T07:20:00.6205325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6205409Z layer_outputs = layer_module( 2025-09-07T07:20:00.6205647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6205738Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6206024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6206114Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6206408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6206502Z self_outputs = self.self( 2025-09-07T07:20:00.6206798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6206970Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6207268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6207345Z x = self.pointwise(x) 2025-09-07T07:20:00.6207348Z 2025-09-07T07:20:00.6207459Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6207685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6207771Z return mod(**inputs) 2025-09-07T07:20:00.6208077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6208164Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6208465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6208550Z hidden_states = self.encoder( 2025-09-07T07:20:00.6208840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6208922Z layer_outputs = layer_module( 2025-09-07T07:20:00.6209163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6209253Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6209551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6209639Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6209937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6210011Z self_outputs = self.self( 2025-09-07T07:20:00.6210341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6210566Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6210570Z 2025-09-07T07:20:00.6210683Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6210904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6210975Z return mod(**inputs) 2025-09-07T07:20:00.6211273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6211360Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6211658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6211735Z hidden_states = self.encoder( 2025-09-07T07:20:00.6212043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6212130Z layer_outputs = layer_module( 2025-09-07T07:20:00.6212369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6212458Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6212762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6212850Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6213145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6213239Z self_outputs = self.self( 2025-09-07T07:20:00.6213545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6213675Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6213678Z 2025-09-07T07:20:00.6213798Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6214016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6214086Z return mod(**inputs) 2025-09-07T07:20:00.6214384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6214524Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6214818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6214898Z hidden_states = self.encoder( 2025-09-07T07:20:00.6215200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6215285Z layer_outputs = layer_module( 2025-09-07T07:20:00.6215525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6215615Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6215902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6216013Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6216324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6216399Z self_outputs = self.self( 2025-09-07T07:20:00.6216691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6216829Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6216832Z 2025-09-07T07:20:00.6216941Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6217027Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6217155Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6217383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6217452Z return mod(**inputs) 2025-09-07T07:20:00.6217746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6217834Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6218121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6218204Z hidden_states = self.encoder( 2025-09-07T07:20:00.6218491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6218573Z layer_outputs = layer_module( 2025-09-07T07:20:00.6218813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6218895Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6219193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6219279Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6219735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6219820Z self_outputs = self.self( 2025-09-07T07:20:00.6220123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6220302Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6220306Z 2025-09-07T07:20:00.6220420Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6220646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6220717Z return mod(**inputs) 2025-09-07T07:20:00.6221011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6221098Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6221383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6221493Z hidden_states = self.encoder( 2025-09-07T07:20:00.6221780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6221867Z layer_outputs = layer_module( 2025-09-07T07:20:00.6222109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6222202Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6222495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6222582Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6222880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6223021Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6223322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6223416Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6223420Z 2025-09-07T07:20:00.6223530Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6223779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6223852Z return mod(**inputs) 2025-09-07T07:20:00.6224175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6224264Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6224562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6224639Z hidden_states = self.encoder( 2025-09-07T07:20:00.6224929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6225013Z layer_outputs = layer_module( 2025-09-07T07:20:00.6225261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6225353Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6225650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6225797Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6226107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6226193Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6226538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6226675Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6226979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6227093Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6227097Z 2025-09-07T07:20:00.6227207Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6227440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6227507Z return mod(**inputs) 2025-09-07T07:20:00.6227782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6227863Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6228133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6228235Z hidden_states = self.encoder( 2025-09-07T07:20:00.6228510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6228594Z layer_outputs = layer_module( 2025-09-07T07:20:00.6228823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6228904Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6229188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6229273Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6229550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6229628Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6229945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6230067Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6230349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6230489Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6230722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6230802Z return self.act(input) 2025-09-07T07:20:00.6230806Z 2025-09-07T07:20:00.6230911Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6231115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6231189Z return mod(**inputs) 2025-09-07T07:20:00.6231464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6231555Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6231826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6231904Z hidden_states = self.encoder( 2025-09-07T07:20:00.6232176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6232249Z layer_outputs = layer_module( 2025-09-07T07:20:00.6232478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6232557Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6232836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6232922Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6233187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6233288Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6233594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6233738Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6234015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6234105Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6234109Z 2025-09-07T07:20:00.6234213Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6234419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6234510Z return mod(**inputs) 2025-09-07T07:20:00.6234788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6234879Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6235158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6235232Z hidden_states = self.encoder( 2025-09-07T07:20:00.6235516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6235587Z layer_outputs = layer_module( 2025-09-07T07:20:00.6235824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6235902Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6236186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6236270Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6236548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6236628Z self_outputs = self.self( 2025-09-07T07:20:00.6236935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6237055Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6237059Z 2025-09-07T07:20:00.6237167Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6237369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6237443Z return mod(**inputs) 2025-09-07T07:20:00.6237715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6237810Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6238081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6238163Z hidden_states = self.encoder( 2025-09-07T07:20:00.6238436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6238508Z layer_outputs = layer_module( 2025-09-07T07:20:00.6238752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6238831Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6239104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6239186Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6239452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6239546Z self_outputs = self.self( 2025-09-07T07:20:00.6239811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6239899Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6239904Z 2025-09-07T07:20:00.6240006Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6240205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6240280Z return mod(**inputs) 2025-09-07T07:20:00.6240546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6240634Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6240916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6240995Z hidden_states = self.encoder( 2025-09-07T07:20:00.6241264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6241334Z layer_outputs = layer_module( 2025-09-07T07:20:00.6241564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6241642Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6241915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6241994Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6242298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6242375Z self_outputs = self.self( 2025-09-07T07:20:00.6242637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6242738Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6242741Z 2025-09-07T07:20:00.6242820Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6242907Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6243028Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6243242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6243317Z return mod(**inputs) 2025-09-07T07:20:00.6243588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6243676Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6243945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6244015Z hidden_states = self.encoder( 2025-09-07T07:20:00.6244293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6244362Z layer_outputs = layer_module( 2025-09-07T07:20:00.6244591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6244669Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6244944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6245034Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6245307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6245388Z self_outputs = self.self( 2025-09-07T07:20:00.6245663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6245792Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6245797Z 2025-09-07T07:20:00.6245877Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6245983Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6246198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6246265Z return mod(**inputs) 2025-09-07T07:20:00.6246545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6246628Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6246900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6246998Z hidden_states = self.encoder( 2025-09-07T07:20:00.6247343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6247436Z layer_outputs = layer_module( 2025-09-07T07:20:00.6247657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6247734Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6248011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6248091Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6248363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6248431Z self_outputs = self.self( 2025-09-07T07:20:00.6248707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6248865Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6249134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6249218Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6249235Z 2025-09-07T07:20:00.6249338Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6249559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6249625Z return mod(**inputs) 2025-09-07T07:20:00.6249889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6249976Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6250244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6250324Z hidden_states = self.encoder( 2025-09-07T07:20:00.6250588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6250665Z layer_outputs = layer_module( 2025-09-07T07:20:00.6250884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6250961Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6251231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6251312Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6251580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6251648Z self_outputs = self.self( 2025-09-07T07:20:00.6251910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6252089Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6252354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6252431Z x = self.pointwise(x) 2025-09-07T07:20:00.6252434Z 2025-09-07T07:20:00.6252537Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6252739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6252804Z return mod(**inputs) 2025-09-07T07:20:00.6253069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6253172Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6253437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6253519Z hidden_states = self.encoder( 2025-09-07T07:20:00.6253782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6253853Z layer_outputs = layer_module( 2025-09-07T07:20:00.6254081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6254157Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6254426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6254507Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6254771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6254852Z self_outputs = self.self( 2025-09-07T07:20:00.6255123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6255289Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6255292Z 2025-09-07T07:20:00.6255414Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6255638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6255706Z return mod(**inputs) 2025-09-07T07:20:00.6255977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6256068Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6256346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6256432Z hidden_states = self.encoder( 2025-09-07T07:20:00.6256721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6256796Z layer_outputs = layer_module( 2025-09-07T07:20:00.6257043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6257121Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6257390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6257468Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6257740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6257811Z self_outputs = self.self( 2025-09-07T07:20:00.6258074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6258218Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6258222Z 2025-09-07T07:20:00.6258323Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6258526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6258591Z return mod(**inputs) 2025-09-07T07:20:00.6258855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6258942Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6259206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6259311Z hidden_states = self.encoder( 2025-09-07T07:20:00.6259582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6259659Z layer_outputs = layer_module( 2025-09-07T07:20:00.6259886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6259961Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6260241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6260320Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6260595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6260663Z self_outputs = self.self( 2025-09-07T07:20:00.6260932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6261071Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6261074Z 2025-09-07T07:20:00.6261158Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6261242Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6261343Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6261568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6261636Z return mod(**inputs) 2025-09-07T07:20:00.6261917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6262009Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6262274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6262353Z hidden_states = self.encoder( 2025-09-07T07:20:00.6262615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6262684Z layer_outputs = layer_module( 2025-09-07T07:20:00.6262910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6262985Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6263258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6263340Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6263606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6263682Z self_outputs = self.self( 2025-09-07T07:20:00.6263948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6264069Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6264072Z 2025-09-07T07:20:00.6264192Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6264393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6264457Z return mod(**inputs) 2025-09-07T07:20:00.6264742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6264839Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6265127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6265209Z hidden_states = self.encoder( 2025-09-07T07:20:00.6265495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6265585Z layer_outputs = layer_module( 2025-09-07T07:20:00.6265906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6265999Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6266297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6266388Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6266689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6266831Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6267118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6267210Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6267214Z 2025-09-07T07:20:00.6267315Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6267517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6267584Z return mod(**inputs) 2025-09-07T07:20:00.6267841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6267945Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6268228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6268305Z hidden_states = self.encoder( 2025-09-07T07:20:00.6268567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6268636Z layer_outputs = layer_module( 2025-09-07T07:20:00.6268859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6268933Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6269208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6269291Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6269562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6269643Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6269947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6270076Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6270354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6270444Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6270447Z 2025-09-07T07:20:00.6270547Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6270760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6270825Z return mod(**inputs) 2025-09-07T07:20:00.6271085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6271177Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6271436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6271514Z hidden_states = self.encoder( 2025-09-07T07:20:00.6271773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6271859Z layer_outputs = layer_module( 2025-09-07T07:20:00.6272081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6272159Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6272432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6272514Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6272767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6272851Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6273137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6273261Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6273525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6273640Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6273847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6273917Z return self.act(input) 2025-09-07T07:20:00.6273920Z 2025-09-07T07:20:00.6274040Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6274251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6274326Z return mod(**inputs) 2025-09-07T07:20:00.6274595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6274677Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6274958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6275033Z hidden_states = self.encoder( 2025-09-07T07:20:00.6275312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6275383Z layer_outputs = layer_module( 2025-09-07T07:20:00.6275618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6275698Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6275969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6276063Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6276328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6276413Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6276727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6276878Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6277153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6277240Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6277243Z 2025-09-07T07:20:00.6277358Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6277566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6277642Z return mod(**inputs) 2025-09-07T07:20:00.6277915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6278015Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6278296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6278372Z hidden_states = self.encoder( 2025-09-07T07:20:00.6278652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6278729Z layer_outputs = layer_module( 2025-09-07T07:20:00.6278957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6279047Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6279319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6279411Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6279685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6279760Z self_outputs = self.self( 2025-09-07T07:20:00.6280040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6280137Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6280141Z 2025-09-07T07:20:00.6280273Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6280494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6280570Z return mod(**inputs) 2025-09-07T07:20:00.6280842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6280923Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6281202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6281274Z hidden_states = self.encoder( 2025-09-07T07:20:00.6281550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6281623Z layer_outputs = layer_module( 2025-09-07T07:20:00.6281849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6281936Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6282212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6282302Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6282574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6282653Z self_outputs = self.self( 2025-09-07T07:20:00.6282933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6283017Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6283047Z 2025-09-07T07:20:00.6283160Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6283364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6283440Z return mod(**inputs) 2025-09-07T07:20:00.6283716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6283797Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6284126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6284200Z hidden_states = self.encoder( 2025-09-07T07:20:00.6284479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6284565Z layer_outputs = layer_module( 2025-09-07T07:20:00.6284806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6284886Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6285161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6285254Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6285525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6285604Z self_outputs = self.self( 2025-09-07T07:20:00.6285876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6285970Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6285974Z 2025-09-07T07:20:00.6286062Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6286143Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6286256Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6286459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6286539Z return mod(**inputs) 2025-09-07T07:20:00.6286840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6286923Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6287198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6287272Z hidden_states = self.encoder( 2025-09-07T07:20:00.6287551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6287624Z layer_outputs = layer_module( 2025-09-07T07:20:00.6287850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6287940Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6288213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6288304Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6288575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6288645Z self_outputs = self.self( 2025-09-07T07:20:00.6288924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6289030Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6289033Z 2025-09-07T07:20:00.6289119Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6289225Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6289440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6289517Z return mod(**inputs) 2025-09-07T07:20:00.6289798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6289892Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6290163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6290244Z hidden_states = self.encoder( 2025-09-07T07:20:00.6290517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6290603Z layer_outputs = layer_module( 2025-09-07T07:20:00.6290837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6290917Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6291205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6291288Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6291569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6291648Z self_outputs = self.self( 2025-09-07T07:20:00.6291927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6292106Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6292374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6292455Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6292460Z 2025-09-07T07:20:00.6292563Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6292762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6292854Z return mod(**inputs) 2025-09-07T07:20:00.6293136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6293225Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6293490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6293559Z hidden_states = self.encoder( 2025-09-07T07:20:00.6293836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6293908Z layer_outputs = layer_module( 2025-09-07T07:20:00.6294133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6294211Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6294498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6294581Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6294850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6294927Z self_outputs = self.self( 2025-09-07T07:20:00.6295205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6295373Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6295645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6295733Z x = self.pointwise(x) 2025-09-07T07:20:00.6295736Z 2025-09-07T07:20:00.6295846Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6296049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6296126Z return mod(**inputs) 2025-09-07T07:20:00.6296396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6296486Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6296755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6296844Z hidden_states = self.encoder( 2025-09-07T07:20:00.6297131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6297205Z layer_outputs = layer_module( 2025-09-07T07:20:00.6297437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6297516Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6297796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6297887Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6298157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6298235Z self_outputs = self.self( 2025-09-07T07:20:00.6298506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6298664Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6298678Z 2025-09-07T07:20:00.6298784Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6298988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6299064Z return mod(**inputs) 2025-09-07T07:20:00.6299367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6299459Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6299733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6299806Z hidden_states = self.encoder( 2025-09-07T07:20:00.6300086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6300160Z layer_outputs = layer_module( 2025-09-07T07:20:00.6300393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6300474Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6300754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6300846Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6301121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6301201Z self_outputs = self.self( 2025-09-07T07:20:00.6301471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6301599Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6301605Z 2025-09-07T07:20:00.6301708Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6301909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6302002Z return mod(**inputs) 2025-09-07T07:20:00.6302274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6302364Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6302636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6302709Z hidden_states = self.encoder( 2025-09-07T07:20:00.6302985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6303057Z layer_outputs = layer_module( 2025-09-07T07:20:00.6303305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6303382Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6303665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6303747Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6304024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6304104Z self_outputs = self.self( 2025-09-07T07:20:00.6304433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6304578Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6304582Z 2025-09-07T07:20:00.6304666Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6304751Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6304870Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6305093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6305172Z return mod(**inputs) 2025-09-07T07:20:00.6305471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6305573Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6305995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6306079Z hidden_states = self.encoder( 2025-09-07T07:20:00.6306392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6306470Z layer_outputs = layer_module( 2025-09-07T07:20:00.6306722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6306809Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6307116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6307215Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6307521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6307609Z self_outputs = self.self( 2025-09-07T07:20:00.6307910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6308035Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6308039Z 2025-09-07T07:20:00.6308162Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6308388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6308473Z return mod(**inputs) 2025-09-07T07:20:00.6308798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6308897Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6309206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6309287Z hidden_states = self.encoder( 2025-09-07T07:20:00.6309595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6309671Z layer_outputs = layer_module( 2025-09-07T07:20:00.6309922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6310026Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6310335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6310432Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6310745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6310894Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6311193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6311283Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6311293Z 2025-09-07T07:20:00.6311401Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6311625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6311706Z return mod(**inputs) 2025-09-07T07:20:00.6312001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6312094Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6312391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6312487Z hidden_states = self.encoder( 2025-09-07T07:20:00.6312809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6312883Z layer_outputs = layer_module( 2025-09-07T07:20:00.6313129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6313211Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6313543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6313645Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6313927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6314018Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6314347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6314490Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6314780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6314869Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6314872Z 2025-09-07T07:20:00.6314989Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6315208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6315286Z return mod(**inputs) 2025-09-07T07:20:00.6315589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6315720Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6316018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6316097Z hidden_states = self.encoder( 2025-09-07T07:20:00.6316396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6316472Z layer_outputs = layer_module( 2025-09-07T07:20:00.6316721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6316821Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6317107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6317209Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6317490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6317582Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6317907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6318038Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6318335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6318457Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6318697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6318773Z return self.act(input) 2025-09-07T07:20:00.6318778Z 2025-09-07T07:20:00.6318897Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6319112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6319187Z return mod(**inputs) 2025-09-07T07:20:00.6319532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6319781Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6320106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6320185Z hidden_states = self.encoder( 2025-09-07T07:20:00.6320493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6320587Z layer_outputs = layer_module( 2025-09-07T07:20:00.6320835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6320932Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6321249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6321340Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6321629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6321713Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6322047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6322197Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6322493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6322632Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6322636Z 2025-09-07T07:20:00.6322747Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6322972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6323043Z return mod(**inputs) 2025-09-07T07:20:00.6323343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6323429Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6323726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6323828Z hidden_states = self.encoder( 2025-09-07T07:20:00.6324120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6324197Z layer_outputs = layer_module( 2025-09-07T07:20:00.6324412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6324496Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6324753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6324837Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6325116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6325190Z self_outputs = self.self( 2025-09-07T07:20:00.6325478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-09-07T07:20:00.6325569Z mixed_query_layer = self.query(hidden_states) 2025-09-07T07:20:00.6325572Z 2025-09-07T07:20:00.6325672Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6325877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6325942Z return mod(**inputs) 2025-09-07T07:20:00.6326240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6326345Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6326630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6326703Z hidden_states = self.encoder( 2025-09-07T07:20:00.6326970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6327048Z layer_outputs = layer_module( 2025-09-07T07:20:00.6327261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6327344Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6327598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6327677Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6327946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6328015Z self_outputs = self.self( 2025-09-07T07:20:00.6328279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-09-07T07:20:00.6328357Z mixed_key_layer = self.key(hidden_states) 2025-09-07T07:20:00.6328362Z 2025-09-07T07:20:00.6328469Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6328666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6328750Z return mod(**inputs) 2025-09-07T07:20:00.6329024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6329105Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6329377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6329447Z hidden_states = self.encoder( 2025-09-07T07:20:00.6329712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6329789Z layer_outputs = layer_module( 2025-09-07T07:20:00.6330006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6330103Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6330371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6330453Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6330729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6330803Z self_outputs = self.self( 2025-09-07T07:20:00.6331083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-09-07T07:20:00.6331172Z mixed_value_layer = self.value(hidden_states) 2025-09-07T07:20:00.6331175Z 2025-09-07T07:20:00.6331261Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6331338Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6331441Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6331640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6331705Z return mod(**inputs) 2025-09-07T07:20:00.6331970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6332047Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6332349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6332426Z hidden_states = self.encoder( 2025-09-07T07:20:00.6332685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6332761Z layer_outputs = layer_module( 2025-09-07T07:20:00.6332978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6333056Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6333328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6333408Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6333678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6333746Z self_outputs = self.self( 2025-09-07T07:20:00.6334018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-09-07T07:20:00.6334119Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-09-07T07:20:00.6334123Z 2025-09-07T07:20:00.6334201Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6334308Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6334502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6334576Z return mod(**inputs) 2025-09-07T07:20:00.6334840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6334937Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6335224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6335297Z hidden_states = self.encoder( 2025-09-07T07:20:00.6335577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6335647Z layer_outputs = layer_module( 2025-09-07T07:20:00.6335880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6335957Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6336253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6336349Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6336639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6336723Z self_outputs = self.self( 2025-09-07T07:20:00.6337013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6337170Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6337444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-09-07T07:20:00.6337521Z x = self.depthwise(hidden_states) 2025-09-07T07:20:00.6337526Z 2025-09-07T07:20:00.6337635Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6337830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6337907Z return mod(**inputs) 2025-09-07T07:20:00.6338170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6338268Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6338552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6338623Z hidden_states = self.encoder( 2025-09-07T07:20:00.6338894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6338963Z layer_outputs = layer_module( 2025-09-07T07:20:00.6339182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6339269Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6339538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6339628Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6339893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6339963Z self_outputs = self.self( 2025-09-07T07:20:00.6340236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-09-07T07:20:00.6340390Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-09-07T07:20:00.6340659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-09-07T07:20:00.6340730Z x = self.pointwise(x) 2025-09-07T07:20:00.6340734Z 2025-09-07T07:20:00.6340843Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6341056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6341121Z return mod(**inputs) 2025-09-07T07:20:00.6341396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6341476Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6341751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6341821Z hidden_states = self.encoder( 2025-09-07T07:20:00.6342083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6342159Z layer_outputs = layer_module( 2025-09-07T07:20:00.6342391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6342473Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6342736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6342820Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6343086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6343154Z self_outputs = self.self( 2025-09-07T07:20:00.6343426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-09-07T07:20:00.6343575Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-09-07T07:20:00.6343579Z 2025-09-07T07:20:00.6343687Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6343883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6343952Z return mod(**inputs) 2025-09-07T07:20:00.6344229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6344310Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6345496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6345590Z hidden_states = self.encoder( 2025-09-07T07:20:00.6345973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6346054Z layer_outputs = layer_module( 2025-09-07T07:20:00.6346296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6346392Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6346683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6346781Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6347076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6347146Z self_outputs = self.self( 2025-09-07T07:20:00.6347420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-09-07T07:20:00.6347540Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-09-07T07:20:00.6347544Z 2025-09-07T07:20:00.6347657Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6347854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6347931Z return mod(**inputs) 2025-09-07T07:20:00.6348196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6348297Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6348568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6348640Z hidden_states = self.encoder( 2025-09-07T07:20:00.6348912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6348983Z layer_outputs = layer_module( 2025-09-07T07:20:00.6349201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6349287Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6349578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6349663Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6349931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6350002Z self_outputs = self.self( 2025-09-07T07:20:00.6350274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-09-07T07:20:00.6350403Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-09-07T07:20:00.6350407Z 2025-09-07T07:20:00.6350497Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6350576Z cudagraph partition due to non gpu ops 2025-09-07T07:20:00.6350685Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6350882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6350948Z return mod(**inputs) 2025-09-07T07:20:00.6351221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6351303Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6351572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6351662Z hidden_states = self.encoder( 2025-09-07T07:20:00.6351942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6352023Z layer_outputs = layer_module( 2025-09-07T07:20:00.6352243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6352329Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6352590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6352679Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6352941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-09-07T07:20:00.6353010Z self_outputs = self.self( 2025-09-07T07:20:00.6353288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-09-07T07:20:00.6353400Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-09-07T07:20:00.6353404Z 2025-09-07T07:20:00.6353513Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6353709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6353774Z return mod(**inputs) 2025-09-07T07:20:00.6354042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6354125Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6354392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6354480Z hidden_states = self.encoder( 2025-09-07T07:20:00.6354747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6354827Z layer_outputs = layer_module( 2025-09-07T07:20:00.6355049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6355133Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6355398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-09-07T07:20:00.6355511Z self_attention_outputs = self.attention( 2025-09-07T07:20:00.6355776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-09-07T07:20:00.6355907Z attention_output = self.output(self_outputs[0], hidden_states) 2025-09-07T07:20:00.6356185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-09-07T07:20:00.6356272Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6356276Z 2025-09-07T07:20:00.6356387Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6356589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6356655Z return mod(**inputs) 2025-09-07T07:20:00.6356946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6357028Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6357301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6357373Z hidden_states = self.encoder( 2025-09-07T07:20:00.6357647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6357716Z layer_outputs = layer_module( 2025-09-07T07:20:00.6357969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6358062Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6358318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6358406Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6358658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6358735Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6359031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6359150Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6359416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-09-07T07:20:00.6359497Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6359500Z 2025-09-07T07:20:00.6359603Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6359793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6359855Z return mod(**inputs) 2025-09-07T07:20:00.6360117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6360198Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6360459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6360546Z hidden_states = self.encoder( 2025-09-07T07:20:00.6360802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6360879Z layer_outputs = layer_module( 2025-09-07T07:20:00.6361094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6361177Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6361433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6361521Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6361791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6361869Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6362166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-09-07T07:20:00.6362286Z intermediate_output = self.intermediate(attention_output) 2025-09-07T07:20:00.6362556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-09-07T07:20:00.6362665Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-09-07T07:20:00.6362873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-09-07T07:20:00.6362950Z return self.act(input) 2025-09-07T07:20:00.6362954Z 2025-09-07T07:20:00.6363055Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6363257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6363322Z return mod(**inputs) 2025-09-07T07:20:00.6363591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-09-07T07:20:00.6363669Z generator_hidden_states = self.convbert( 2025-09-07T07:20:00.6363955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-09-07T07:20:00.6364037Z hidden_states = self.encoder( 2025-09-07T07:20:00.6364297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-09-07T07:20:00.6364374Z layer_outputs = layer_module( 2025-09-07T07:20:00.6364588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-09-07T07:20:00.6364667Z return super().__call__(*args, **kwargs) 2025-09-07T07:20:00.6364934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-09-07T07:20:00.6365016Z layer_output = apply_chunking_to_forward( 2025-09-07T07:20:00.6365281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-09-07T07:20:00.6365358Z return forward_fn(*input_tensors) 2025-09-07T07:20:00.6365668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-09-07T07:20:00.6365807Z layer_output = self.output(intermediate_output, attention_output) 2025-09-07T07:20:00.6366066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-09-07T07:20:00.6366153Z hidden_states = self.dense(hidden_states) 2025-09-07T07:20:00.6366156Z 2025-09-07T07:20:00.6366254Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6366454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6366535Z return mod(**inputs) 2025-09-07T07:20:00.6366796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-09-07T07:20:00.6366952Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-09-07T07:20:00.6367213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 876, in forward 2025-09-07T07:20:00.6367319Z hidden_states = self.dense(generator_hidden_states) 2025-09-07T07:20:00.6367322Z 2025-09-07T07:20:00.6367421Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6367621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6367700Z return mod(**inputs) 2025-09-07T07:20:00.6367959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 939, in forward 2025-09-07T07:20:00.6368090Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-09-07T07:20:00.6368094Z 2025-09-07T07:20:00.6368190Z cudagraph partition due to non gpu ops. Found from : 2025-09-07T07:20:00.6368392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 533, in forward_pass 2025-09-07T07:20:00.6368456Z return mod(**inputs) 2025-09-07T07:20:00.6368717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 945, in forward 2025-09-07T07:20:00.6368889Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-09-07T07:20:00.6368893Z 2025-09-07T07:20:12.7622642Z Compilation time (from dynamo_timed): 23.423675767 2025-09-07T07:20:12.7664249Z pass 2025-09-07T07:20:12.7664732Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-09-07T07:20:12.7666401Z TIMING: _recursive_pre_grad_passes:0.01034 _recursive_joint_graph_passes:0.62425 _recursive_post_grad_passes:0.54272 async_compile.wait:0.61224 code_gen:11.54342 inductor_compile:14.0679 backend_compile:19.0705 gc:0.00104 entire_frame_compile:23.42368 total_wall_time:23.42368 2025-09-07T07:20:12.7667499Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:23079 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2025-09-07T07:20:12.7668029Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-09-07T07:20:14.9368470Z accuracy pass_rate=95.35% 2025-09-07T07:20:14.9373324Z calls_captured gmean=0.00x mean=609.233x 2025-09-07T07:20:14.9378948Z unique_graphs gmean=0.00x mean=1.093x 2025-09-07T07:20:14.9379449Z graph_breaks gmean=0.00x mean=0.140x 2025-09-07T07:20:14.9379721Z unique_graph_breaks gmean=0.00x mean=0.047x 2025-09-07T07:20:14.9380065Z autograd_captures gmean=0.00x mean=0.000x 2025-09-07T07:20:14.9381031Z autograd_compiles gmean=0.00x mean=0.000x 2025-09-07T07:20:14.9381331Z cudagraph_skips gmean=0.00x mean=1.093x 2025-09-07T07:20:14.9381880Z compilation_latency mean=22.825 seconds 2025-09-07T07:20:15.9810111Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-09-07T07:20:16.2523316Z AlbertForMaskedLM PASS 2025-09-07T07:20:16.2523681Z AlbertForQuestionAnswering PASS 2025-09-07T07:20:16.2523919Z AllenaiLongformerBase PASS 2025-09-07T07:20:16.2528103Z BartForCausalLM PASS 2025-09-07T07:20:16.2528456Z BartForConditionalGeneration PASS 2025-09-07T07:20:16.2533248Z BertForMaskedLM PASS 2025-09-07T07:20:16.2533571Z BertForQuestionAnswering PASS 2025-09-07T07:20:16.2540556Z BlenderbotForCausalLM XFAIL 2025-09-07T07:20:16.2541362Z BlenderbotSmallForCausalLM PASS 2025-09-07T07:20:16.2546047Z BlenderbotSmallForConditionalGeneration PASS 2025-09-07T07:20:16.2546395Z CamemBert PASS 2025-09-07T07:20:16.2555388Z DebertaV2ForMaskedLM XFAIL 2025-09-07T07:20:16.2555753Z DebertaV2ForQuestionAnswering PASS 2025-09-07T07:20:16.2561671Z DistilBertForMaskedLM PASS 2025-09-07T07:20:16.2567178Z DistilBertForQuestionAnswering PASS 2025-09-07T07:20:16.2567525Z DistillGPT2 PASS 2025-09-07T07:20:16.2570735Z ElectraForCausalLM PASS 2025-09-07T07:20:16.2571201Z ElectraForQuestionAnswering PASS 2025-09-07T07:20:16.2577804Z GPT2ForSequenceClassification PASS 2025-09-07T07:20:16.2578253Z GoogleFnet PASS 2025-09-07T07:20:16.2578917Z LayoutLMForMaskedLM PASS 2025-09-07T07:20:16.2583964Z LayoutLMForSequenceClassification PASS 2025-09-07T07:20:16.2584419Z M2M100ForConditionalGeneration PASS 2025-09-07T07:20:16.2585537Z MBartForCausalLM PASS 2025-09-07T07:20:16.2596022Z MBartForConditionalGeneration PASS 2025-09-07T07:20:16.2596334Z MT5ForConditionalGeneration PASS 2025-09-07T07:20:16.2604989Z MegatronBertForCausalLM PASS 2025-09-07T07:20:16.2605363Z MegatronBertForQuestionAnswering PASS 2025-09-07T07:20:16.2605642Z MobileBertForMaskedLM PASS 2025-09-07T07:20:16.2609691Z MobileBertForQuestionAnswering PASS 2025-09-07T07:20:16.2617456Z OPTForCausalLM PASS 2025-09-07T07:20:16.2617795Z PLBartForCausalLM PASS 2025-09-07T07:20:16.2618180Z PLBartForConditionalGeneration PASS 2025-09-07T07:20:16.2622183Z PegasusForCausalLM PASS 2025-09-07T07:20:16.2622605Z PegasusForConditionalGeneration PASS 2025-09-07T07:20:16.2626050Z RobertaForCausalLM PASS 2025-09-07T07:20:16.2634120Z RobertaForQuestionAnswering PASS 2025-09-07T07:20:16.2634461Z T5ForConditionalGeneration PASS 2025-09-07T07:20:16.2638664Z T5Small PASS 2025-09-07T07:20:16.2640973Z TrOCRForCausalLM PASS 2025-09-07T07:20:16.2646042Z XGLMForCausalLM PASS 2025-09-07T07:20:16.2646967Z XLNetLMHeadModel PASS 2025-09-07T07:20:16.2653356Z YituTechConvBert PASS 2025-09-07T07:20:16.3179307Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-09-07T07:20:16.6144108Z AlbertForMaskedLM PASS 2025-09-07T07:20:16.6144448Z AlbertForQuestionAnswering PASS 2025-09-07T07:20:16.6146122Z AllenaiLongformerBase PASS 2025-09-07T07:20:16.6146431Z BartForCausalLM PASS 2025-09-07T07:20:16.6155798Z BartForConditionalGeneration PASS 2025-09-07T07:20:16.6160506Z BertForMaskedLM PASS 2025-09-07T07:20:16.6162710Z BertForQuestionAnswering PASS 2025-09-07T07:20:16.6172771Z BlenderbotForCausalLM PASS 2025-09-07T07:20:16.6177926Z BlenderbotSmallForCausalLM PASS 2025-09-07T07:20:16.6180383Z BlenderbotSmallForConditionalGeneration PASS 2025-09-07T07:20:16.6180712Z CamemBert PASS 2025-09-07T07:20:16.6180961Z DebertaV2ForMaskedLM PASS 2025-09-07T07:20:16.6181208Z DebertaV2ForQuestionAnswering PASS 2025-09-07T07:20:16.6182831Z DistilBertForMaskedLM PASS 2025-09-07T07:20:16.6183104Z DistilBertForQuestionAnswering PASS 2025-09-07T07:20:16.6185341Z DistillGPT2 PASS 2025-09-07T07:20:16.6200818Z ElectraForCausalLM PASS 2025-09-07T07:20:16.6205688Z ElectraForQuestionAnswering PASS 2025-09-07T07:20:16.6206001Z GPT2ForSequenceClassification PASS 2025-09-07T07:20:16.6206262Z GoogleFnet PASS 2025-09-07T07:20:16.6212058Z LayoutLMForMaskedLM PASS 2025-09-07T07:20:16.6212350Z LayoutLMForSequenceClassification PASS 2025-09-07T07:20:16.6212586Z M2M100ForConditionalGeneration PASS 2025-09-07T07:20:16.6220492Z MBartForCausalLM PASS 2025-09-07T07:20:16.6222732Z MBartForConditionalGeneration PASS 2025-09-07T07:20:16.6223007Z MT5ForConditionalGeneration PASS 2025-09-07T07:20:16.6223241Z MegatronBertForCausalLM PASS 2025-09-07T07:20:16.6229986Z MegatronBertForQuestionAnswering PASS 2025-09-07T07:20:16.6235419Z MobileBertForMaskedLM PASS 2025-09-07T07:20:16.6235919Z MobileBertForQuestionAnswering PASS 2025-09-07T07:20:16.6236163Z OPTForCausalLM PASS 2025-09-07T07:20:16.6243023Z PLBartForCausalLM PASS 2025-09-07T07:20:16.6243476Z PLBartForConditionalGeneration PASS 2025-09-07T07:20:16.6252676Z PegasusForCausalLM PASS 2025-09-07T07:20:16.6253123Z PegasusForConditionalGeneration PASS 2025-09-07T07:20:16.6253377Z RobertaForCausalLM PASS 2025-09-07T07:20:16.6253600Z RobertaForQuestionAnswering PASS 2025-09-07T07:20:16.6262899Z T5ForConditionalGeneration PASS 2025-09-07T07:20:16.6263203Z T5Small PASS 2025-09-07T07:20:16.6263913Z TrOCRForCausalLM PASS 2025-09-07T07:20:16.6277309Z XGLMForCausalLM PASS_BUT_FLAKY 2025-09-07T07:20:16.6277789Z XLNetLMHeadModel PASS 2025-09-07T07:20:16.6278047Z YituTechConvBert PASS 2025-09-07T07:20:16.6788673Z + sccache_epilogue 2025-09-07T07:20:16.6796509Z + echo '::group::Sccache Compilation Log' 2025-09-07T07:20:16.6800549Z ##[group]Sccache Compilation Log 2025-09-07T07:20:16.6802068Z + echo '=================== sccache compilation log ===================' 2025-09-07T07:20:16.6802369Z =================== sccache compilation log =================== 2025-09-07T07:20:16.6802775Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-09-07T07:20:16.7017149Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-09-07T07:20:16.7017910Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-09-07T07:20:16.7018300Z + sccache --show-stats 2025-09-07T07:20:16.7058890Z Compile requests 383 2025-09-07T07:20:16.7061378Z Compile requests executed 0 2025-09-07T07:20:16.7062127Z Cache hits 0 2025-09-07T07:20:16.7062417Z Cache misses 0 2025-09-07T07:20:16.7062630Z Cache hits rate - 2025-09-07T07:20:16.7062835Z Cache timeouts 0 2025-09-07T07:20:16.7063100Z Cache read errors 0 2025-09-07T07:20:16.7063360Z Forced recaches 0 2025-09-07T07:20:16.7063572Z Cache write errors 0 2025-09-07T07:20:16.7063814Z Cache errors 0 2025-09-07T07:20:16.7064030Z Compilations 0 2025-09-07T07:20:16.7064255Z Compilation failures 0 2025-09-07T07:20:16.7064482Z Non-cacheable compilations 0 2025-09-07T07:20:16.7064768Z Non-cacheable calls 41 2025-09-07T07:20:16.7065032Z Non-compilation calls 342 2025-09-07T07:20:16.7065269Z Unsupported compiler calls 0 2025-09-07T07:20:16.7065514Z Average cache write 0.000 s 2025-09-07T07:20:16.7065880Z Average compiler 0.000 s 2025-09-07T07:20:16.7066131Z Average cache read hit 0.000 s 2025-09-07T07:20:16.7066378Z Failed distributed compilations 0 2025-09-07T07:20:16.7066539Z 2025-09-07T07:20:16.7066619Z Non-cacheable reasons: 2025-09-07T07:20:16.7066825Z -E 41 2025-09-07T07:20:16.7067001Z 2025-09-07T07:20:16.7067176Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-09-07T07:20:16.7067689Z Version (client) 0.10.0 2025-09-07T07:20:16.7067900Z + sccache --stop-server 2025-09-07T07:20:16.7082460Z Stopping sccache server... 2025-09-07T07:20:16.7094805Z Compile requests 383 2025-09-07T07:20:16.7095152Z Compile requests executed 0 2025-09-07T07:20:16.7095395Z Cache hits 0 2025-09-07T07:20:16.7095616Z Cache misses 0 2025-09-07T07:20:16.7095874Z Cache hits rate - 2025-09-07T07:20:16.7096097Z Cache timeouts 0 2025-09-07T07:20:16.7096355Z Cache read errors 0 2025-09-07T07:20:16.7096591Z Forced recaches 0 2025-09-07T07:20:16.7097050Z Cache write errors 0 2025-09-07T07:20:16.7097263Z Cache errors 0 2025-09-07T07:20:16.7097475Z Compilations 0 2025-09-07T07:20:16.7097681Z Compilation failures 0 2025-09-07T07:20:16.7097916Z Non-cacheable compilations 0 2025-09-07T07:20:16.7098170Z Non-cacheable calls 41 2025-09-07T07:20:16.7098387Z Non-compilation calls 342 2025-09-07T07:20:16.7098599Z Unsupported compiler calls 0 2025-09-07T07:20:16.7098826Z Average cache write 0.000 s 2025-09-07T07:20:16.7099060Z Average compiler 0.000 s 2025-09-07T07:20:16.7099270Z Average cache read hit 0.000 s 2025-09-07T07:20:16.7099479Z Failed distributed compilations 0 2025-09-07T07:20:16.7099625Z 2025-09-07T07:20:16.7099697Z Non-cacheable reasons: 2025-09-07T07:20:16.7099880Z -E 41 2025-09-07T07:20:16.7100007Z 2025-09-07T07:20:16.7100178Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-09-07T07:20:16.7100476Z Version (client) 0.10.0 2025-09-07T07:20:16.7100716Z + echo ::endgroup:: 2025-09-07T07:20:16.7101163Z ##[endgroup] 2025-09-07T07:20:16.7101391Z + cleanup_workspace 2025-09-07T07:20:16.7101740Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-09-07T07:20:16.7102303Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-09-07T07:20:16.7102760Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-09-07T07:20:16.7103076Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-09-07T07:20:16.7103448Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-09-07T07:20:16.7103842Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-09-07T07:20:16.7104164Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-09-07T07:20:17.1471180Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-09-07T07:20:17.1471554Z with: 2025-09-07T07:20:17.1471781Z benchmark-results-dir: test/test-reports 2025-09-07T07:20:17.1472054Z dry-run: false 2025-09-07T07:20:17.1472256Z schema-version: v3 2025-09-07T07:20:17.1472670Z github-token: *** 2025-09-07T07:20:17.1472859Z env: 2025-09-07T07:20:17.1473044Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:17.1473415Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:17.1473799Z ##[endgroup] 2025-09-07T07:20:17.1497839Z ##[group]Run set -eux 2025-09-07T07:20:17.1498076Z set -eux 2025-09-07T07:20:17.1498238Z  2025-09-07T07:20:17.1498407Z if [[ -n "" ]]; then 2025-09-07T07:20:17.1498608Z  source "" 2025-09-07T07:20:17.1498783Z fi 2025-09-07T07:20:17.1499028Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-09-07T07:20:17.1499358Z  2025-09-07T07:20:17.1499520Z DEVICE_NAME="" 2025-09-07T07:20:17.1499704Z DEVICE_TYPE="" 2025-09-07T07:20:17.1499881Z  2025-09-07T07:20:17.1500132Z if command -v nvidia-smi; then 2025-09-07T07:20:17.1500454Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-09-07T07:20:17.1500836Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-09-07T07:20:17.1501207Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-09-07T07:20:17.1501510Z  python3 -mpip install torch==2.7.1 2025-09-07T07:20:17.1501794Z elif command -v rocminfo; then 2025-09-07T07:20:17.1502120Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-09-07T07:20:17.1502499Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-09-07T07:20:17.1502885Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-09-07T07:20:17.1503273Z  # GPU device name coming from rocminfo instead 2025-09-07T07:20:17.1503527Z  DEVICE_NAME=rocm 2025-09-07T07:20:17.1503900Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-09-07T07:20:17.1504241Z fi 2025-09-07T07:20:17.1504410Z  2025-09-07T07:20:17.1504619Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-09-07T07:20:17.1504910Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-09-07T07:20:17.1514145Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:17.1514427Z env: 2025-09-07T07:20:17.1514612Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:17.1514955Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:17.1515300Z ##[endgroup] 2025-09-07T07:20:17.1544157Z + [[ -n '' ]] 2025-09-07T07:20:17.1544494Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-09-07T07:20:17.3420153Z Defaulting to user installation because normal site-packages is not writeable 2025-09-07T07:20:18.1682168Z Collecting boto3==1.35.33 2025-09-07T07:20:18.1841360Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-09-07T07:20:18.4150795Z Collecting psutil==7.0.0 2025-09-07T07:20:18.4190585Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-09-07T07:20:18.4526882Z Collecting pynvml==12.0.0 2025-09-07T07:20:18.4577079Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-09-07T07:20:18.5006147Z Collecting s3transfer<0.11.0,>=0.10.0 2025-09-07T07:20:18.5046826Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-09-07T07:20:18.5105555Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-09-07T07:20:19.3363909Z Collecting botocore<1.36.0,>=1.35.33 2025-09-07T07:20:19.3402368Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-09-07T07:20:19.4717709Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-09-07T07:20:19.4764462Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-09-07T07:20:19.4863112Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-09-07T07:20:19.4863823Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-09-07T07:20:19.6675874Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-09-07T07:20:19.7779617Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-09-07T07:20:20.1310790Z Attempting uninstall: nvidia-ml-py 2025-09-07T07:20:20.1314806Z Found existing installation: nvidia-ml-py 11.525.84 2025-09-07T07:20:20.1320365Z Uninstalling nvidia-ml-py-11.525.84: 2025-09-07T07:20:20.1452555Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-09-07T07:20:20.1948906Z Attempting uninstall: psutil 2025-09-07T07:20:20.1949854Z Found existing installation: psutil 5.9.8 2025-09-07T07:20:20.1994858Z Uninstalling psutil-5.9.8: 2025-09-07T07:20:20.2001113Z Successfully uninstalled psutil-5.9.8 2025-09-07T07:20:20.3372576Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-09-07T07:20:20.4527134Z + DEVICE_NAME= 2025-09-07T07:20:20.4532034Z + DEVICE_TYPE= 2025-09-07T07:20:20.4536907Z + command -v nvidia-smi 2025-09-07T07:20:20.4538987Z + command -v rocminfo 2025-09-07T07:20:20.4539395Z + echo DEVICE_NAME= 2025-09-07T07:20:20.4539595Z + echo DEVICE_TYPE= 2025-09-07T07:20:20.4566408Z ##[group]Run set -eux 2025-09-07T07:20:20.4566611Z set -eux 2025-09-07T07:20:20.4566843Z  2025-09-07T07:20:20.4567082Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-09-07T07:20:20.4567309Z  echo "Missing github-token input" 2025-09-07T07:20:20.4567511Z  exit 1 2025-09-07T07:20:20.4567670Z fi 2025-09-07T07:20:20.4574050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:20.4574288Z env: 2025-09-07T07:20:20.4574448Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:20.4574781Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:20.4575098Z DEVICE_NAME: 2025-09-07T07:20:20.4575252Z DEVICE_TYPE: 2025-09-07T07:20:20.4575631Z GITHUB_TOKEN: *** 2025-09-07T07:20:20.4575800Z ##[endgroup] 2025-09-07T07:20:20.4598520Z + [[ -z *** ]] 2025-09-07T07:20:20.4637311Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-09-07T07:20:20.4637592Z with: 2025-09-07T07:20:20.4637912Z github-token: *** 2025-09-07T07:20:20.4638071Z env: 2025-09-07T07:20:20.4638237Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:20.4638524Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:20.4638826Z DEVICE_NAME: 2025-09-07T07:20:20.4638990Z DEVICE_TYPE: 2025-09-07T07:20:20.4639138Z ##[endgroup] 2025-09-07T07:20:20.4655671Z ##[group]Run set -eux 2025-09-07T07:20:20.4655857Z set -eux 2025-09-07T07:20:20.4656011Z  2025-09-07T07:20:20.4656297Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-09-07T07:20:20.4660834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:20.4661069Z env: 2025-09-07T07:20:20.4661224Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:20.4661516Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:20.4661808Z DEVICE_NAME: 2025-09-07T07:20:20.4661966Z DEVICE_TYPE: 2025-09-07T07:20:20.4662265Z GITHUB_TOKEN: *** 2025-09-07T07:20:20.4662560Z ##[endgroup] 2025-09-07T07:20:20.4685761Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 17524754606 i-085acfb4aecab35f4 2025-09-07T07:20:21.3682808Z setting job-id=49774397867 2025-09-07T07:20:21.3684549Z setting job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T07:20:21.3786852Z ##[group]Run set -eux 2025-09-07T07:20:21.3787088Z set -eux 2025-09-07T07:20:21.3787264Z  2025-09-07T07:20:21.3787439Z if [[ -n "" ]]; then 2025-09-07T07:20:21.3787653Z  source "" 2025-09-07T07:20:21.3787810Z fi 2025-09-07T07:20:21.3787962Z  2025-09-07T07:20:21.3788225Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-09-07T07:20:21.3788559Z  --schema-version "${SCHEMA_VERSION}" \ 2025-09-07T07:20:21.3788782Z  --repo "${REPO}" \ 2025-09-07T07:20:21.3789018Z  --head-branch "${HEAD_BRANCH}" \ 2025-09-07T07:20:21.3789240Z  --head-sha "${HEAD_SHA}" \ 2025-09-07T07:20:21.3789467Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-09-07T07:20:21.3789771Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-09-07T07:20:21.3789988Z  --job-id "${JOB_ID}" \ 2025-09-07T07:20:21.3790195Z  --job-name "${JOB_NAME}" 2025-09-07T07:20:21.3794983Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:21.3795231Z env: 2025-09-07T07:20:21.3795394Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:21.3795708Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:21.3796022Z DEVICE_NAME: 2025-09-07T07:20:21.3796190Z DEVICE_TYPE: 2025-09-07T07:20:21.3796355Z SCHEMA_VERSION: v3 2025-09-07T07:20:21.3796532Z REPO: pytorch/pytorch 2025-09-07T07:20:21.3796707Z HEAD_BRANCH: refs/heads/main 2025-09-07T07:20:21.3796943Z HEAD_SHA: 93fb23d6fae7c4e82c4239a1033e522088742634 2025-09-07T07:20:21.3797221Z WORKFLOW_RUN_ID: 17524754606 2025-09-07T07:20:21.3797398Z RUN_ATTEMPT: 1 2025-09-07T07:20:21.3797548Z JOB_ID: 49774397867 2025-09-07T07:20:21.3797848Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T07:20:21.3798167Z ##[endgroup] 2025-09-07T07:20:21.3825990Z + [[ -n '' ]] 2025-09-07T07:20:21.3827736Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 93fb23d6fae7c4e82c4239a1033e522088742634 --workflow-id 17524754606 --run-attempt 1 --job-id 49774397867 --job-name 'inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-09-07T07:20:21.4111234Z ##[group]Run set -eux 2025-09-07T07:20:21.4111441Z set -eux 2025-09-07T07:20:21.4111605Z  2025-09-07T07:20:21.4111798Z if [[ -n "" ]]; then 2025-09-07T07:20:21.4111991Z  source "" 2025-09-07T07:20:21.4112145Z fi 2025-09-07T07:20:21.4112291Z  2025-09-07T07:20:21.4112547Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-09-07T07:20:21.4117327Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:21.4117561Z env: 2025-09-07T07:20:21.4117719Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:21.4118019Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:21.4118335Z DEVICE_NAME: 2025-09-07T07:20:21.4118487Z DEVICE_TYPE: 2025-09-07T07:20:21.4118645Z ##[endgroup] 2025-09-07T07:20:21.4138629Z + [[ -n '' ]] 2025-09-07T07:20:21.4139225Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-09-07T07:20:21.4472651Z INFO:root:Fail to import torch to get the device name 2025-09-07T07:20:21.4597565Z ##[group]Run set -eux 2025-09-07T07:20:21.4597784Z set -eux 2025-09-07T07:20:21.4597946Z  2025-09-07T07:20:21.4598125Z # TODO (huydhn): Implement this part 2025-09-07T07:20:21.4598393Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-09-07T07:20:21.4603569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:21.4603822Z env: 2025-09-07T07:20:21.4603979Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:21.4604284Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:21.4604602Z DEVICE_NAME: 2025-09-07T07:20:21.4604766Z DEVICE_TYPE: 2025-09-07T07:20:21.4604925Z ##[endgroup] 2025-09-07T07:20:21.4626527Z + echo 'dependencies={}' 2025-09-07T07:20:21.4671320Z ##[group]Run set -eux 2025-09-07T07:20:21.4671533Z set -eux 2025-09-07T07:20:21.4671691Z  2025-09-07T07:20:21.4671840Z if [[ -n "" ]]; then 2025-09-07T07:20:21.4672052Z  source "" 2025-09-07T07:20:21.4672219Z fi 2025-09-07T07:20:21.4672367Z  2025-09-07T07:20:21.4672553Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-09-07T07:20:21.4672932Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-09-07T07:20:21.4673233Z  # We don't want the job to fail if the directory doesn't exist 2025-09-07T07:20:21.4673478Z  exit 0 2025-09-07T07:20:21.4673627Z fi 2025-09-07T07:20:21.4673775Z  2025-09-07T07:20:21.4673944Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-09-07T07:20:21.4674255Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-09-07T07:20:21.4674613Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-09-07T07:20:21.4674886Z  --metadata "${BENCHMARK_METADATA}" \ 2025-09-07T07:20:21.4675119Z  --runners "${RUNNER_INFO}" \ 2025-09-07T07:20:21.4675411Z  --dependencies "${DEPENDENCIES}" \ 2025-09-07T07:20:21.4675633Z  --dry-run 2025-09-07T07:20:21.4675798Z else 2025-09-07T07:20:21.4676046Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-09-07T07:20:21.4676387Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-09-07T07:20:21.4676662Z  --metadata "${BENCHMARK_METADATA}" \ 2025-09-07T07:20:21.4676876Z  --runners "${RUNNER_INFO}" \ 2025-09-07T07:20:21.4677094Z  --dependencies "${DEPENDENCIES}" 2025-09-07T07:20:21.4677300Z fi 2025-09-07T07:20:21.4681598Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:21.4681841Z env: 2025-09-07T07:20:21.4681998Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:21.4682308Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:21.4682624Z DEVICE_NAME: 2025-09-07T07:20:21.4682784Z DEVICE_TYPE: 2025-09-07T07:20:21.4682963Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-09-07T07:20:21.4683172Z DRY_RUN: false 2025-09-07T07:20:21.4683981Z BENCHMARK_METADATA: {"timestamp": 1757229621, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "93fb23d6fae7c4e82c4239a1033e522088742634", "workflow_id": 17524754606, "run_attempt": 1, "job_id": 49774397867} 2025-09-07T07:20:21.4685003Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-10-208.ec2.internal"}, "name": "", "type": ""}] 2025-09-07T07:20:21.4685385Z DEPENDENCIES: {} 2025-09-07T07:20:21.4685556Z ##[endgroup] 2025-09-07T07:20:21.4711137Z + [[ -n '' ]] 2025-09-07T07:20:21.4711388Z + [[ ! -d test/test-reports ]] 2025-09-07T07:20:21.4711626Z + [[ false == \t\r\u\e ]] 2025-09-07T07:20:21.4713689Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1757229621, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "93fb23d6fae7c4e82c4239a1033e522088742634", "workflow_id": 17524754606, "run_attempt": 1, "job_id": 49774397867}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-10-208.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-09-07T07:20:21.5946572Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17524754606/49774397867/inference_huggingface.json 2025-09-07T07:20:21.6258147Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-09-07T07:20:21.8504637Z ##[group]Run cat test/**/*_toprint.log || true 2025-09-07T07:20:21.8504949Z cat test/**/*_toprint.log || true 2025-09-07T07:20:21.8510013Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:21.8510368Z env: 2025-09-07T07:20:21.8510549Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:21.8510882Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:21.8511220Z DEVICE_NAME: 2025-09-07T07:20:21.8511398Z DEVICE_TYPE: 2025-09-07T07:20:21.8511573Z ##[endgroup] 2025-09-07T07:20:21.8583157Z cat: 'test/**/*_toprint.log': No such file or directory 2025-09-07T07:20:21.8609991Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-09-07T07:20:21.8610270Z kill "$MONITOR_SCRIPT_PID" 2025-09-07T07:20:21.8614822Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:21.8615084Z env: 2025-09-07T07:20:21.8615259Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:21.8615583Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:21.8615978Z DEVICE_NAME: 2025-09-07T07:20:21.8616149Z DEVICE_TYPE: 2025-09-07T07:20:21.8616324Z MONITOR_SCRIPT_PID: 48150 2025-09-07T07:20:21.8616526Z ##[endgroup] 2025-09-07T07:20:21.8719499Z Prepare all required actions 2025-09-07T07:20:21.8720113Z Getting action download info 2025-09-07T07:20:22.0402017Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-09-07T07:20:22.3124517Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-09-07T07:20:22.7450879Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-09-07T07:20:22.7451189Z with: 2025-09-07T07:20:22.7451546Z file-suffix: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T07:20:22.7451961Z s3-bucket: gha-artifacts 2025-09-07T07:20:22.7452187Z env: 2025-09-07T07:20:22.7452379Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:22.7452759Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:22.7453172Z DEVICE_NAME: 2025-09-07T07:20:22.7453361Z DEVICE_TYPE: 2025-09-07T07:20:22.7453552Z ##[endgroup] 2025-09-07T07:20:22.7480234Z ##[group]Run # Remove any previous test jsons if they exist 2025-09-07T07:20:22.7480611Z # Remove any previous test jsons if they exist 2025-09-07T07:20:22.7480890Z rm -f test-jsons-*.zip 2025-09-07T07:20:22.7481218Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-09-07T07:20:22.7485995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:22.7486254Z env: 2025-09-07T07:20:22.7486418Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:22.7486888Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:22.7487232Z DEVICE_NAME: 2025-09-07T07:20:22.7487406Z DEVICE_TYPE: 2025-09-07T07:20:22.7487705Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T07:20:22.7488053Z ##[endgroup] 2025-09-07T07:20:22.7684606Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-09-07T07:20:22.7714578Z ##[group]Run # Remove any previous test reports if they exist 2025-09-07T07:20:22.7714964Z # Remove any previous test reports if they exist 2025-09-07T07:20:22.7715254Z rm -f test-reports-*.zip 2025-09-07T07:20:22.7715559Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-09-07T07:20:22.7720590Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:22.7720866Z env: 2025-09-07T07:20:22.7721045Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:22.7721383Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:22.7721738Z DEVICE_NAME: 2025-09-07T07:20:22.7721948Z DEVICE_TYPE: 2025-09-07T07:20:22.7722245Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T07:20:22.7722585Z ##[endgroup] 2025-09-07T07:20:22.7777736Z adding: test/test-reports/inference_huggingface.csv (deflated 69%) 2025-09-07T07:20:22.7782048Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-09-07T07:20:22.7782840Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 63%) 2025-09-07T07:20:22.7810538Z ##[group]Run # Remove any previous usage logs if they exist 2025-09-07T07:20:22.7810887Z # Remove any previous usage logs if they exist 2025-09-07T07:20:22.7811156Z rm -f logs-*.zip 2025-09-07T07:20:22.7811421Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-09-07T07:20:22.7811768Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-09-07T07:20:22.7816549Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:22.7816823Z env: 2025-09-07T07:20:22.7816997Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:22.7817408Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:22.7817783Z DEVICE_NAME: 2025-09-07T07:20:22.7817959Z DEVICE_TYPE: 2025-09-07T07:20:22.7818379Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T07:20:22.7818713Z ##[endgroup] 2025-09-07T07:20:22.7888791Z adding: usage_log.txt (deflated 96%) 2025-09-07T07:20:22.7897447Z 2025-09-07T07:20:22.7897930Z zip error: Nothing to do! (logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip) 2025-09-07T07:20:22.7923490Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-09-07T07:20:22.7923834Z # Remove any previous debugging artifacts if they exist 2025-09-07T07:20:22.7924090Z rm -f debug-*.zip 2025-09-07T07:20:22.7924279Z if [ -d 'test/debug' ]; then 2025-09-07T07:20:22.7924508Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-09-07T07:20:22.7924728Z fi 2025-09-07T07:20:22.7929177Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:22.7929417Z env: 2025-09-07T07:20:22.7929569Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:22.7929885Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:22.7930198Z DEVICE_NAME: 2025-09-07T07:20:22.7930354Z DEVICE_TYPE: 2025-09-07T07:20:22.7930619Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867 2025-09-07T07:20:22.7930916Z ##[endgroup] 2025-09-07T07:20:22.8007779Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-09-07T07:20:22.8008020Z with: 2025-09-07T07:20:22.8008207Z s3-bucket: gha-artifacts 2025-09-07T07:20:22.8008454Z s3-prefix: pytorch/pytorch/17524754606/1/artifact 2025-09-07T07:20:22.8008709Z retention-days: 14 2025-09-07T07:20:22.8008896Z if-no-files-found: warn 2025-09-07T07:20:22.8009101Z path: test-jsons-*.zip 2025-09-07T07:20:22.8009292Z name: artifact 2025-09-07T07:20:22.8009478Z region: us-east-1 2025-09-07T07:20:22.8009642Z env: 2025-09-07T07:20:22.8009807Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:22.8010193Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:22.8010547Z DEVICE_NAME: 2025-09-07T07:20:22.8010716Z DEVICE_TYPE: 2025-09-07T07:20:22.8010887Z ##[endgroup] 2025-09-07T07:20:23.0762417Z NOTE: s3-prefix specified, ignoring name parameter 2025-09-07T07:20:23.0762916Z With the provided path, there will be 1 file uploaded 2025-09-07T07:20:23.0763438Z Uploading to s3 prefix: pytorch/pytorch/17524754606/1/artifact 2025-09-07T07:20:23.0800227Z Starting upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:23.1947871Z Finished upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:23.2099094Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-09-07T07:20:23.2099346Z with: 2025-09-07T07:20:23.2099534Z s3-bucket: gha-artifacts 2025-09-07T07:20:23.2099772Z s3-prefix: pytorch/pytorch/17524754606/1/artifact 2025-09-07T07:20:23.2100017Z retention-days: 14 2025-09-07T07:20:23.2100212Z if-no-files-found: error 2025-09-07T07:20:23.2100493Z path: test-reports-*.zip 2025-09-07T07:20:23.2100681Z name: artifact 2025-09-07T07:20:23.2100853Z region: us-east-1 2025-09-07T07:20:23.2101017Z env: 2025-09-07T07:20:23.2101178Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:23.2101502Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:23.2101843Z DEVICE_NAME: 2025-09-07T07:20:23.2102006Z DEVICE_TYPE: 2025-09-07T07:20:23.2102172Z ##[endgroup] 2025-09-07T07:20:23.4639665Z NOTE: s3-prefix specified, ignoring name parameter 2025-09-07T07:20:23.4643717Z With the provided path, there will be 1 file uploaded 2025-09-07T07:20:23.4644066Z Uploading to s3 prefix: pytorch/pytorch/17524754606/1/artifact 2025-09-07T07:20:23.4670132Z Starting upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:23.5963898Z Finished upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:23.6153828Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-09-07T07:20:23.6154063Z with: 2025-09-07T07:20:23.6154232Z s3-bucket: gha-artifacts 2025-09-07T07:20:23.6154448Z s3-prefix: pytorch/pytorch/17524754606/1/artifact 2025-09-07T07:20:23.6154672Z retention-days: 14 2025-09-07T07:20:23.6154842Z if-no-files-found: ignore 2025-09-07T07:20:23.6155046Z path: logs-*.zip 2025-09-07T07:20:23.6155223Z name: artifact 2025-09-07T07:20:23.6155389Z region: us-east-1 2025-09-07T07:20:23.6155558Z env: 2025-09-07T07:20:23.6155714Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:23.6156022Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:23.6156357Z DEVICE_NAME: 2025-09-07T07:20:23.6156518Z DEVICE_TYPE: 2025-09-07T07:20:23.6156683Z ##[endgroup] 2025-09-07T07:20:23.8701985Z NOTE: s3-prefix specified, ignoring name parameter 2025-09-07T07:20:23.8702348Z With the provided path, there will be 1 file uploaded 2025-09-07T07:20:23.8702698Z Uploading to s3 prefix: pytorch/pytorch/17524754606/1/artifact 2025-09-07T07:20:23.8734367Z Starting upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:23.9968465Z Finished upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:24.0117492Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-09-07T07:20:24.0117736Z with: 2025-09-07T07:20:24.0117917Z s3-bucket: gha-artifacts 2025-09-07T07:20:24.0118144Z s3-prefix: pytorch/pytorch/17524754606/1/artifact 2025-09-07T07:20:24.0118377Z retention-days: 14 2025-09-07T07:20:24.0118549Z if-no-files-found: ignore 2025-09-07T07:20:24.0118744Z path: debug-*.zip 2025-09-07T07:20:24.0118915Z name: artifact 2025-09-07T07:20:24.0119101Z region: us-east-1 2025-09-07T07:20:24.0119261Z env: 2025-09-07T07:20:24.0119422Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:24.0119942Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:24.0120288Z DEVICE_NAME: 2025-09-07T07:20:24.0120449Z DEVICE_TYPE: 2025-09-07T07:20:24.0120618Z ##[endgroup] 2025-09-07T07:20:24.2602818Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-09-07T07:20:24.2789645Z ##[group]Run # shellcheck disable=SC2156 2025-09-07T07:20:24.2789918Z # shellcheck disable=SC2156 2025-09-07T07:20:24.2790307Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-09-07T07:20:24.2795488Z shell: /usr/bin/bash -e {0} 2025-09-07T07:20:24.2795675Z env: 2025-09-07T07:20:24.2795838Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:24.2796150Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:24.2796478Z DEVICE_NAME: 2025-09-07T07:20:24.2796635Z DEVICE_TYPE: 2025-09-07T07:20:24.2796800Z ##[endgroup] 2025-09-07T07:20:24.4514068Z Prepare all required actions 2025-09-07T07:20:24.4514495Z Getting action download info 2025-09-07T07:20:24.5813468Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-09-07T07:20:24.5813730Z with: 2025-09-07T07:20:24.5813895Z job_id: 49774397867 2025-09-07T07:20:24.5814242Z job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T07:20:24.5814556Z workflow_name: inductor 2025-09-07T07:20:24.5814741Z workflow_run_id: 17524754606 2025-09-07T07:20:24.5814929Z workflow_attempt: 1 2025-09-07T07:20:24.5815094Z env: 2025-09-07T07:20:24.5815240Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:24.5815543Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:24.5815855Z DEVICE_NAME: 2025-09-07T07:20:24.5816037Z DEVICE_TYPE: 2025-09-07T07:20:24.5816272Z ##[endgroup] 2025-09-07T07:20:24.5839569Z ##[group]Run echo "workflow_id: 17524754606" 2025-09-07T07:20:24.5839873Z echo "workflow_id: 17524754606" 2025-09-07T07:20:24.5840102Z echo "workflow_attempt: 1" 2025-09-07T07:20:24.5840307Z echo "workflow_Name: inductor" 2025-09-07T07:20:24.5840516Z echo "job_id: 49774397867" 2025-09-07T07:20:24.5840874Z echo "job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" 2025-09-07T07:20:24.5841242Z echo "artifact_prefix: " 2025-09-07T07:20:24.5841444Z python3 --version 2025-09-07T07:20:24.5846708Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:24.5846944Z env: 2025-09-07T07:20:24.5847099Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:24.5847381Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:24.5847685Z DEVICE_NAME: 2025-09-07T07:20:24.5847843Z DEVICE_TYPE: 2025-09-07T07:20:24.5847997Z ##[endgroup] 2025-09-07T07:20:24.5870348Z workflow_id: 17524754606 2025-09-07T07:20:24.5870683Z workflow_attempt: 1 2025-09-07T07:20:24.5875038Z workflow_Name: inductor 2025-09-07T07:20:24.5879603Z job_id: 49774397867 2025-09-07T07:20:24.5884677Z job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-09-07T07:20:24.5889021Z artifact_prefix: 2025-09-07T07:20:24.5893519Z Python 3.9.23 2025-09-07T07:20:24.5926535Z ##[group]Run nick-fields/retry@v3.0.0 2025-09-07T07:20:24.5926753Z with: 2025-09-07T07:20:24.5926902Z shell: bash 2025-09-07T07:20:24.5927106Z timeout_minutes: 5 2025-09-07T07:20:24.5927285Z max_attempts: 5 2025-09-07T07:20:24.5927458Z retry_wait_seconds: 30 2025-09-07T07:20:24.5927805Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7 2025-09-07T07:20:24.5928172Z polling_interval_seconds: 1 2025-09-07T07:20:24.5928391Z warning_on_retry: true 2025-09-07T07:20:24.5928575Z continue_on_error: false 2025-09-07T07:20:24.5928756Z env: 2025-09-07T07:20:24.5928910Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:24.5929213Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:24.5929526Z DEVICE_NAME: 2025-09-07T07:20:24.5929688Z DEVICE_TYPE: 2025-09-07T07:20:24.5929844Z ##[endgroup] 2025-09-07T07:20:24.8569953Z Defaulting to user installation because normal site-packages is not writeable 2025-09-07T07:20:24.9158060Z Collecting python-dateutil==2.8.2 2025-09-07T07:20:24.9314022Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2025-09-07T07:20:25.6274420Z Collecting boto3==1.35.42 2025-09-07T07:20:25.6310414Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-09-07T07:20:26.0084770Z Collecting pandas==2.1.3 2025-09-07T07:20:26.0123888Z Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) 2025-09-07T07:20:26.1289765Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-09-07T07:20:26.1301096Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-09-07T07:20:26.1337240Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-09-07T07:20:26.1344158Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-09-07T07:20:26.1345044Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-09-07T07:20:26.7620457Z Collecting numpy<2,>=1.22.4 2025-09-07T07:20:26.7672913Z Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) 2025-09-07T07:20:26.9132778Z Collecting tzdata>=2022.1 2025-09-07T07:20:26.9171232Z Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB) 2025-09-07T07:20:26.9256127Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-09-07T07:20:26.9283221Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-09-07T07:20:26.9288712Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-09-07T07:20:26.9346031Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-09-07T07:20:26.9412426Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-09-07T07:20:26.9489917Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0) 2025-09-07T07:20:26.9491029Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.15.0) 2025-09-07T07:20:27.0660979Z Installing collected packages: python-dateutil, tzdata, numpy, pandas, boto3 2025-09-07T07:20:31.0193780Z Attempting uninstall: boto3 2025-09-07T07:20:31.0199171Z Found existing installation: boto3 1.35.33 2025-09-07T07:20:31.0265506Z Uninstalling boto3-1.35.33: 2025-09-07T07:20:31.0274072Z Successfully uninstalled boto3-1.35.33 2025-09-07T07:20:31.0719924Z Successfully installed boto3-1.35.42 numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2 2025-09-07T07:20:31.6636969Z Command completed after 1 attempt(s). 2025-09-07T07:20:31.6692224Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-09-07T07:20:31.6692652Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-09-07T07:20:31.6692986Z  --workflow-run-id "17524754606" \ 2025-09-07T07:20:31.6693204Z  --workflow-name "inductor" \ 2025-09-07T07:20:31.6693425Z  --workflow-run-attempt "1" \ 2025-09-07T07:20:31.6693628Z  --job-id "49774397867" \ 2025-09-07T07:20:31.6693952Z  --job-name "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" \ 2025-09-07T07:20:31.6694295Z  --local-path "" \ 2025-09-07T07:20:31.6694492Z  --artifact-prefix "" 2025-09-07T07:20:31.6699186Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:31.6699419Z env: 2025-09-07T07:20:31.6699577Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:31.6699878Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:31.6700189Z DEVICE_NAME: 2025-09-07T07:20:31.6700351Z DEVICE_TYPE: 2025-09-07T07:20:31.6700503Z ##[endgroup] 2025-09-07T07:20:32.5076296Z repo: pytorch/pytorch 2025-09-07T07:20:32.5077140Z Search for test log in s3 bucket: ossci-utilization 2025-09-07T07:20:32.5077601Z Downloading logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:32.5078231Z extracting usage_log.txt from zip file logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_49774397867.zip 2025-09-07T07:20:32.5078690Z Converted Log Model: UtilizationMetadata: 2025-09-07T07:20:32.5079530Z UtilizationMetadata(level='metadata', workflow_id='17524754606', job_id='49774397867', workflow_name='inductor', job_name='inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)', usage_collect_interval=1.0, data_model_version=1.5, start_at=1757228016, gpu_count=0, cpu_count=32, gpu_type=None, error=None) 2025-09-07T07:20:32.5080412Z [Db Segments] detected pytest cmd: 9, generated segments: 9 2025-09-07T07:20:32.5080674Z [db model] Peek db timeseries 2025-09-07T07:20:32.5080929Z :{ 2025-09-07T07:20:32.5081090Z "created_at": 1757229632, 2025-09-07T07:20:32.5081297Z "type": "utilization", 2025-09-07T07:20:32.5081487Z "tags": [ 2025-09-07T07:20:32.5081653Z "record" 2025-09-07T07:20:32.5081826Z ], 2025-09-07T07:20:32.5081989Z "time_stamp": 1757228016, 2025-09-07T07:20:32.5082190Z "repo": "pytorch/pytorch", 2025-09-07T07:20:32.5082397Z "workflow_id": 17524754606, 2025-09-07T07:20:32.5082596Z "run_attempt": 1, 2025-09-07T07:20:32.5082781Z "job_id": 49774397867, 2025-09-07T07:20:32.5082964Z "workflow_name": "inductor", 2025-09-07T07:20:32.5083322Z "job_name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", 2025-09-07T07:20:32.5083668Z "json_data": "{}" 2025-09-07T07:20:32.5083841Z } 2025-09-07T07:20:32.5084189Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/17524754606/1/49774397867/metadata 2025-09-07T07:20:32.5084786Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/17524754606/1/49774397867/metadata 2025-09-07T07:20:32.5085399Z Writing 320 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/17524754606/1/49774397867/time_series 2025-09-07T07:20:32.5086020Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/17524754606/1/49774397867/time_series 2025-09-07T07:20:32.6113070Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-09-07T07:20:32.6113397Z with: 2025-09-07T07:20:32.6113572Z env: 2025-09-07T07:20:32.6113741Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:32.6114106Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:32.6114465Z DEVICE_NAME: 2025-09-07T07:20:32.6114648Z DEVICE_TYPE: 2025-09-07T07:20:32.6114816Z ##[endgroup] 2025-09-07T07:20:32.6135753Z ##[group]Run set -eou pipefail 2025-09-07T07:20:32.6136141Z set -eou pipefail 2025-09-07T07:20:32.6136358Z  2025-09-07T07:20:32.6136624Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-09-07T07:20:32.6136923Z for _ in $(seq 1440); do 2025-09-07T07:20:32.6137153Z  # Break if no ssh session exists anymore 2025-09-07T07:20:32.6137372Z  if [ "$(who)" = "" ]; then 2025-09-07T07:20:32.6137560Z  break 2025-09-07T07:20:32.6137749Z  fi 2025-09-07T07:20:32.6137902Z  echo "." 2025-09-07T07:20:32.6138058Z  sleep 5 2025-09-07T07:20:32.6138213Z done 2025-09-07T07:20:32.6142757Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:32.6143010Z env: 2025-09-07T07:20:32.6143167Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:32.6143474Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:32.6143792Z DEVICE_NAME: 2025-09-07T07:20:32.6143952Z DEVICE_TYPE: 2025-09-07T07:20:32.6144102Z ##[endgroup] 2025-09-07T07:20:32.6171694Z Holding runner for 2 hours until all ssh sessions have logged out 2025-09-07T07:20:32.6244533Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-09-07T07:20:32.6245005Z # ignore expansion of "docker ps -q" since it could be empty 2025-09-07T07:20:32.6245289Z # shellcheck disable=SC2046 2025-09-07T07:20:32.6245527Z docker stop $(docker ps -q) || true 2025-09-07T07:20:32.6245764Z # Prune all of the docker images 2025-09-07T07:20:32.6245988Z docker system prune -af 2025-09-07T07:20:32.6250533Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:32.6250778Z env: 2025-09-07T07:20:32.6250930Z GIT_DEFAULT_BRANCH: main 2025-09-07T07:20:32.6251236Z DOCKER_CONTAINER_ID: 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:32.6251559Z DEVICE_NAME: 2025-09-07T07:20:32.6251721Z DEVICE_TYPE: 2025-09-07T07:20:32.6251872Z ##[endgroup] 2025-09-07T07:20:43.6122799Z 9c09efa4294e 2025-09-07T07:20:43.9060807Z Deleted Containers: 2025-09-07T07:20:43.9061165Z 9c09efa4294e419fd30a2085cbfbabe974e73f614da453bd3f1592b2360dfd6e 2025-09-07T07:20:43.9061400Z 2025-09-07T07:20:51.1319011Z Deleted Images: 2025-09-07T07:20:51.1319919Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 2025-09-07T07:20:51.1320808Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:383efb45082f20b8c808cb0ba4df693a01359592233f641f1f486911ac320a9a 2025-09-07T07:20:51.1321429Z deleted: sha256:662d8c9dfc7db2f5d004293de4f2b7647941dee4c916479ef082d17fcdfd9c47 2025-09-07T07:20:51.1321885Z deleted: sha256:ea5ad443c754124b3a5a209c2663376b4c156947edef1b982a336148bbf9114d 2025-09-07T07:20:51.1322302Z deleted: sha256:284be7504f072e0c04da4e2190e8d0e1de73835ed67be81f3ddd7eafd5d06a3a 2025-09-07T07:20:51.1322737Z deleted: sha256:2f49ff4be65f7ca55de8d7028fb3df7d08232a9f043aa7ba27d9393724286281 2025-09-07T07:20:51.1323194Z deleted: sha256:f63b503fdd1cca198aecefb9eef7ffbeb5fbc723f2a8462f50316e56cd403cbc 2025-09-07T07:20:51.1323633Z deleted: sha256:f9d46e08457013f0e71d608ac3dd95b79c41120060a80baefa684048cc15574e 2025-09-07T07:20:51.1324081Z deleted: sha256:cab76e28615751b6d6a703103b1da790a67cb3a4ee2e8814de51de18ff8b595d 2025-09-07T07:20:51.1324843Z deleted: sha256:0b2d09aa482371591a32563a5db71472822abd096a347967a9bd2a177737109f 2025-09-07T07:20:51.1325292Z deleted: sha256:d306d346d5da05e9fd04284304b1637a0bf01ee97397c688d19d783d5e133de9 2025-09-07T07:20:51.1325713Z deleted: sha256:bb3381a916d410a6e304540bb0796099dc780cd11f5829e734b337e0e79acfe4 2025-09-07T07:20:51.1326126Z deleted: sha256:bcf487c27e826c092985285163fb896e3324460b1774f3eb2a66623cd31e7d87 2025-09-07T07:20:51.1326542Z deleted: sha256:7d13485a9bdc5c0e64ac5085b25f4dded75c60f74090369c1b6f3f546ee37e94 2025-09-07T07:20:51.1326973Z deleted: sha256:55351d98a4197542fa7c78089671f447a6ef88cc554b7fad4fc522e8d4d187b6 2025-09-07T07:20:51.1327402Z deleted: sha256:f884bc0c4f9a994f3b3f1d82205f3a7014b05c84ad0c1c2fa3254d15a44f31e1 2025-09-07T07:20:51.1327822Z deleted: sha256:cdd16785a15239e518604ea9ea31405d5225fa6411d1c6d74d6523bcebf759ab 2025-09-07T07:20:51.1328246Z deleted: sha256:2c5bc1dc49446d7df5784578ae7c99460a93b502aa0c3b9deffbb95ec5216860 2025-09-07T07:20:51.1328674Z deleted: sha256:bae1e956be98416ce7d1a6c2c6ef0917f467238e19291786f8e1fed36fa81956 2025-09-07T07:20:51.1329087Z deleted: sha256:2cb1f002ab1126b0606999a9557b3f7f5da1e453d5376d29d95d60a979a215c4 2025-09-07T07:20:51.1329513Z deleted: sha256:25055a5f67b9bce8fac50ee1508dcb0f862ed154de5ded734e55f60edaca385f 2025-09-07T07:20:51.1329939Z deleted: sha256:98024e2dd34a5899240e41ae14f59c657cdc005040773e6ad7cfe3d67cdac7a8 2025-09-07T07:20:51.1330346Z deleted: sha256:8d2e75659096b4af8a20c3e9a6cce899b6e720f638eacdfd7d41ec8a736efdde 2025-09-07T07:20:51.1330791Z deleted: sha256:7741a6bf043548509c51c32e44734f30dfe07f91ca56c64422b004c3c0444e68 2025-09-07T07:20:51.1331194Z deleted: sha256:e2e63edbd2512e413c388888eabade05a2a7876adf20e7f0e0c3660ac3acbd3d 2025-09-07T07:20:51.1331589Z deleted: sha256:7fdea0f7711ee22084f87dc6d651598b5e5c5237de828105f698cb6a937d4c9c 2025-09-07T07:20:51.1331981Z deleted: sha256:486a2cf42f9492f291d59d48f3cec5a0a72449d8b6ad7d7a02596da237cdd154 2025-09-07T07:20:51.1332436Z deleted: sha256:a17da64c93a4939fad81a3ff6b6cb30f988176a6e0062fcf9c65e06cd9b9c3fb 2025-09-07T07:20:51.1332875Z deleted: sha256:70b4a3a917b8f95b19ae5dab6f404af8fa1c886022e4a1d785654013d5d876af 2025-09-07T07:20:51.1333258Z deleted: sha256:bd1b9d6a8aa636a67023800dcd85e4a3a7a7a21d65c6e6491d169fa65b4404a9 2025-09-07T07:20:51.1333642Z deleted: sha256:e3befcf3d3693c1d7bf0535e6e6722f0aabb0123805443ef5915dd5441ed0b00 2025-09-07T07:20:51.1334024Z deleted: sha256:4b4f846f1c4266b015f5fdf8dac5346c083c3aee2375e337172c112677c5a8c0 2025-09-07T07:20:51.1334398Z deleted: sha256:f05dc4d1350267b90e07af241a64f86a928fb3d8de75717ac04ec5a0433d042f 2025-09-07T07:20:51.1334794Z deleted: sha256:b6b4de696915fa2db09844ec9ac44dbb2940b655cd356404cf1ff03eec644dad 2025-09-07T07:20:51.1335255Z deleted: sha256:da008bbe1fc29cb35b3949040e97eb801f3264a56c4dd1b9d43a3cb54f2a39b2 2025-09-07T07:20:51.1335656Z deleted: sha256:261da5d14cad99ee11dcdaeb6055726f38fc12b7c559ee9c6d2ddc3f288f4828 2025-09-07T07:20:51.1336080Z deleted: sha256:16f900c60e70d685a85ca571ee0dada993a02217bdd6bb8b1d49169e7e28cf41 2025-09-07T07:20:51.1336522Z deleted: sha256:f57b18c5cde1d1dc553a15e1e98141d4afc0b4d0bb1182cc85b2c21bd18bb783 2025-09-07T07:20:51.1336930Z deleted: sha256:3c79105088ac60b231e4553752ee42cb6a87f9d32736b32f0c2123dddec724e7 2025-09-07T07:20:51.1337352Z deleted: sha256:df1ffff478908236efb6ceb8e05e6e078f12b864f4d24ce598cba7b961fad65c 2025-09-07T07:20:51.1337759Z deleted: sha256:8170255b562b59b76768f18a5b84b1ba887db93d3fe43b87a74bdc6be4f82014 2025-09-07T07:20:51.1338145Z deleted: sha256:c863cfe6bed704be5a54617331e27158b6f5a492dd6b9ed9c99d23db017cf5e1 2025-09-07T07:20:51.1338539Z deleted: sha256:e9e5a98c073f72c3abf9cc98724a31a3791535574ac78aeda7eb5df4580b21d0 2025-09-07T07:20:51.1338930Z deleted: sha256:0a42ac98735ca6578911218be7a7918001fe8aee1eb33d98f0d0a153d0e1102d 2025-09-07T07:20:51.1339323Z deleted: sha256:77d5a8aaa4d0fe1210dda9ac1f0fa3cf6141fea925b6240b9839d7505d021d3f 2025-09-07T07:20:51.1339712Z deleted: sha256:fa6ec46c43532dc01449df1cc403de8bb5872f859076e90658534c51c1487ef9 2025-09-07T07:20:51.1340175Z deleted: sha256:424a12dd5083283e19af48d31b7f2e33911ca8f459796f17280eaf5777a9aa25 2025-09-07T07:20:51.1340559Z deleted: sha256:8f0499601e14f1073e20ce889b45d12ab33264f9cf30359ac29dddbf58a311aa 2025-09-07T07:20:51.1340992Z deleted: sha256:5a5fae32dfb81abcd7bf374018b11e8e42a5aa39841d4b94e822d306c9af015b 2025-09-07T07:20:51.1341420Z deleted: sha256:d1bda89f22d383d38dfb7f7590b3bb202ccb91814034e7c7e2493306a10151ef 2025-09-07T07:20:51.1341851Z deleted: sha256:dbf16c1fcae146528685a8f745f9c505b24ba9ef009c42b1bd711ff7bf51b936 2025-09-07T07:20:51.1342291Z deleted: sha256:f9ec0065788f638325536a37427e2635b760a32457f20ca0acbcef6946b1041b 2025-09-07T07:20:51.1342707Z deleted: sha256:9d9911dac8fb2ff7db87329f38625d73f452dfef8822830048bbc00541c7df14 2025-09-07T07:20:51.1343118Z deleted: sha256:de4c1937129850e357b0de484d230569f628ac0bc883b12eff42932cd1e193ce 2025-09-07T07:20:51.1343528Z deleted: sha256:7b3c9e5b56a1d74226a5c1a54e5cb5e749012aa9b1d2376c6e7503757e29c35b 2025-09-07T07:20:51.1343946Z deleted: sha256:8062a6f28fc5fe2a199e1c1c40b6c43b7e29eb0c452492b47ec6900413b19cb6 2025-09-07T07:20:51.1344366Z deleted: sha256:f879aeffe6886f8da80462b571f9307aa63bb961645bec55ff579187a81cfd0b 2025-09-07T07:20:51.1344786Z deleted: sha256:5c6ef06b3536a430194aee509a784ee889c4a9d6248cb20fd9290e87e4ee2245 2025-09-07T07:20:51.1345199Z deleted: sha256:461aea034a25a2d72be6adfe9213c457c4cbf48724e9cb1c57987afb87668f21 2025-09-07T07:20:51.1345611Z deleted: sha256:e342cd1c71b7d0b024ea16b4a11f3f7fbbc2e3d11ef754c9d242aa50c4f8b0a3 2025-09-07T07:20:51.1346204Z deleted: sha256:bffd35a7fa1ddcfe05f79b7d3cae4180928eeea00eaab7ed7f484bc31adfc1d5 2025-09-07T07:20:51.1346666Z deleted: sha256:b34e33e7b04b5cbb5d5852199430593bfa18ddfe9081df42284230a14ebb739e 2025-09-07T07:20:51.1347106Z deleted: sha256:21d9b55338774d9ddc66d0bfcc92af9c8d2ecd94d1710b7049f5a811e411af7b 2025-09-07T07:20:51.1347544Z deleted: sha256:6cc2b33909585d17bf269fb8297ff881249e136137254734f7d23b9583208718 2025-09-07T07:20:51.1347976Z deleted: sha256:ca7f55b7c6d6cb11ddd8e187da34c2695fc2ce7655d652b9c9dc140a01ed056f 2025-09-07T07:20:51.1348403Z deleted: sha256:a3ece3d0ab6e99ef783c4f8d27d0e38504ab4477590ef556c16d22d92ba63a43 2025-09-07T07:20:51.1348822Z deleted: sha256:c137b0d41177c753aa1b69b11d0dd1f82420bf8520371866c845b53dca10b2d0 2025-09-07T07:20:51.1349246Z deleted: sha256:1e0d92b07bce12e511af59f608edd1932b10704d700f5e7538e406b90ecbb615 2025-09-07T07:20:51.1349664Z deleted: sha256:2ec3d01b3031e9da124d67410f54866ec5c679a0d6e4aee6b31608c45ce7fd77 2025-09-07T07:20:51.1350080Z deleted: sha256:308cffbd71363688c672b2043c6b9bf647cfb84593c42c3d88e3f36ee8f7f1b4 2025-09-07T07:20:51.1350503Z deleted: sha256:d965d9873fa450daba50a85d961f0835b14374167d84cfafa6060d16229f4229 2025-09-07T07:20:51.1350954Z deleted: sha256:effd997e222f62a34133bb2ecf9c0ffee151e5797f72e734d86a270d2e722374 2025-09-07T07:20:51.1351394Z deleted: sha256:0bbc1c78c10ee09c2697cfcce347dc9edbf82a7ccc25a6db6ee0a8dda398f7f2 2025-09-07T07:20:51.1351828Z deleted: sha256:214858e773d1ad73c2965c19b29cbfd3e2a974daa879163e1c1eb96567a7ee06 2025-09-07T07:20:51.1352247Z deleted: sha256:a9c7a2cd7ae229b26e84c093de657d0f4334d6cc9301991c6c3245ff62a9a71d 2025-09-07T07:20:51.1352670Z deleted: sha256:749a80551ef3f272e2517cb065bc7a5250da47d0b36bf74ed453caa9a5fee265 2025-09-07T07:20:51.1353088Z deleted: sha256:39b014c4e62d21c11df6c6d775d3f345675014292198981f455bacc4515a0f7b 2025-09-07T07:20:51.1353505Z deleted: sha256:0f087c9a894566644f825f5f87308d92e4cf149c51f7cd4769cbfaeefd3df791 2025-09-07T07:20:51.1353919Z deleted: sha256:dc6eb6dad5f9e332f00af553440e857b1467db1be43dd910cdb6830ba0898d50 2025-09-07T07:20:51.1354184Z 2025-09-07T07:20:51.1354281Z Total reclaimed space: 52.84GB 2025-09-07T07:20:51.1452871Z Post job cleanup. 2025-09-07T07:20:51.1499730Z Post job cleanup. 2025-09-07T07:20:51.2328817Z [command]/usr/bin/git version 2025-09-07T07:20:51.2367815Z git version 2.47.1 2025-09-07T07:20:51.2401431Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/a4d1c484-33a8-4d6f-b4f7-7b78bb1ca7d7/.gitconfig' 2025-09-07T07:20:51.2411461Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a4d1c484-33a8-4d6f-b4f7-7b78bb1ca7d7' before making global git config changes 2025-09-07T07:20:51.2412067Z Adding repository directory to the temporary git global config as a safe directory 2025-09-07T07:20:51.2418177Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-09-07T07:20:51.2465482Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-09-07T07:20:51.2498685Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-09-07T07:20:51.2814404Z Entering 'android/libs/fbjni' 2025-09-07T07:20:51.2877113Z Entering 'third_party/FP16' 2025-09-07T07:20:51.2929408Z Entering 'third_party/FXdiv' 2025-09-07T07:20:51.2982370Z Entering 'third_party/NNPACK' 2025-09-07T07:20:51.3041554Z Entering 'third_party/NVTX' 2025-09-07T07:20:51.3103157Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T07:20:51.3159157Z Entering 'third_party/XNNPACK' 2025-09-07T07:20:51.3223670Z Entering 'third_party/aiter' 2025-09-07T07:20:51.3277346Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T07:20:51.3342732Z Entering 'third_party/benchmark' 2025-09-07T07:20:51.3397413Z Entering 'third_party/composable_kernel' 2025-09-07T07:20:51.3464508Z Entering 'third_party/cpp-httplib' 2025-09-07T07:20:51.3520129Z Entering 'third_party/cpuinfo' 2025-09-07T07:20:51.3581196Z Entering 'third_party/cudnn_frontend' 2025-09-07T07:20:51.3638290Z Entering 'third_party/cutlass' 2025-09-07T07:20:51.3703494Z Entering 'third_party/fbgemm' 2025-09-07T07:20:51.3762438Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T07:20:51.3811818Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T07:20:51.3882797Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T07:20:51.3932484Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T07:20:51.3991873Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T07:20:51.4048937Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T07:20:51.4099575Z Entering 'third_party/fbgemm/external/json' 2025-09-07T07:20:51.4164428Z Entering 'third_party/flash-attention' 2025-09-07T07:20:51.4219093Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T07:20:51.4282985Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T07:20:51.4344111Z Entering 'third_party/flatbuffers' 2025-09-07T07:20:51.4399827Z Entering 'third_party/fmt' 2025-09-07T07:20:51.4459390Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T07:20:51.4514374Z Entering 'third_party/gloo' 2025-09-07T07:20:51.4573594Z Entering 'third_party/googletest' 2025-09-07T07:20:51.4626903Z Entering 'third_party/ideep' 2025-09-07T07:20:51.4681360Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T07:20:51.4743774Z Entering 'third_party/ittapi' 2025-09-07T07:20:51.4798581Z Entering 'third_party/kineto' 2025-09-07T07:20:51.4854985Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T07:20:51.4907284Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T07:20:51.4964567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T07:20:51.5019894Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T07:20:51.5072021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T07:20:51.5119528Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T07:20:51.5197798Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T07:20:51.5235506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T07:20:51.5294699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T07:20:51.5351657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T07:20:51.5411196Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T07:20:51.5468326Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T07:20:51.5530253Z Entering 'third_party/kleidiai' 2025-09-07T07:20:51.5583420Z Entering 'third_party/mimalloc' 2025-09-07T07:20:51.5640880Z Entering 'third_party/nlohmann' 2025-09-07T07:20:51.5703100Z Entering 'third_party/onnx' 2025-09-07T07:20:51.5776139Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T07:20:51.5826869Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T07:20:51.5888089Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T07:20:51.5943278Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T07:20:51.5995051Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T07:20:51.6058632Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T07:20:51.6108025Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T07:20:51.6163496Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T07:20:51.6218143Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T07:20:51.6275165Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T07:20:51.6336072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T07:20:51.6392103Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T07:20:51.6466450Z Entering 'third_party/pocketfft' 2025-09-07T07:20:51.6525636Z Entering 'third_party/protobuf' 2025-09-07T07:20:51.6581114Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T07:20:51.6637554Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T07:20:51.6698389Z Entering 'third_party/psimd' 2025-09-07T07:20:51.6758308Z Entering 'third_party/pthreadpool' 2025-09-07T07:20:51.6809575Z Entering 'third_party/pybind11' 2025-09-07T07:20:51.6866890Z Entering 'third_party/python-peachpy' 2025-09-07T07:20:51.6920051Z Entering 'third_party/sleef' 2025-09-07T07:20:51.6981044Z Entering 'third_party/tensorpipe' 2025-09-07T07:20:51.7035335Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T07:20:51.7090977Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T07:20:51.7146646Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T07:20:51.7198742Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T07:20:51.7253055Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T07:20:51.7331155Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-09-07T07:20:51.7357711Z http.https://github.com/.extraheader 2025-09-07T07:20:51.7362962Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-09-07T07:20:51.7399050Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-09-07T07:20:51.7703591Z Entering 'android/libs/fbjni' 2025-09-07T07:20:51.7740762Z http.https://github.com/.extraheader 2025-09-07T07:20:51.7777037Z Entering 'third_party/FP16' 2025-09-07T07:20:51.7811901Z http.https://github.com/.extraheader 2025-09-07T07:20:51.7847326Z Entering 'third_party/FXdiv' 2025-09-07T07:20:51.7882363Z http.https://github.com/.extraheader 2025-09-07T07:20:51.7917240Z Entering 'third_party/NNPACK' 2025-09-07T07:20:51.7955144Z http.https://github.com/.extraheader 2025-09-07T07:20:51.7988408Z Entering 'third_party/NVTX' 2025-09-07T07:20:51.8028306Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8076572Z Entering 'third_party/VulkanMemoryAllocator' 2025-09-07T07:20:51.8109380Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8145439Z Entering 'third_party/XNNPACK' 2025-09-07T07:20:51.8183899Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8229404Z Entering 'third_party/aiter' 2025-09-07T07:20:51.8263101Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8303425Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-09-07T07:20:51.8342437Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8386452Z Entering 'third_party/benchmark' 2025-09-07T07:20:51.8425892Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8460856Z Entering 'third_party/composable_kernel' 2025-09-07T07:20:51.8495605Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8541168Z Entering 'third_party/cpp-httplib' 2025-09-07T07:20:51.8580334Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8615683Z Entering 'third_party/cpuinfo' 2025-09-07T07:20:51.8656079Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8696503Z Entering 'third_party/cudnn_frontend' 2025-09-07T07:20:51.8732710Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8772031Z Entering 'third_party/cutlass' 2025-09-07T07:20:51.8802456Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8853325Z Entering 'third_party/fbgemm' 2025-09-07T07:20:51.8887391Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8920029Z Entering 'third_party/fbgemm/external/asmjit' 2025-09-07T07:20:51.8955478Z http.https://github.com/.extraheader 2025-09-07T07:20:51.8997040Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-09-07T07:20:51.9033486Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9078365Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-09-07T07:20:51.9110624Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9145445Z Entering 'third_party/fbgemm/external/cutlass' 2025-09-07T07:20:51.9177034Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9224526Z Entering 'third_party/fbgemm/external/googletest' 2025-09-07T07:20:51.9261278Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9295113Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-09-07T07:20:51.9334810Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9374939Z Entering 'third_party/fbgemm/external/json' 2025-09-07T07:20:51.9410830Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9456370Z Entering 'third_party/flash-attention' 2025-09-07T07:20:51.9489327Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9521135Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-09-07T07:20:51.9560311Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9599889Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-09-07T07:20:51.9636930Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9682173Z Entering 'third_party/flatbuffers' 2025-09-07T07:20:51.9718047Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9761398Z Entering 'third_party/fmt' 2025-09-07T07:20:51.9795540Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9841022Z Entering 'third_party/gemmlowp/gemmlowp' 2025-09-07T07:20:51.9866464Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9905747Z Entering 'third_party/gloo' 2025-09-07T07:20:51.9943245Z http.https://github.com/.extraheader 2025-09-07T07:20:51.9981820Z Entering 'third_party/googletest' 2025-09-07T07:20:52.0014472Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0058795Z Entering 'third_party/ideep' 2025-09-07T07:20:52.0094252Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0133943Z Entering 'third_party/ideep/mkl-dnn' 2025-09-07T07:20:52.0169031Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0214353Z Entering 'third_party/ittapi' 2025-09-07T07:20:52.0254997Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0288301Z Entering 'third_party/kineto' 2025-09-07T07:20:52.0328751Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0364500Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-09-07T07:20:52.0398012Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0432762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-09-07T07:20:52.0465981Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0502663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-09-07T07:20:52.0541149Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0577457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-09-07T07:20:52.0611082Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0737724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-09-07T07:20:52.0776307Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0810845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-09-07T07:20:52.0843639Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0881280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-09-07T07:20:52.0916086Z http.https://github.com/.extraheader 2025-09-07T07:20:52.0958979Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-09-07T07:20:52.0999306Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1026007Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-09-07T07:20:52.1062787Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1104911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-09-07T07:20:52.1144751Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1181266Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-09-07T07:20:52.1215059Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1253998Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-09-07T07:20:52.1287470Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1326458Z Entering 'third_party/kleidiai' 2025-09-07T07:20:52.1365797Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1402734Z Entering 'third_party/mimalloc' 2025-09-07T07:20:52.1442785Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1479818Z Entering 'third_party/nlohmann' 2025-09-07T07:20:52.1517414Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1559919Z Entering 'third_party/onnx' 2025-09-07T07:20:52.1593551Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1643852Z Entering 'third_party/onnx/third_party/pybind11' 2025-09-07T07:20:52.1679880Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1713752Z Entering 'third_party/opentelemetry-cpp' 2025-09-07T07:20:52.1758223Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1794062Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-09-07T07:20:52.1837977Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1876984Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-09-07T07:20:52.1910221Z http.https://github.com/.extraheader 2025-09-07T07:20:52.1946795Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-09-07T07:20:52.1980068Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2012729Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-09-07T07:20:52.2051616Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2088882Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-09-07T07:20:52.2120125Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2160260Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-09-07T07:20:52.2200967Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2235638Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-09-07T07:20:52.2273514Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2307731Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-09-07T07:20:52.2345426Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2381596Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-09-07T07:20:52.2416319Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2457605Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-09-07T07:20:52.2489979Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2543487Z Entering 'third_party/pocketfft' 2025-09-07T07:20:52.2578288Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2612246Z Entering 'third_party/protobuf' 2025-09-07T07:20:52.2651171Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2687963Z Entering 'third_party/protobuf/third_party/benchmark' 2025-09-07T07:20:52.2728810Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2765718Z Entering 'third_party/protobuf/third_party/googletest' 2025-09-07T07:20:52.2804256Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2845467Z Entering 'third_party/psimd' 2025-09-07T07:20:52.2876837Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2910190Z Entering 'third_party/pthreadpool' 2025-09-07T07:20:52.2947218Z http.https://github.com/.extraheader 2025-09-07T07:20:52.2987039Z Entering 'third_party/pybind11' 2025-09-07T07:20:52.3023449Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3063161Z Entering 'third_party/python-peachpy' 2025-09-07T07:20:52.3092006Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3127090Z Entering 'third_party/sleef' 2025-09-07T07:20:52.3170197Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3201710Z Entering 'third_party/tensorpipe' 2025-09-07T07:20:52.3241715Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3275319Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-09-07T07:20:52.3309621Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3355058Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-09-07T07:20:52.3388756Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3421310Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-09-07T07:20:52.3456272Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3494974Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-09-07T07:20:52.3531111Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3568807Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-09-07T07:20:52.3600552Z http.https://github.com/.extraheader 2025-09-07T07:20:52.3729442Z A job completed hook has been configured by the self-hosted runner administrator 2025-09-07T07:20:52.3748417Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-09-07T07:20:52.3752160Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-09-07T07:20:52.3752444Z ##[endgroup] 2025-09-07T07:20:52.3847716Z [!ALERT!] Swap in detected! [!ALERT!] 2025-09-07T07:21:01.4986196Z [!ALERT!] Swap out detected [!ALERT!] 2025-09-07T07:21:17.1492223Z Cleaning up orphan processes